Antialiased polygon fill doesn't respect area borders

edit

This is because Point is Point2i, an int type, and your floats are rounded to ints implicitly in the constructor. In your case, all points are halfway between two ints, and the default rounds to the nearest even int. Thus the polygon actually being rendered is degenerate, and only touches 3 pixels:

[(6,2), (6,2), (6,4), (6,4)]

fillPoly won't take floats; however, you can use the shift parameter to get a similar effect.

Modifying your example to multiply the points by 2 (a bit shift of 1) when converting the Point2f array to Point2i:

cv::Point points[4] = { A*2, B*2, C*2, D*2 };

And then the fillPoly command looks like:

cv::fillPoly(p, contours, lengths, 1, cv::Scalar(100), 0, 1);

Resulting in:

[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0, 100, 100,   0,   0;
   0,   0,   0,   0,   0,   0, 100, 100,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0]

and with CV_AA,

[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   4,  10,   6,   0,   0;
   0,   0,   0,   0,   0,  57, 100,  62,   0,   0;
   0,   0,   0,   0,   0,  60,  87,  63,   0,   0;
   0,   0,   0,   0,   0,   6,  12,   6,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0;
   0,   0,   0,   0,   0,   0,   0,   0,   0,   0]

...still a weird blob, but closer to being proportional.

Ok, the problem is this one: the 4th parameter of fillPoly is the position of the contour to fill, which in your case is 0, because you have only one contour (for verification just put 2 contours, to see that you ar filling the second). Just change like this:

cv::fillPoly(p, contours, lengths, 0, cv::Scalar(100));

If you run under Debug, the initial values are all the same, so if you call the function for the element on position 1 you are attempting to fill the poly on the position "after last element" (because you have just one element). If you will run this in Release mode, you will always get different values and on different positions in the matrix. In C++ the first element is always on the position 0. You have seen that if you changed the contour values, you are getting new different values in matrix, this is because the element "after the last element" has changed. In fact you are not using vectors, to see that if you are calling the contours[contours.size()] (which in your case is contours[1]) you are getting an error. This is a problem of using pointers and arrays, please try to use vectors and other types of STL (like smart pointers) for better seeing the problem.