The problem is not with the function but with how you're passing the parameters. The points defining the vertices must be in the appropriate order to achieve your desired result (clockwise or counter-clockwise, but not skipping middle points).
Some examples. If the points are passed in order:
Mat image(50, 50, CV_8U, Scalar(0));
vector<Point> pts;
pts.push_back(Point(5, 10)); //top-left
pts.push_back(Point(45, 5)); //top-right
pts.push_back(Point(45, 45)); //bottom-right
pts.push_back(Point(5, 40)); //bottom-left
fillConvexPoly(image, pts, 255);
If the points are not in order:
Mat image(50, 50, CV_8U, Scalar(0));
vector<Point> pts;
pts.push_back(Point(5, 10)); //top-left
pts.push_back(Point(45, 45)); //bottom-right
pts.push_back(Point(5, 40)); //bottom-left
pts.push_back(Point(45, 5)); //top-right
fillConvexPoly(image, pts, 255);
So, you have to input the points in the correct order. For simple rectangle-shaped figures it might be simple enough to do it manually. However, you can do it automatically using function convexHull()
:
Mat image(50, 50, CV_8U, Scalar(0));
vector<Point> pts;
pts.push_back(Point(5, 10)); //top-left
pts.push_back(Point(45, 45)); //bottom-right
pts.push_back(Point(5, 40)); //bottom-left
pts.push_back(Point(45, 5)); //top-right
convexHull(pts, pts); //reorder the points
fillConvexPoly(image, pts, 255);