Aruco Markers point position estimation

edit

I need to detect and track an area of fixed size (let's say 1mt by 1mt).

I am using 4 sets of 3 markers (each one for a different corner) so that by reading the marker Id I can know which corner it belongs to. Each set of marker has an L disposition (oriented to match the corner shape).

If I can detect all the markers I am able to successfully identify the target area. But, if one or more markers are not detected (e.g. they are outside the frame) I would like to use the others to estimate the missing points.

Unfortuately pose estimation is good locally around the marker but gives very poor results if i try to estimate the position of farther points.

The code I am using at the moment is something like this

Marker m; //filled, imagine this is the bottom left marker and I want the top right one
CameraParameters camParams; //filled
Mat objPoint(1,3,CV_32FC1);
objPoint.at<float>(0,0) = 1.0f;
objPoint.at<float>(0,1) = 1.0f;
objPoint.at<float>(0,2) = 0.0f;
// From the center of the marker I want to move 1mt along x axis and 1mt along y axis
vector<Point2f> imagePoints;
projectPoints(objectPoints, m.Rvec, m.Tvec, camParams.CameraMatrix, camParams.Distorsion, imagePoints);
Point2f estimate = imagePoints[0];

At this point I expect estimate to be the point I wanted expressed in screen pixels. But the result is often very bad. I also tried using PoseTrakcers and "homemade tracking" by saving the previous positions of the markers and then by trying to predict the next.

Am I doing something wrong? Is there a better way?

I am using OpenCV 3.4.1 and Aruco 3.0.6.

Thanks in advance.