Translation from SolvePnP seems wrong
I am currently working on an Android AR app. I try to track an Aruco Marker to set the Cameraposition in a volume rendering process on a server.
To get the translation- and rotationvectors I am using solvePnP.
// set the obj 3D points
double halfSize = sizeMeters / 2.0;
List<Point3> objPoints = new ArrayList<Point3>();
objPoints.add(new Point3(-halfSize, -halfSize, 0));
objPoints.add(new Point3(-halfSize, halfSize, 0));
objPoints.add(new Point3(halfSize, halfSize, 0));
objPoints.add(new Point3(halfSize, -halfSize, 0));
MatOfPoint3f objPointsMat = new MatOfPoint3f();
objPointsMat.fromList(objPoints);
MatOfPoint2f tempPoints= new MatOfPoint2f();
tempPoints.fromList(this.points);
Calib3d.solvePnP(objPointsMat, tempPoints, camMatrix, distCoeffs, Rvec, Tvec,false,Calib3d.CV_ITERATIVE);
I then try to generate the view matrix from both vectors.
Matrix rotMat;
{
Mat rVec = data.getRVec();
Mat rMat = new Mat( 3, 3, CvType.CV_64F );
Calib3d.Rodrigues( rVec, rMat );
rotMat = new Matrix( rMat );
}
// Generate translation vector from tVec
Matrix tVec = new Matrix( data.getTVec() );
Matrix viewMat;
{
Matrix compMat = new Matrix( 4, 4, new double[]{
rotMat.get( 0, 0 ), rotMat.get( 0, 1 ), rotMat.get( 0, 2 ), tVec.get( 0, 0 ),
rotMat.get( 1, 0 ), rotMat.get( 1, 1 ), rotMat.get( 1, 2 ), tVec.get( 1, 0 ),
-rotMat.get( 2, 0 ), -rotMat.get( 2, 1 ), -rotMat.get( 2, 2 ), -tVec.get( 2, 0 ),
0.0, 0.0, 0.0, 1.0
}
viewMat = Matrix.transpose(compMat);
But I encountered the issue, that the object that gets rendered is not displayed correctly. It seemed, that the translation vector is not correct since the object won't stay at the marker position.
I then tried to render the object without the rotation vector and saw, that the translation vector is really a bit off. Furthermore I saw, that the error gets bigger when the distance to the marker is increased.
Did anybody encounter simiilar problems and knows where I fail? I am happy for any help.
Thank you in Advance. Best Regards Schotti
Some possible pointers:
Thank you for your reply. I found out, that the problem seems to be the solvePnP itself. The X and Y Values of the TVec seem to be wrong. They get bigger when the Z Value gets higher. For example: I only moved the camera away from the marker. The resulting TVecs were
(2.451448134824027) (1.2727915154079479) (6.332965784104173)
(4.625307053467822) (2.9585316223655993) (13.495538522635183)
(6.6035148163498) (3.6465885749382143) (20.11573099565936)
How do you compute
tempPoints? You have to make sure that the 3D object points and the 2D image points match.There is already an Aruco module in OpenCV contrib that does the marker detection and the pose estimation but I don't know if there is a Java binding.
Consider to display the object frame (
cv::aruco::drawAxis()) to debug.Your
tvecvalues are meters as they seem really high? What is the value ofhalfSize?Hello Eduardo and thanks again for your quick reply. I did found the Aruco module for OpenCV but it seems that there are no Java bindings. The halfSize schould be 1 currently since the sizeMeters are currently set to 2, which should be some kind of relative size of the marker.
I tried to draw a cube on the image using the projectPoints method and the following picture is displayed. Picture Cube First I thought, that the marker detection itself works as intendet, but then I saw some strange behavior. I saw, that the IDs(that are currently displayed, which show the position of the point in the Collection) swap in some cases
Furthermore there is sometimes a strange flip: Here
This Flip also happens, when displaying the Axis with project Points. Here and Here
While this seems to be a big issue it does not explain why the translation vectors are wrong. There seems to be no correlation between the flipping axis and the error in the translation.
It is currently not possible to find a point of the camera, where the front face of the cube overlays the back face. This should be the case when the camera is directly above the marker.
To detect the markers I am using a github project you can find here But you here is the class that detects the marker, which seem to cause the problem.
Also I saw some strange behaviour when rotating the camera. The z Axis(and the cube accordingly) are always pointed towards the lower right corner of the display.
I thank you a lot for trying to help me.
What you can do is save some raw images and upload them somewhere. Upload also the corresponding
objPointsMat,tempPoints,camMatrix,distCoeffsyou have for the corresponding images.This way I should be able to test with the C++ version (probably only next week).
If you use OpenGL to render the cube and the axis frame, the
viewMatlooks not correct according to this.Hello, I thank you again for your help. I managed to save the image and the corresponding Points/Mats. Thank you for your effort. It is much appreciated. I am not using OpenGL for the rendering. I am using the built in Imgprocessing methods of OpenCV. For the Cube and the Axis the projectPoints function was used.
Okay i think I found the failure. When calibrating the camera I am saving the CamMat. I read it out when I need it. The problem happened on reading the CamMat from the file. I did not read it correctly so that the center of the coordinates were expected to be in 0,0 rather than 640,360. I will test it out more if this resolves the Issue. if this is the case I am sorry for wasting your time.