Ask Your Question

Translation from SolvePnP seems wrong

asked 2018-01-02 07:16:33 -0500

Schotti gravatar image

updated 2018-01-02 07:28:15 -0500

I am currently working on an Android AR app. I try to track an Aruco Marker to set the Cameraposition in a volume rendering process on a server.

To get the translation- and rotationvectors I am using solvePnP.

    // set the obj 3D points
    double halfSize = sizeMeters / 2.0;
    List<Point3> objPoints = new ArrayList<Point3>();
    objPoints.add(new Point3(-halfSize, -halfSize, 0));
    objPoints.add(new Point3(-halfSize, halfSize, 0));
    objPoints.add(new Point3(halfSize, halfSize, 0));
    objPoints.add(new Point3(halfSize, -halfSize, 0));

    MatOfPoint3f objPointsMat = new MatOfPoint3f();
    MatOfPoint2f tempPoints= new MatOfPoint2f();
    Calib3d.solvePnP(objPointsMat, tempPoints, camMatrix, distCoeffs, Rvec, Tvec,false,Calib3d.CV_ITERATIVE);

I then try to generate the view matrix from both vectors.

            Matrix rotMat;
                Mat rVec = data.getRVec();
                Mat rMat = new Mat( 3, 3, CvType.CV_64F );
                Calib3d.Rodrigues( rVec, rMat );
                rotMat = new Matrix( rMat );

            // Generate translation vector from tVec
            Matrix tVec = new Matrix( data.getTVec() );

            Matrix viewMat;
               Matrix compMat = new Matrix( 4, 4, new double[]{
               rotMat.get( 0, 0 ), rotMat.get( 0, 1 ), rotMat.get( 0, 2 ), tVec.get( 0, 0 ),
               rotMat.get( 1, 0 ), rotMat.get( 1, 1 ), rotMat.get( 1, 2 ), tVec.get( 1, 0 ),
               -rotMat.get( 2, 0 ), -rotMat.get( 2, 1 ), -rotMat.get( 2, 2 ), -tVec.get( 2, 0 ),
               0.0, 0.0, 0.0, 1.0
                viewMat =  Matrix.transpose(compMat);

But I encountered the issue, that the object that gets rendered is not displayed correctly. It seemed, that the translation vector is not correct since the object won't stay at the marker position.

I then tried to render the object without the rotation vector and saw, that the translation vector is really a bit off. Furthermore I saw, that the error gets bigger when the distance to the marker is increased.


Did anybody encounter simiilar problems and knows where I fail? I am happy for any help.

Thank you in Advance. Best Regards Schotti

edit retag flag offensive close merge delete


Thank you for your reply. I found out, that the problem seems to be the solvePnP itself. The X and Y Values of the TVec seem to be wrong. They get bigger when the Z Value gets higher. For example: I only moved the camera away from the marker. The resulting TVecs were

(2.451448134824027) (1.2727915154079479) (6.332965784104173)

(4.625307053467822) (2.9585316223655993) (13.495538522635183)

(6.6035148163498) (3.6465885749382143) (20.11573099565936)

Schotti gravatar imageSchotti ( 2018-01-03 06:13:31 -0500 )edit

How do you compute tempPoints? You have to make sure that the 3D object points and the 2D image points match.

There is already an Aruco module in OpenCV contrib that does the marker detection and the pose estimation but I don't know if there is a Java binding.

Consider to display the object frame (cv::aruco::drawAxis()) to debug.

Your tvec values are meters as they seem really high? What is the value of halfSize?

Eduardo gravatar imageEduardo ( 2018-01-03 13:42:54 -0500 )edit

Hello Eduardo and thanks again for your quick reply. I did found the Aruco module for OpenCV but it seems that there are no Java bindings. The halfSize schould be 1 currently since the sizeMeters are currently set to 2, which should be some kind of relative size of the marker.

I tried to draw a cube on the image using the projectPoints method and the following picture is displayed. Picture Cube First I thought, that the marker detection itself works as intendet, but then I saw some strange behavior. I saw, that the IDs(that are currently displayed, which show the position of the point in the Collection) swap in some cases

Schotti gravatar imageSchotti ( 2018-01-04 04:55:05 -0500 )edit

Furthermore there is sometimes a strange flip: Here

This Flip also happens, when displaying the Axis with project Points. Here and Here

While this seems to be a big issue it does not explain why the translation vectors are wrong. There seems to be no correlation between the flipping axis and the error in the translation.

It is currently not possible to find a point of the camera, where the front face of the cube overlays the back face. This should be the case when the camera is directly above the marker.

Schotti gravatar imageSchotti ( 2018-01-04 06:06:38 -0500 )edit

To detect the markers I am using a github project you can find here But you here is the class that detects the marker, which seem to cause the problem.

Also I saw some strange behaviour when rotating the camera. The z Axis(and the cube accordingly) are always pointed towards the lower right corner of the display.

I thank you a lot for trying to help me.

Schotti gravatar imageSchotti ( 2018-01-04 08:11:15 -0500 )edit

What you can do is save some raw images and upload them somewhere. Upload also the corresponding objPointsMat, tempPoints, camMatrix, distCoeffs you have for the corresponding images.

This way I should be able to test with the C++ version (probably only next week).

If you use OpenGL to render the cube and the axis frame, the viewMat looks not correct according to this.

Eduardo gravatar imageEduardo ( 2018-01-04 14:34:42 -0500 )edit

Hello, I thank you again for your help. I managed to save the image and the corresponding Points/Mats. Thank you for your effort. It is much appreciated. I am not using OpenGL for the rendering. I am using the built in Imgprocessing methods of OpenCV. For the Cube and the Axis the projectPoints function was used.

Schotti gravatar imageSchotti ( 2018-01-08 08:20:05 -0500 )edit

Okay i think I found the failure. When calibrating the camera I am saving the CamMat. I read it out when I need it. The problem happened on reading the CamMat from the file. I did not read it correctly so that the center of the coordinates were expected to be in 0,0 rather than 640,360. I will test it out more if this resolves the Issue. if this is the case I am sorry for wasting your time.

Schotti gravatar imageSchotti ( 2018-01-09 08:08:01 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2018-01-10 04:57:52 -0500

Schotti gravatar image

As I metioned in the comments the problem was the CameraMatrix, which was not loaded correctly. It works now just as intended. Thank you for your contribution.

edit flag offensive delete link more

Question Tools

1 follower


Asked: 2018-01-02 07:16:33 -0500

Seen: 2,375 times

Last updated: Jan 10 '18