This is an answer because it's so long.

Those videos are listed as unavailable.

Wait, so those are Vive tracking pucks in the corners? And those are the XYZ points displayed in between the images? Is the display just cutting off the negative sign that's always in those boxes?

I think what you want to be doing is forgetting the Vive tracking pucks entirely, and using the stereoCalibrate function. With a collection of images with chessboard detections, that will give you the translation and rotation of the cameras relative to eachother. If you set the vive as the first camera, it will be at (0,0,0) in both rvec and tvec, and the R and T outputs will be the relative location of the second camera.

Then you can use the triangulatePoints function to find the 3d world points of the chessboard, then verify that against what you measure from the Vive tracking pucks.

Do be careful, because the rotation and translation you get from the Vive feedback (if you're reading that directly from the API) is an OpenGL coordinate system, which takes some altering to make the OpenCV coordinate system. Specifically, multiply the rotation by

```
[1 0 0]
[0 -1 0]
[0 0 -1]
```

Then you have the location and orientation of the Vive relative to the world coordinate system, which is the inverse of what the rvec and tvec stores.

EDIT:

So there are three separate problems I see.

- The findChessboard function is returning very messy results. The problem isn't with SolvePnP, it's with the detected chessboard corners. You can see they don't match up in the frames where it goes bad.
- Finding the relative location of the second camera. This is exactly what stereoCalibrate is for. Use that, and you can see it gives you the relative transformation from the Vive to the other camera, if the Vive is in imagePoints1, and the other camera in imagePoints2. Importantly, this does not care about what coordinate system you put the objectPoints in. Just use the chessboard corners all at z = 0 like any normal camera calibration.
- Finding the location of things in the real world. Once you've stereoCalibrated, you can read the absolute location of the Vive, and calculate the absolute location of the camera from the calibration values. From there, you triangulate to get the absolute location of points in the images. This is properly another question entirely, as finding the relative location of the cameras shouldn't require any dealings in absolute coordinates.

Can you try drawing the axes on the video? The ArUco module has a function for that HERE.

It is possible for more than one set of rvec to refer to the same orientation, which could be the problem.

Secondly, what do the numbers mean? Are they locations relative to the camera?

It looks like the numbers are just negative numbers of the same magnitude, so you might be able to get away with just taking the absolute value, depending on what they represent.

Can you explain what points do you use for the pose estimation method? Because if you use

`findChessboardCorners`

, no need to use a RANSAC approach neither to use use 3 markers.Also, 3 marker points are not enough for the PnP problem (minimum is 4 and RANSAC is useless in this case).

Hi Tetragramm,

My opencv version have yet the ArUco module so I draw the axis of the chessboard by projecting XYZ axis using the rvec and tvec from solvepnp.

Here's the result with solvePnPRansac https://youtu.be/-n_NQgM97rU

Here's the result with solvePnP https://youtu.be/5tmpLn2L5sQ

About the numbers, they are world coordinate of 3 markers and the negative number is correct due to the chessboard in the negative area. The 3D world coordinate of chessboard corners are calculated base on these markers position. Is it possible to run solvePnP with negative coordinate?

@Eduardo,

My camera is attached on htc vive hmd and I want to calculate the relative between the camera and the vive hmd. The chessboard is used to calculated this correspondence. After get the relative position between the camera and the vive hmd, the chessboard will be removed.

I use 3 markers to calculate the chessboard corners (9x6) in world coordinate, so I have 54 points in total.