# Triangulation with Ground Plane as Origin

Hello, I am working on a project where I have two calibrated cameras (c1, c2) mounted to the ceiling of my lab and I want to triangulate points on objects that I place in the capture volume. I want my final output 3D points to be relative to the world origin that is placed on the ground plane (floor of the lab). I have some questions about my process and multiplying the necessary transformations. Here is what I have done so far...

To start I have captured an image, with c1, of the ChArUco board on the ground that will act as the origin of my "world". I detect corners ( cv::aruco::detectMarkers/ cv::aruco::interpolateCornersCharuco) in the image taken by c1 and obtain the transformation (with cv::projectPoints) from 3D world coordinates to 3D camera coordinates.

$\begin{pmatrix}{X_c_1} \\ {Y_c_1} \\ {Z_c_1} \\ 1 \end{pmatrix} = ^{c1}M_{board} \begin{pmatrix}{X_{board}} \\ {Y_{board}} \\ {Z_{board}} \\ 1 \end{pmatrix}$

I followed the same process of detecting corners on the ChArUco board with c2 (board in same position) and obtained the transformation that takes a point relative to the board origin to the camera origin...

$\begin{pmatrix}{X_c_2} \\ {Y_c_2} \\ {Z_c_2} \\ 1 \end{pmatrix} = ^{c2}M_{board} \begin{pmatrix}{X_{board}} \\ {Y_{board}} \\ {Z_{board}} \\ 1 \end{pmatrix}$

Q1. With the two transformations, and my calibrated intrinsic parameters, should I be able to pass these to cv::triangulatePoints to obtain 3D points that are relative to the ChArUco board origin?

Next, I was curious if I use cv::stereoCalibrate with my camera pair to obtain the transformation from camera 2 relative points to camera 1 relative points, could I combine this with the transform from camera 1 relative points to board relative points...to get a transform from camera 2 relative points to board relative points... After running cv::stereoCalibrate I obtain (where c1 is the origin camera that c2 transforms to)...

$\begin{pmatrix}{X_c_1} \\ {Y_c_1} \\ {Z_c_1} \\ 1 \end{pmatrix} = ^{c1}M_{c2} \begin{pmatrix}{X_{c2}} \\ {Y_{c2}} \\ {Z_{c2}} \\ 1 \end{pmatrix}$

Q2. Should I be able to combine transforms in the follow manner to get a transform that is the same (or very close) as my transform for board points to camera 2 points?

$\begin{pmatrix}{X_c_2} \\ {Y_c_2} \\ {Z_c_2} \\ 1 \end{pmatrix} = (^{c1}M_{c2})^{-1} \cdot ^{c1}M_{board} \begin{pmatrix}{X_{board}} \\ {Y_{board}} \\ {Z_{board}} \\ 1 \end{pmatrix}$

$^{c2}M_{board} \approx (^{c1}M_{c2})^{-1} \cdot ^{c1}M_{board}$

I tried to do this and noticed that the transform obtained by detecting the ChArUco board corners is significantly different than the one obtained by combing the transformations. Should this work as I stated, or have I misunderstood something and done the math incorrectly? Here is output I get for the two methods (translation units are meters)...

Output from projectPoints

$^{c2}M_{board} = \begin{bmatrix} 0.9844968 & -0.14832049 & 0.09363274 & -0.7521725\\ 0.01426749 & -0.46433134 & -0.88554664 & 1.10571043 \\ 0.17482132 & 0.87315373 & -0.45501656 & 3.89971067 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

Output from combined transforms (projectPoints w/ c1 and board, and stereoCalibrate w/ c1 and c2)

$(^{c1}M_{c2})^{-1} \cdot ^{c1}M_{board} = \begin{bmatrix} 0.9621638 & -0.00173254 & 0.01675597 & -1.03920386\\ -0.00161398 & -0.51909025 & -0.06325754 & 0.02077932 \\ -0.01954778 & -0.07318432 & -0.49902605 & 1.0988982 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

Looking at the transform obtained from projectPoints the translation makes sense as in the physical setup the ChArUco board is about 4m away from the camera. This makes me think the combined transform doesn't really make sense...

edit/update: Adding raw data from projectPoints and stereoCalibrate:

Sorry for the delay. Going through my code I use estimatePoseCharucoBoard to get my transformation matric from board coords to camera, sorry about that! Here are the matrices that I obtained;

Note: Any time that a calibration board object is needed the board dimensions given are in meters. So scaling should remain the same between matrices.

board to camera 190 from estimatePoseCharucoBoard --> c1^M_board

[[ 0.99662517 0.05033606 ...

edit retag close merge delete

I'd like clarification: how do you use projectPoints to get a transformation from board to camera frame? I believe that function has a different purpose and I'm not sure it's suited to the task.

( 2020-12-04 16:29:12 -0500 )edit

please post both board-to-camera matrices and the c2-to-c1 matrix you get from the calibration. I'd like to stare at them. -- basically, your matrix math is right. you know how to indicate the frames on the appropriate sides of the matrices, and you know to invert them when needed. at most the resulting matrix may need some scaling to get the bottom right element to be 1 again, but that's cosmetic. -- for purely mathematical correctness, I would introduce some w factor like w v' = M v to be perfectly correct about the happenings in a projective space. what's coming out of the matrix needn't be on the projective plane. -- I would suggest setting up some way to visualize your coordinate frames in a 3d space and maybe take some measurements off of them.

( 2020-12-04 16:37:04 -0500 )edit

I've stared at what's there some more. what's your input to stereoCalibrate? are you sure your objectPoints coordinates are scaled right? that's the easiest way I know to mess the resulting transformation up.

( 2020-12-04 16:39:52 -0500 )edit

Sorry for the delay! I've added some updated information. Thanks for the help, this is really confusing me...

( 2020-12-07 07:40:58 -0500 )edit

Sort by » oldest newest most voted

The issue was with the matrix multiplication library that I was using. I misunderstood one of the functions which led me to this nonsensical transformation matrix. Multiplying the matrices I added with my edit will output the correct transformation.

[[ 0.9829962 -0.1535926 0.10063695 -0.74704943]

[ 0.02044092 -0.45311975 -0.89121528 1.09928188]

[ 0.18248466 0.87811834 -0.44227541 3.86385995]

[ 0. 0. 0. 1. ]]

more

Official site

GitHub

Wiki

Documentation