# openCV triangulatePoints() and stereoRectify() relation ?

Hi,

I am currently trying to triangulate points from 2 images obtained by a stereo camera ( 2 cameras ) setup.

**I have already done** -

camera calibration for intrinsic parameters for each camera

then found their relative pose using the stereoCalibrate() function ( Output is R, and T vectors between the two cameras).

From what I understand, I need to now use stereoRectify() to simplify the stereoCorrespondence problem later during triangulation. The triangulation is not working hence I think the issue is during stereoRectify( ).

On getting new input image, I detect corners, then use cv::undistortPoints( ) on them.

Finally I try t triangulate but this is

**not working.**

The relevant part of my code is ( I am detecting corners using cv::findChessBoardCorners( ) )-

```
cv::stereoRectify(cameraMatrix1, distCoeffs1,
cameraMatrix2, distCoeffs2,
imageSize, R, T, R1, R2, P1, P2, Q,
CALIB_ZERO_DISPARITY, 1, imageSize, &validRoi[0], &validRoi[1]);
cv::triangulatePoints(P1,P2,undistortedCorners1,undistortedCorners2,pnts4D);
```

**However, can someone please explain the output of stereoRectify( ) ?** The output has P1 and P2 separately, and the documentation refers to them as Projection Matrix for each camera. However what I don't understand -

1) Why are these called projection matrices when they **dont have rotation elements** contained in it ?
It seems more like new camera matrix.

2) The output also contains R1 and R2. Why isn't this needed by cv::triangulatePoints( ) ?

3) **Am I supposed to pass P1 and P2 directly to cv::triangulatePoints()** ? Or do I need to multiply it with R1/R2 like
projMatrix1 = R1 * P1
projMatrix2 = R2 * P2

and then pass this to cv::triangulatePoints() ?

I would be really grateful if someone can help out with this!

Thanks !!

Hi, maybe it is easier to omit stereoRectify and start with simple sample which gives you more reasonable projection matrices. Try to take a look on this answer: http://answers.opencv.org/question/11....

I am still new to this, so would be better to check my answers with someone else 1. Projection matrix consist of instristic and extristic matrix (decomposition could be done). I feel like projection matrix is more named as camera matrix More about decomposition could be found here: http://ksimek.github.io/2012/08/14/de... (the article has 3 parts). 2. I do not know. 3. Yes. It should be passed directly.

So far with the answer mentioned earlier I got reasonable results.

Good luck!