Ask Your Question

Stereo Rectify doesn't rectify even with correct M and D

asked 2016-04-18 09:01:01 -0500

HG gravatar image

I cannot figure out a problem with stereoRectify or initUndistortRectifyMap. It might be related to this question but the user didn't post enough information for me to determine it. I do stereo calibration in two passes, because then we can split the intrinsic and extrinsic calibration, but it doesn't seem to work. These are inputs: Uncalibrated inputs

You can see radial lens distortion that needs to be corrected. So after each camera calibration I get a pretty good rectified images:


Left Camera Matrix: [1.0316489177476062e+003, 0., 6.5285947709947743e+002, 0., 1.0306889512875423e+003, 5.2195206055917618e+002, 0., 0., 1.]

Left Camera D: [-3.6474261084702730e-001, 1.7160954305339343e-001, 1.3998315284130826e-004, -5.7337307704477671e-005, -4.3398439779963091e-002]

Right Camera Matrix: [1.0294741711514341e+003, 0., 6.5282835377635274e+002, 0., 1.0285664286763802e+003, 5.2393079228919225e+002, 0., 0., 1..]

Right Camera D: [-3.8333204303394736e-001, 2.7030722876029428e-001, 1.2659691660140408e-004, -6.2717216712805863e-004, -1.4868158456944205e-001]

Now I do calibration together. I call stereoCalibrate like this:

double rms = stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    Size(width,height), R, T, E, F,
                    cv::CALIB_USE_INTRINSIC_GUESS + cv::CALIB_FIX_INTRINSIC, TermCriteria(cv::TermCriteria::COUNT+cv::TermCriteria::EPS, 100, 1e-6));

RMS error: 0.716278 Average reprojection error: 0.932500

I get quite sane results. R is almost identity, showing that the cameras are very parallel:

9.9825297964692494e-001, -5.6272429338183161e-002, -1.8011172155504525e-002,
5.6492686748686502e-002,  9.9833134239913945e-001,   1.1962739128096707e-002,
1.7307945283909901e-002, -1.2959339485918969e-002,   9.9976621794804499e-001

And T is also showing correct results:

-2.8507174779791137e+002, -5.9310534218857187e+000,  9.6822612661982728e+000

It shows that the distance between cameras is 285mm, which is as precise as I can physically measure. So it seems all the inputs are correct. Now I call stereoRectify() like so:

    stereoRectify(cameraMatrix[0], distCoeffs[0],
          cameraMatrix[1], distCoeffs[1],
          Size(width,height), R, T, R1, R2, P1, P2, Q,
          cv::CALIB_ZERO_DISPARITY, 1.0, Size(width,height), &validRoi[0], &validRoi[1]);


9.9804912529642753e-001, -3.5033737187826153e-002, -5.1677662037772229e-002,
3.5384693276990765e-002,  9.9935639816774224e-001,   5.8917673825730115e-003,
5.1437991569745038e-002, -7.7088715031055979e-003,  9.9864643709544143e-001


9.9920772192907659e-001, 2.0788992329479110e-002, -3.3937386982939709e-002,
-2.1019454438531062e-002, 9.9975827258716765e-001, -6.4481725030493993e-003,
3.3795132377480969e-002, 7.1564091168283793e-003,  9.9940315930866519e-001


-4.7663940994070654e+003, 0., 4.8584126815795898e+003, 0., 
0.,  -4.7663940994070654e+003, 3.7089497299194336e+003, 0., 
0., 0., 1.,  0.


-4.7663940994070654e+003, 0., 4.8584126815795898e+003,  1.3598416693461742e+006,
0., -4.7663940994070654e+003, 3.7089497299194336e+003, 0., 
0., 0., 1., 0.


1., 0., 0., -4.8584126815795898e+003,
0., 1., 0., -3.7089497299194336e+003, 
0., 0., 0., -4.7663940994070654e+003,
0., 0., 3.5051096071344805e-003, 0.

Q also seems correct (1/B = 1/235 = 3.505e-003). But now when I use the remap function I get this:

image description

I use OpenCV3.0. Maybe this was fixed in 3.1? I have calibrated stereo cameras with OpenCV before, and while it was painful it wasn't impossible. I have calibrated numerous times and I always get the same result. When I set Alpha to 0.0 in stereoRectify I get image like this:

image description

But it is wrong as well ... (more)

edit retag flag offensive close merge delete


I think you are confusing undistorted with rectified. Undistorted means straight lines are straight. Rectified means they are scanline aligned. Try holding your calibration target closer to the cameras and such that the entire field of view is eventually covered by the samples. And if the results aren't good, do it again.

Der Luftmensch gravatar imageDer Luftmensch ( 2016-04-18 20:47:11 -0500 )edit

Yes, sorry, I meant undistorted when I said rectified at top image. I have calibrated numerous times and it doesn't matter. And the numbers I get suggest the calibration is successful. I get good undistorted images after intrinsic calibration (as can be seen in the second image) and I get good T and R values after extrinsic calibration, but the rectification result is extremely incorrect. I do have a confusion that OpenCV docs have never clarified - when a function asks for distortion coefficients D, then does it internally use this to distort the input, or undistort it? Like in solvePnP, do I give distorted points (so I do "raw" points as input) and CV uses D to compensate distortion or I give undistorted points (so I do point extraction from a remap()'d image) and CV uses D compensate?

HG gravatar imageHG ( 2016-04-19 02:38:18 -0500 )edit

I think your issue is identical to mineprobem. it seems that your distortion model is not well set. in my problem it is not nessecery to use k6 in distortion model. your distortion model use k3 and k4 , i think it is not appropriate.

Roby gravatar imageRoby ( 2016-04-19 02:55:48 -0500 )edit

With solvePnP and similar functions you need to keep track of which image you are using, the raw image or the remapped image. If it is remapped then you should have all zeros for distortion parameters and you should be using the camera matrix of the remapped image.

Der Luftmensch gravatar imageDer Luftmensch ( 2016-04-19 07:25:20 -0500 )edit

if your chessboards do not vary a lot(camera view angle to the chessboard's principle axis) in each image, you won't get the good intrinsic params( LM estimation won't converge to good result. all the input are similar and reduce the rank of Matrix). eventually after remaping image you get worse result.

Roby gravatar imageRoby ( 2016-04-24 23:14:37 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2017-01-20 06:57:32 -0500

HG gravatar image

I fixed it by changing the stereoCalibrate flags. For example using: cv::CALIB_RATIONAL_MODEL + cv::CALIB_FIX_K3 + cv::CALIB_FIX_K4 + cv::CALIB_FIX_K5 + cv::CALIB_FIX_K6

Another thing that gave larger RMS error, but at least a correct remap was calibrating intrinsics and extrinsics together. So no cv::CALIB_USE_INTRINSIC_GUESS + cv::CALIB_FIX_INTRINSIC

For now there is nothing else I found to be working.

edit flag offensive delete link more

Question Tools

1 follower


Asked: 2016-04-18 09:01:01 -0500

Seen: 3,072 times

Last updated: Jan 20 '17