Stereo Rectify doesn't rectify even with correct M and D
I cannot figure out a problem with stereoRectify or initUndistortRectifyMap. It might be related to this question but the user didn't post enough information for me to determine it. I do stereo calibration in two passes, because then we can split the intrinsic and extrinsic calibration, but it doesn't seem to work. These are inputs:
You can see radial lens distortion that needs to be corrected. So after each camera calibration I get a pretty good rectified images:
Left Camera Matrix: [1.0316489177476062e+003, 0., 6.5285947709947743e+002, 0., 1.0306889512875423e+003, 5.2195206055917618e+002, 0., 0., 1.]
Left Camera D: [-3.6474261084702730e-001, 1.7160954305339343e-001, 1.3998315284130826e-004, -5.7337307704477671e-005, -4.3398439779963091e-002]
Right Camera Matrix: [1.0294741711514341e+003, 0., 6.5282835377635274e+002, 0., 1.0285664286763802e+003, 5.2393079228919225e+002, 0., 0., 1..]
Right Camera D: [-3.8333204303394736e-001, 2.7030722876029428e-001, 1.2659691660140408e-004, -6.2717216712805863e-004, -1.4868158456944205e-001]
Now I do calibration together. I call stereoCalibrate like this:
double rms = stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
cameraMatrix[0], distCoeffs[0],
cameraMatrix[1], distCoeffs[1],
Size(width,height), R, T, E, F,
cv::CALIB_USE_INTRINSIC_GUESS + cv::CALIB_FIX_INTRINSIC, TermCriteria(cv::TermCriteria::COUNT+cv::TermCriteria::EPS, 100, 1e-6));
RMS error: 0.716278 Average reprojection error: 0.932500
I get quite sane results. R is almost identity, showing that the cameras are very parallel:
9.9825297964692494e-001, -5.6272429338183161e-002, -1.8011172155504525e-002,
5.6492686748686502e-002, 9.9833134239913945e-001, 1.1962739128096707e-002,
1.7307945283909901e-002, -1.2959339485918969e-002, 9.9976621794804499e-001
And T is also showing correct results:
-2.8507174779791137e+002, -5.9310534218857187e+000, 9.6822612661982728e+000
It shows that the distance between cameras is 285mm, which is as precise as I can physically measure. So it seems all the inputs are correct. Now I call stereoRectify() like so:
stereoRectify(cameraMatrix[0], distCoeffs[0],
cameraMatrix[1], distCoeffs[1],
Size(width,height), R, T, R1, R2, P1, P2, Q,
cv::CALIB_ZERO_DISPARITY, 1.0, Size(width,height), &validRoi[0], &validRoi[1]);
R1:
9.9804912529642753e-001, -3.5033737187826153e-002, -5.1677662037772229e-002,
3.5384693276990765e-002, 9.9935639816774224e-001, 5.8917673825730115e-003,
5.1437991569745038e-002, -7.7088715031055979e-003, 9.9864643709544143e-001
R2:
9.9920772192907659e-001, 2.0788992329479110e-002, -3.3937386982939709e-002,
-2.1019454438531062e-002, 9.9975827258716765e-001, -6.4481725030493993e-003,
3.3795132377480969e-002, 7.1564091168283793e-003, 9.9940315930866519e-001
P1:
-4.7663940994070654e+003, 0., 4.8584126815795898e+003, 0.,
0., -4.7663940994070654e+003, 3.7089497299194336e+003, 0.,
0., 0., 1., 0.
P2:
-4.7663940994070654e+003, 0., 4.8584126815795898e+003, 1.3598416693461742e+006,
0., -4.7663940994070654e+003, 3.7089497299194336e+003, 0.,
0., 0., 1., 0.
Q:
1., 0., 0., -4.8584126815795898e+003,
0., 1., 0., -3.7089497299194336e+003,
0., 0., 0., -4.7663940994070654e+003,
0., 0., 3.5051096071344805e-003, 0.
Q also seems correct (1/B = 1/235 = 3.505e-003). But now when I use the remap function I get this:
I use OpenCV3.0. Maybe this was fixed in 3.1? I have calibrated stereo cameras with OpenCV before, and while it was painful it wasn't impossible. I have calibrated numerous times and I always get the same result. When I set Alpha to 0.0 in stereoRectify I get image like this:
But it is wrong as well ...
I think you are confusing undistorted with rectified. Undistorted means straight lines are straight. Rectified means they are scanline aligned. Try holding your calibration target closer to the cameras and such that the entire field of view is eventually covered by the samples. And if the results aren't good, do it again.
Yes, sorry, I meant undistorted when I said rectified at top image. I have calibrated numerous times and it doesn't matter. And the numbers I get suggest the calibration is successful. I get good undistorted images after intrinsic calibration (as can be seen in the second image) and I get good T and R values after extrinsic calibration, but the rectification result is extremely incorrect. I do have a confusion that OpenCV docs have never clarified - when a function asks for distortion coefficients D, then does it internally use this to distort the input, or undistort it? Like in solvePnP, do I give distorted points (so I do "raw" points as input) and CV uses D to compensate distortion or I give undistorted points (so I do point extraction from a remap()'d image) and CV uses D compensate?
I think your issue is identical to mineprobem. it seems that your distortion model is not well set. in my problem it is not nessecery to use k6 in distortion model. your distortion model use k3 and k4 , i think it is not appropriate.
With solvePnP and similar functions you need to keep track of which image you are using, the raw image or the remapped image. If it is remapped then you should have all zeros for distortion parameters and you should be using the camera matrix of the remapped image.
if your chessboards do not vary a lot(camera view angle to the chessboard's principle axis) in each image, you won't get the good intrinsic params( LM estimation won't converge to good result. all the input are similar and reduce the rank of Matrix). eventually after remaping image you get worse result.