Revision history [back]

I see this is an old thread, but this problem set me back a couple of months, so hopefully this helps somebody. I used the OpenCV chessboard calibration technique but had the same problem. Everything before the stereo calibration was easy--I just followed the steps in this book: (http://www.it-ebooks.info/book/3145). I'll assume you already have the left and right camera matricies, left and right distortion coefficients, and an appropriate set of chessboard corners (image points) for the stereo calibration.

First, I used initUndistortRectifyMap() create pixel mappings that remove the distortion on the individual camera outputs:

// initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2) 
initUndistortRectifyMap(left_cameraMatrix, left_distCoeffs, empty, left_cameraMatrix, left_image_size, 32FC1,   monomap_l1,monomap_l2);
initUndistortRectifyMap(right_cameraMatrix, right_distCoeffs, empty, right_cameraMatrix, right_image_size, CV_32FC1, monomap_r1, monomap_r2);

Then I applied the undistortPoints() to my image points.

// undistortPoints( InputArray src, OutputArray dst, InputArray cameraMatrix, InputArray distCoeffs, InputArray R=noArray(), InputArray P=noArray())
undistortPoints( left_image_points, left_image_points, l_cameraMatrix, l_distCoeffs, empty, l_cameraMatrix);
undistortPoints( right_image_points, right_image_points, r_cameraMatrix, r_distCoeffs, empty, r_cameraMatrix);

Now if I remap() the images using the "monomaps" and drawChessboardCorners() on those images, the chessboard corners line up with the distortion-corrected images.

Last step: use the distortion-corrected chessboard corners to do stereoRectify():

Mat R, T, E, F; //stereo calibration information
Mat zerodistortion = Mat::zeros(1,5,CV_32FC1);  //empty distortion matrix
stereoCalibrate(object_points, left_image_points, right_image_points, left_cameraMatrix, zerodistortion, right_cameraMatrix, zerodistortion, left_image_size, R, T, E, F);

This worked! I was at a conference recently where I spoke with a guy teaching in Spain and he said he was doing the same exact thing with his cameras.

Now, to see the rectified outputs, you have to create the stereo rectification maps:

// initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2) 
initUndistortRectifyMap(l_cameraMatrix, zerodistortion, Rl, Pl, image_size, CV_32FC1, map_l1, map_l2);
initUndistortRectifyMap(r_cameraMatrix, zerodistortion, Rr, Pr, image_size, CV_32FC1, map_r1, map_r2);

Then (I was too lazy to fix this), to see the rectified output, you have to remap your input images twice: once for monocular undistortion and once for stereo rectification:

// Remap images to remove monocular distortions
remap(left, left_undist, monomap_l1, monomap_l2, INTER_LINEAR);
remap(right, right_undist, monomap_r1, monomap_r2, INTER_LINEAR);

// Remap images to rectify
remap(left_undist, left_rect, map_l1, map_l2, INTER_LINEAR);
remap(right_undist, right_rect, map_r1, map_r2, INTER_LINEAR);

Long story short, stereoCalibrate() couldn't handle the extreme distortion, but it you can remove that distortion before calling stereoCalibration().