rectify fisheye stereo setup

edit

Hello,

I have had the same problem, also working with ultra-wide angle lenses. It seems the problem is the StereoRectify function which may calculate some invalid inner and outer rectangles as a basis for centering and scaling the rectified images. I think the reason for this is that it tries to undistort the extreme boundaries of the original image, and this does not always go well, especially in the corners of images when you have a lot of radial distortion. My solution to the problem has been to rewrite StereoRectify so that it only calculates the inner rectangle and only on the basis of four points on the image borders: North, south, east and west. That way I get a stable behaviour. The price to pay is that the free scaling parameter alpha will not work as specified, but for my purposes it is enough to perform some scaling relative to the inner rectangle.

I have also run into this issue with my setup, although my lens distortion isn't even that heavy. I also had the problem with the calculation of the inner and outer rectangles for determining the new camera matrices. The way OpenCV calculates these inner and outer rectangles is based on the assumption that the radial distortion is monotonic, so they assume that the edges of the input images should contain the most heavily distorted points and only sample along the image border for determining the most extreme distortion.

This however, is not necessarily the case, when you use the higher order radial model (with k_2 and k_3) or the "rational" model it is possible that the camera calibration converges to a set of distortion parameters that are not monotonic, and so these assumptions are invalid and resulting inner and outer rectangles make no sense, as can be seen in the example image by Thomas. This has happened to me on occasion, where the outer third of my image has a lower distortion factor than the inner two thirds.

I think the only real solution to this would be to modify the camera calibration optimization objective function to include constraints so that the distortion factors are monotonic throughout the image.

I see this is an old thread, but this problem set me back a couple of months, so hopefully this helps somebody. I used the OpenCV chessboard calibration technique but had the same problem. Everything before the stereo calibration was easy--I just followed the steps in this book: (http://www.it-ebooks.info/book/3145). I'll assume you already have the left and right camera matricies, left and right distortion coefficients, and an appropriate set of chessboard corners (image points) for the stereo calibration.

First, I used initUndistortRectifyMap() create pixel mappings that remove the distortion on the individual camera outputs:

// initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2) 
initUndistortRectifyMap(left_cameraMatrix, left_distCoeffs, empty, left_cameraMatrix, left_image_size, 32FC1,   monomap_l1,monomap_l2);
initUndistortRectifyMap(right_cameraMatrix, right_distCoeffs, empty, right_cameraMatrix, right_image_size, CV_32FC1, monomap_r1, monomap_r2);

Then I applied the undistortPoints() to my image points.

// undistortPoints( InputArray src, OutputArray dst, InputArray cameraMatrix, InputArray distCoeffs, InputArray R=noArray(), InputArray P=noArray())
undistortPoints( left_image_points, left_image_points, l_cameraMatrix, l_distCoeffs, empty, l_cameraMatrix);
undistortPoints( right_image_points, right_image_points, r_cameraMatrix, r_distCoeffs, empty, r_cameraMatrix);

Now if I remap() the images using the "monomaps" and drawChessboardCorners() on those images, the chessboard corners line up with the distortion-corrected images.

Last step: use the distortion-corrected chessboard corners to do stereoRectify():

Mat R, T, E, F; //stereo calibration information
Mat zerodistortion = Mat::zeros(1,5,CV_32FC1);  //empty distortion matrix
stereoCalibrate(object_points, left_image_points, right_image_points, left_cameraMatrix, zerodistortion, right_cameraMatrix, zerodistortion, left_image_size, R, T, E, F);

This worked! I was at a conference recently where I spoke with a guy teaching in Spain and he said he was doing the same exact thing with his cameras.

Now, to see the rectified outputs, you have to create the stereo rectification maps:

// initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2) 
initUndistortRectifyMap(l_cameraMatrix, zerodistortion, Rl, Pl, image_size, CV_32FC1, map_l1, map_l2);
initUndistortRectifyMap(r_cameraMatrix, zerodistortion, Rr, Pr, image_size, CV_32FC1, map_r1, map_r2);

Then (I was too lazy to fix this), to see the rectified output, you have to remap your input images twice: once for monocular undistortion and once for stereo rectification:

// Remap images to remove monocular distortions
remap(left, left_undist, monomap_l1, monomap_l2, INTER_LINEAR);
remap(right, right_undist, monomap_r1, monomap_r2, INTER_LINEAR);

// Remap images to rectify
remap(left_undist, left_rect, map_l1, map_l2, INTER_LINEAR);
remap(right_undist, right_rect, map_r1, map_r2, INTER_LINEAR);

Long story short, stereoCalibrate() couldn't handle the extreme distortion, but it you can remove that distortion before calling stereoCalibration().

Since you are using fisheye camera instead of perspective one, I suggest you using cv::fisheye::stereoRectify instead of cv::stereoRectify.