Ask Your Question

Stereo Matching/Calibration Help

asked 2013-10-03 20:41:48 -0500

BumblebeeUser89 gravatar image

updated 2013-10-04 17:21:21 -0500


I am using the Bumblebee XB3 Stereo Camera and it has 3 lenses. I've spent about three weeks reading forums, tutorials, the Learning OpenCV book and the actual OpenCV documentation on using the stereo calibration and stereo matching functionality. In summary, my issue is that I have a good disparity map generated but very poor point-clouds, that seem skewed/squished and are not representative of the actual scene.

What I have done so far:

Used the OpenCV stereo_calibration and stereo_matching examples to:

1) Calibrate my stereo camera using chess board images
Raw Scene Images: image description
2) Rectified the raw images obtained from the camera using the matrices after camera calibration
image description
3) Generated a disparity image from the rectified images using Stereo Matching (SGBM)
image description
4) Projected these disparities to a 3D Point Cloud
image description
image description

What I have done so far as elimination towards my problem:

  • I have tried the 1st and 2nd images, then the 2nd and 3rd lenses and finally the 1st and 2nd.
  • I've re-run calibration of my chess-board captures by varying the distance (closer/farther away)
  • I have used over 20 stereo pairs for the calibration
  • Varying Chessboard size used: I have used a 9x6 Chessboard image for calibration and now switched to using a 8x5 one instead
  • I've tried using the Block Matching as well as SGBM variants and get
    relatively similar results. Getting
    better results with SGBM so far.
  • I've varied the disparity ranges, changed the SAD Window size etc. with little improvement

What I suspect the problem is:

My disparity image looks relatively acceptable, but the next step is to go to 3D Point cloud using the Q matrix. I suspect, I am not calibrating the cameras correctly to generate the right Q matrix. Unfortunately, I've hit the wall in terms of thinking what else I can do to get a better Q matrix. Can someone please suggest ways ahead?

The other thing that I think might be problematic is the assumptions I am making when using the cv::stereoCalibrate function. For the moment, I individually calibrate each camera to get the camera and distortion (cameraMatrix[0], distCoeffs[0] and cameraMatrix[1], distCoeffs[1]) matrices so it makes the complexity for the stereoCalibrate function a little easier.

stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    imageSize, R, T, E, F,
                    TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                    //CV_CALIB_FIX_ASPECT_RATIO +
                    //CV_CALIB_ZERO_TANGENT_DIST +
                    //CV_CALIB_SAME_FOCAL_LENGTH +
                    //CV_CALIB_FIX_K3 + CV_CALIB_FIX_K4 + CV_CALIB_FIX_K5

Additionally, I think it might be useful to mention how I am going from disparity to point cloud. I am using OpenCV's cv::reprojectImageTo3D and then writing the data to a PCL Point cloud structure. Here is the relevant code:

cv::reprojectImageTo3D( imgDisparity16S, reconstructed3D, Q, false, CV_32F);
  for (int i = 0; i < reconstructed3D.rows; i++)
    for (int j = 0; j < reconstructed3D.cols; j++)
        cv::Point3f cvPoint =<cv::Point3f>(i, j);  
            //Filling in a PCL structure
            pcl::PointXYZRGB point;
            point.x = cvPoint.x;
            point.y = cvPoint.y;
            point.z = cvPoint ...
edit retag flag offensive close merge delete


Have you found something ? Because I'm having the same issue right now. I'm trying to get a better disparity map too, but it really difficult.

Alexandre Bizeau gravatar imageAlexandre Bizeau ( 2013-11-19 08:51:20 -0500 )edit

I'm interested in this answer as well. I'm trying to compare OpenCV stereo processing to what I get using the Point Grey (PG) libraries. I am trying to use images rectified by the PG software, and then generate the Q matrix using stereoRectify and parameters that I get from the PG calibration data. The resulting point cloud is curved improperly, and fails to generate point clouds at a distance. Are there any guidelines for comparing OpenCV and PG results?

dconner gravatar imagedconner ( 2014-02-19 14:13:28 -0500 )edit

I met this problem, too. Have you figure it out? Would you mind if you could share some new progress of it? thanks a lot!

bennygato gravatar imagebennygato ( 2014-07-10 11:29:27 -0500 )edit

We are also met the same problem -- Can any one solved this recently

utp gravatar imageutp ( 2017-02-21 00:08:42 -0500 )edit

same here......exact same problem...has anyone solved this? This is so frustrating =(

waterchan gravatar imagewaterchan ( 2019-01-02 08:37:58 -0500 )edit

maybe @Eduardo could share his remarks

sturkmen gravatar imagesturkmen ( 2019-01-02 08:58:13 -0500 )edit


Sorry, I don't have much experience on stereo matching topic.

My advices would be to:

  • try to tweak the stereobm parameters to have a better disparity result (the quality of the depth map can vary drastically with wrong parameters, see for instance)
  • post filter the disparity map, see this tutorial
  • you can also have a look at this tutorial for general information on depth from stereo images

You can also try to remove outliers for points having non coherent depth.

Eduardo gravatar imageEduardo ( 2019-01-05 22:33:28 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2014-02-18 15:57:20 -0500

The disparity map looks like it contains outliers, which can place the reprojected point far out of the scene. OpenCV has a function, filterSpeckles, to clean the disparity map of such outliers. You'll have to play with the parameters, especially maxSpeckleSize. In general, the larger your block size used in the stereo matching stage, the larger maxSpeckleSize needs to be.

edit flag offensive delete link more

Question Tools



Asked: 2013-10-03 20:41:48 -0500

Seen: 2,754 times

Last updated: Feb 18 '14