# Stereo calibration fails

I am trying to properly calibrate a multi-stereo system. For this, I need to first calibrate camera intrinsic parameters, then the stereo extrinsic parameters for each pair, then the extrinsics between each stereo pair.

The cameras that I have don't have very good lenses for this. These are Xiaomi Yi sport-action cameras, each one currently has 155-degree lens (alternative lenses ordered, on the way). More on that in the appendix.

I'm using the calibrateCamera function to calibrate intrinsics. I use the rational model with 6 radial coefficients from the latest OpenCV build (see 'Detailed Description' here). Neither tangential coefficients nor the thin-prism coefficients seem to improve the reprojection error significantly, so I don't use them. I calibrate each camera on around 200 images, and I'm getting re-projection error within 0.35 to 0.40 pixels. I have no idea as of yet what gets rejected by RANSAC and what doesn't, but I use alternative sets of images to verify each calibration, which yields about the same reprojection error (the intrinsics & distortion coefficients being fixed during testing).

I start seeing a problem at the stereo stage. I lock the intrinsics (incl. distortion coefficients) during stereo calibration. I get reprojection errors in the range of 2.5-4.0 pixels. Again, I have no idea what's being rejected by RANSAC as of yet.

Here are sample left & right images (with distortion), scaled down:

Here is an overlay of the same images after stereo-rectification (blue=left, orange=right), with horizontal lines depicting where epipolar lines are supposed to be:

Here is the resulting disparity using the StereoSGBM algorithm (tuning its parameters in any way doesn't yield any dramatic improvements over this):

Conclusion: stereo calibration is not accurate enough to yield appropriate depth-from-stereo. What can I do to fix that?

 I have been long under suspicion that high re-projection error during calibration results from cameras not being perfectly synced. Currently, I can only synchronize them in post-processing, which means of sync accuracy within 1/2 frame. The cameras run at 60 fps. I have yet to verify this by shooting a still scene, but it seems like epipolar "lines" don't come out horizontal & straight for still objects either....

Appendix (on lenses)

Cameras' manufacturer claims the lenses are 155 "fish-eye" lenses, where DFOV = 155 degrees and focal length is 2.73 mm +-5%. The sensor's diagonal is 7.77 mm. If you plug the focal length & diagonal into the most basic formula for calculating DFOV, DFOV = 2*arctan( sensor_diagonal/(2f)), you get about 110 degrees. Hence I feel the lens is a compound lens with some weird configuration, and hence does not adhere to the formula (most camera lenses are compound nowadays...) Manually I measured the HFOV to be about 120 degrees, which does not work out math-wise with 155 degrees either. It may be that the camera is not using the width of the entire sensor in video mode, but all of this is guesswork.

edit retag close merge delete

Sort by » oldest newest most voted

Hi, wich method do you use to connect the camera with opencv, because streaming over rtsp has a lot of lag. Thanks

more

( 2016-12-09 16:21:34 -0500 )edit

Official site

GitHub

Wiki

Documentation