Greg Kramida's profile - activity

I am trying to properly calibrate a multi-stereo system. For this, I need to first calibrate camera intrinsic parameters, then the stereo extrinsic parameters for each pair, then the extrinsics between each stereo pair.

The cameras that I have don't have very good lenses for this. These are Xiaomi Yi sport-action cameras, each one currently has 155-degree lens (alternative lenses ordered, on the way). More on that in the appendix.

I'm using the calibrateCamera function to calibrate intrinsics. I use the rational model with 6 radial coefficients from the latest OpenCV build (see 'Detailed Description' here). Neither tangential coefficients nor the thin-prism coefficients seem to improve the reprojection error significantly, so I don't use them. I calibrate each camera on around 200 images, and I'm getting re-projection error within 0.35 to 0.40 pixels. I have no idea as of yet what gets rejected by RANSAC and what doesn't, but I use alternative sets of images to verify each calibration, which yields about the same reprojection error (the intrinsics & distortion coefficients being fixed during testing).

I start seeing a problem at the stereo stage. I lock the intrinsics (incl. distortion coefficients) during stereo calibration. I get reprojection errors in the range of 2.5-4.0 pixels. Again, I have no idea what's being rejected by RANSAC as of yet.

Here are sample left & right images (with distortion), scaled down:

Here is an overlay of the same images after stereo-rectification (blue=left, orange=right), with horizontal lines depicting where epipolar lines are supposed to be:

Here is the resulting disparity using the StereoSGBM algorithm (tuning its parameters in any way doesn't yield any dramatic improvements over this):

Conclusion: stereo calibration is not accurate enough to yield appropriate depth-from-stereo. What can I do to fix that?

[Edit] I have been long under suspicion that high re-projection error during calibration results from cameras not being perfectly synced. Currently, I can only synchronize them in post-processing, which means of sync accuracy within 1/2 frame. The cameras run at 60 fps. I have yet to verify this by shooting a still scene, but it seems like epipolar "lines" don't come out horizontal & straight for still objects either....

Appendix (on lenses)

Cameras' manufacturer claims the lenses are 155 "fish-eye" lenses, where DFOV = 155 degrees and focal length is 2.73 mm +-5%. The sensor's diagonal is 7.77 mm. If you plug the focal length & diagonal into the most basic formula for calculating DFOV, DFOV = 2*arctan( sensor_diagonal/(2f)), you get about 110 degrees. Hence I feel the lens is a compound lens with some weird configuration, and hence does not adhere to the formula (most camera lenses are compound nowadays...) Manually I measured the HFOV to be about 120 degrees, which does not work out math-wise with 155 degrees either. It may be that the camera is not using the width of the entire sensor in video mode, but all of this is guesswork.

This can be caused by the way the CMake is set up on your system.

It turns out if CMake does not support the right download protocol, it will download an empty file. When it tries to match the hash of that to what it expects, it obviously fails. Even if the hash is altered to the hash of the empty file, CMake will still fail reporting "protocol not supported." The correct way is to rebuild CMake with local curl library support on your platform, see this SO answer here. This is the proper long-term solution, since the ICV file is sometimes updated, along with the expected hash.

The problem had nothing to do with the displayed warning. I was trying to write single-channel images, while the VideoWriter was expecting three-channel color images (default value of isColor, the 5-th argument to VideoWriter's constructor, is "true"). The solution was setting isColor to false.

I've seen this post here, but I think I'm having a different problem with the said combo. Code:

cv::VideoWriter writer(output_path.string(), CV_FOURCC('X','2','6','4'), 60, frame_size);

Output:

OpenCV: FFMPEG: tag 0x34363258/'X264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'

The video produced is unreadable. The FFmpeg I have is packaged by ubuntu maintainers, and it lists it was configured with "--enable-libx264". I made a repost here on SO. Any idea how I can make this combination work? Is it an OpenCV bug, FFmpeg bug, or neither?

If you post this as an answer, I'll mark it as accepted.

If I find a bug in the master branch, do I specify the target version as 3.0 beta or 3.0? What is the general policy about this (couldn't find any guidelines)?