OpenCV fisheye calibration fails
I am trying to calibrate a fisheye camera with 120° angle of view. It works quite well when using the classical calibration function cv::calibrateCamera() and the rational model. However, the rectified image is still a bit distorted (see images below).
I then tried to use the fisheye calibration function cv::fisheye::calibrate() but it returns me a NaN camera matrix or the following error when i add cv::fisheye::CALIB_CHECK_COND :
OpenCV Error: Assertion failed (svd.w.at<double>(0) / svd.w.at<double>((int)svd.w.total() - 1) < thresh_cond) in CalibrateExtrinsics, file /build/opencv-L2vuMj/opencv-3.2.0+dfsg/modules/calib3d/src/fisheye.cpp, line 1427 terminate called after throwing an instance of 'cv::Exception'
what(): /build/opencv-L2vuMj/opencv-3.2.0+dfsg/modules/calib3d/src/fisheye.cpp:1427: error: (-215) svd.w.at<double>(0) / svd.w.at<double>((int)svd.w.total() - 1) < thresh_cond in function CalibrateExtrinsics
I am using exactly the same detected chessboard points in the classical function and in the fisheye function. And it does not change anything when I use more or less images. Did someone has a clue about why it happens ? I already checked similar posts but I wasn't able to find an appropriate solution.
Thanks
I have had problems with the fisheye calibration with OpenCV 3.2. As I recall my issue was that it would run "successfully" but with high reprojection error (terrible / nonsensical results), and then other times it would converge with low error.
I found that using the normal calibration with the CV_CALIB_RATIONAL_MODEL flag, I can get excellent results (0.15 pixel typical). The distortion is similar (possibly more significant) to what you have, but the image quality I am getting is much higher. You seem to have significant image noise / contrast and focus issues in the image you shared - can this be improved with camera settings or different lighting?
I would also suggest using the CHARUCO style calibration with smaller squares (more points).
Thank you for your answer swebb_denver ! I managed to run "successfully" the fisheye calibration by using only a few images but the camera matrix is totally aberrant. Do you think that it might be better if I use the latest version of OpenCV ?
When using the normal calibration with the rational model, my mean reprojection error is around 0.45 pixels, which I think is quite good. But I am not totally satisfied with the undistorted results. I do have a lot of noise with this sensor but I don't need to improve the image quality as the chessboard corners detection is accurate and I have a good image coverage (i.e. detected corners in every areas of the image, even the very distorted ones).
A few ideas: If your calibration target and camera can be held in a fixed position, take multiple images and compute an average image - this will reduce image noise. You will still need multiple orientations for the target - I recommend a minimum of 8.
One problem with the chessboard calibration target is that you have to see the entire pattern in the image, which makes it difficult to get calibration points near the corners. This results in a distortion model that fits the central part of the image well, but doesn't necessarily fit the edges / corners of the image. By using a CHARUCO calibration target (you will have to build the contrib modules to get this) you can use partial images, and therefore get calibration points near the corners.
One more thing is that with high distortion you might run into issues where the calibration either doesn't detect the corners that are further out (higher distortion areas) because it can't correctly identify the Aruco markers (particularly with noisy image or high density target (small aruco markers)). If you run into this issue, you can do an iterative approach where you use the initial calibration results to predict where undetected chessboard corners "should" appear in the image. You can then add those corners, and calibrate again, and then repeat the process until you have found as many corners as you can.
You may not need to do this, but I have found that I get extremely good results across the full image (very high distortion lens) by doing this (using RATIONAL model)
I had thought of using ChArUco calibration target but I can't print it correctly using a consumer grade printer as my camera is designed for very low depth of field and small targets. For example, the calibration target I am using is a 7 by 7 chessboard with squares of side 1mm.
I am using the rational model of distorsion, which gives me the best results for rectified images, but the result is far from perfect (as you can see on the images from the original post, which are not used for calibration but only used for visualisation of the distortion and rectification).
Thank you for your ideas to improve the corners detection. The iterative approach could be a nice idea to improve the number of detected corners or the precision of the detected corners.