Camera calibration: Can it be "over-undistorted"?
My earlier camera calibration attempts went very smooth and gave good results. Now I wanted to get serious with it and calibrate each of my cameras (I'm using 3 of the same type in my system), each at their specific focus.
What I did before:
- used a 7x9 (inner corners) squares chessboard pattern with square size 10 mm
- printed on paper
- put on the floor to be hopefully very flat
- took ~20 pictures from various angles, trying to have the pattern appear in all areas of the image
- selected 12 best pictures (with respect to focus, movement, light)
- calibrated and created camera matrix and distortion vector
The result was good, the RMS reprojection error value is 0.425 (i believed that meant "good'), and the undistorted images looked nice and undistored (straight lines were straight).
Now, I got my chessboard pattern printed on aluminium di-bond using an online foto printing service, hoping to optimize the procedure. I used a 20 mm 7x9 pattern, but because I wanted to make it "beautiful" and made the outer squares end round (as I had seen in some example chessboard patterns), my high-resolution camera only recognizes 5x7 patterns in it (my lower resolution cameras somehow recognize 7x9 easily). Also, because of how the foto printing service works, my square size is not 20 mm but 17.08 mm - which I don't care much about.
Anyway, I put my perfectly straight 17.x mm 5x7 squares aluminium chesspattern in absolute perfect surround LED lighting and did my foto session. I thought "more is more" and took many pictures, from which I selected 20.
The result is devestating!
The RMS value is from 0.95 to 1.15 for my 3 cameras (maybe not too bad?). But the undistortion for 2 of the 3 results creates heavily undistorted images (of size 2048x1536), in which the actual undistorted part is concentrated in around 300x200 pixels somewhere in the bottom right or top right corner. The undistortion looks about right, but obviously something is wrong. This is what I would perhaps call "over-undistorted".
Just by luck, I selected only 12 images to calculate the camera matrix and distortion vector, and received a fairly good result. But there the RMS reprojection error was ~4.5 (which I believed was worse than the 0.4 which I had with my first setup).
I went ahead and selected different combinations of chessboard recordings, trying to find if there was one that destroyed the result. But there wasn't - any combination would lead to more or less over-undistorted results. Something in between with 17 images:
So obviously I am very much confused with the results I'm getting, especially compared with the first results I had, where everything was pretty easy and straight forward. I'm always reading that we need at least 10 images to get a good calibration. However, in my case, I might feel like the higher number gets me into trouble. This is not really true ...
Usually when camera it's calibrated A LOT of images are taken, In my setup I took more than 200 images and I carefully select the best ones from different viewpoint. Some borders of the chessboard look sharp on the images, you need to correct that. I suggest you to switch to b&w images if your camera allow that and use some script to better optimize the images (I personally use Photoshop). In the pic you posted the "white" square are not that white and in the black square, due to the debayer proceudre (probably) have the contours of the square are not well defined. Also acquire images from different angulation, the chessboard should be visible entirely in most of the calibration procedure. Is not always that easy perform a good calibration, it take usually some (boring) time..
are the un-distortion algorithms supposed to work for a fish-eye camera with its huge non-linear distortions?
@HYPEREGO thanks a lot for your insights! I did take my pictures from different angles, orientations, rotations, positions. I guess I will just give it a shot and try more. Before, I seemed to be very lucky with different cameras, where 10 images were just perfect. What I don't understand is, why the image quality is so much concerning. As long as the chessboard corners are detected quite accurate, shouldn't I be happy? Even if the edges are blurry sometimes, the corner detection looks spot on. Does it really have to be that super accurate? Especially when I take pictures from different angles of the chessboard, I will surely have focus blur. @andrei186 Is my camera that fisheye'y? ~110deg is the diagonal fov I believe. Before though, I received good cal results easily.
Quite accurate unfortunately is not accurate. Most of the algorithms out there for offline calibration rely on the Zhang work, that basically detect the corner of the chessboard to estimate the 3d position of the points (obviously I'm shortcutting many steps). Using an optimization algorithm the parameters are then estimated. Try to get more images and choose carefully the one that seems to be the better and not sharp. Take pictures from different inclination and point of view.. You can try to calibrate the camera even with the interactive mode mode without taking the single shots. Personally your camera doesn't look as a fisheye, but is just my opinion.
" Before though, I received good cal results easily." - Do I understand you correctly that your problems started with your new shiny chessboard plate, all other things being the same? If so, your program and camera are fine but the plate is wrong. I would go back to the old one and try to see how it is different form the new. Perhaps those "beautiful outer squares end round" make the difference? Or reflecting surface?
@HYPEREGO I am just reading again - you are stating twice that I should choose pictures which are not sharp. What is that about, I didn’t hear that before? Why should they not be sharp. Should they be blurry? Should I artificially blur them, or just by focus blur? I am confused by that.
@andrei I think before I was just lucky, getting a good calibration with just a few images. When I tried again with the paper-printout, I also got a bad result now. The "beautiful rounded corners" I saw at the doc of the secor based (SB) findChessboardCornersSB() function. When I use that one with my printed metal board, I recognize the chessboard nicely.
@HYPEREGO sorry for all the comments, but I want to take all your valuable feedback seriously. By now I did recalibrations with my cameras, took about 800 pictures each, selected 300-400 for each, and the results look much more promising. Still, it looks like I can improve a lot at the quality of my chessboard shots.
What I am uncertain about, regarding focus: The calibration is only valid for a certain focus setting, so I must never change the focus during or ever after calibration. But how will I take pictures from different inclinations then? The pictures will be out of focus then.
Sorry for the late reply. No, the images should not be blurred. The problem is very visible in the picture you posted here. Looking at the corners is look like there is a "double" border. And that's where is the problem. It's correct, you should not change the focus or the focal length and thus the internal parameters of the camera changes. To better optimize the camera you can follow some of the tips here. Some of them cannot apply to OpenCV calibration algorithm, because they use an internal tool (that I suggest you to don't use if you're not very confident), but there are many valid suggestions. You can find even a script for GIMP to adjust brightness and contrast. It can help to improve the result, guaranteed ;)