Undistortion at far edges of image

edit

What you are seeing here are deficiencies of both the getOptimalNewCameraMatrix and initUndistortRectifyMap (actually undistortPoints) functions that only become noticeable for cameras with strong radial distortion. In short getOptimalNewCameraMatrix tries to estimate how much the current camera matrix would have to be scaled (and the resulting image cropped), in order to avoid having empty (black) dents or outwards bubbles visible in the output image. The function makes the assumption that the radial distortion is monotonous (from your distortion coeffecients), that you either have constant outwards or inwards warping in your images. With this assumption the getOptimalNewCameraMatrix discretely samples an 8x8 grid of points evenly distributed throughout the image, undistorts them (using undistortPoints) with an approximative algorithm that again assumes the distortion function is monotonous, and uses a heuristic to find the subset of the grid points guaranteed to be within in the image after undistortion. With these points it can estimate how much to zoom the original camera matrix so only valid image pixels are visible.

So here it where things break down and why you get that weird reflective ring. The OpenCV calibration algorithm does not guarantee that the estimated distortion function is monotonous, in fact it returns an arbitrary function highly sensitive to your input images, if you have enabled the coeffecients K_2 and K_3. I have noticed lots of times that the estimated distortion function changes from outwards warping to strong inwards warping around the border of the image (in cameras with strong radial distortion). So what happens to the getOptimalNewCameraMatrix and undistortPoints functions when the monotony constraint is violated? The first is that at points where the distortion function changes sign (and warping type), the undistortPoints function estimates completely wrong undistorted point locations. The second and more severe effect is that the getOptimalNewCameraMatrix then fails to estimate the correct visible subset of the image, resulting in arbitrary and counter intuitive results.

In your case, what you are seeing is that your distortion function is not monotonous, so that around the image border there is a strong switch in the distortion type from outwards to inwards. The ring you see is a a result of the distortion function being so strongly inward warping at those positions that it samples part of the image again. Because of the switch between outward and inward warping around the border (and errors discussed above), the getOptimalNewCameraMatrix mistakenly believes it must zoom out and not zoom in.

There is no quick solution and guaranteed to this problem. You must recalibrate your camera and make sure to get plenty of views close the image border. Then sample the distortion function around the image border and make sure it does not switch distortion type. Keep repeating this process until you converge to an acceptable solution.

Here is a python implementation of the relevant functions that uses Newton-Raphson followed by a binary line searches for better calculations of the inverse distortion function. This will result in a more correct OptimalCameraMatrix.

The repository also contains a function that enables using calibration images where not all of the corners are visible or detected, hence allowing better constraints on the areas near the boundary of the image. This is required so the distortion parameters will be calculated more precisely and the undistorted image will be more accurate. Moreover, it reduces the chance that the OptimalCameraMatrix bug will be encountered.

Please note that this implementation uses only the first 5 distortion parameters.

Ok, I've been wracking my brain on this and I think I've finally solved it!

Tl;dr -- make sure the edges are well represented by the checkerboard in your calibration images.

The key is that the distortion is extreme at the edges, and my theory was that without good checkerboard representation at the edges, the edge distortion solution would not be very accurate. This seems to have fixed it. If you want some numbers for navel-gazing, here are my distCoeffs for two different calibration image sets: one close up, the other mostly far away.

Note: the coefficients are both positive and negative (meaning "moustache distortion" and not monotonic.) Note in particular the really big values for the far away images -- indicating severe edge distortion. My idea was that since distortion is mild in the center, if all my checkerboards are in the center, the distortion could solve in widely varying ways, subject to factors like roundoff error and such.

Looks like my theory is correct -- the coefficients and ROI converge nicely as I increase the images for the near images. Not quite so nice for the far ones.

Bonus ProTip (from Paul Debevec's PhD thesis, chapter 4) -- you can't do gamma correction (or any lightening) on calibration images, because this can shrink or expand the edges of white on black squares!

(k1, k2, p1, p2, k3 -- p is the center point of the distortion.)

Far:

• Dec 13, 10 far images: dist [[-0.36835 5.26118 -0.01009 -0.00408 -31.85337]] roi (31, 94, 1843, 889)
• Dec 13, 50 far images: dist [[ 0.10918 0.78833 -0.00348 0.00295 -5.18248]] roi (0, 0, 0, 0)
• Dec 13, 100 far images: dist [[ 0.11246 0.55816 -0.00238 0.00048 -3.41276]] roi (0, 0, 0, 0)

Close:

• Dec 14, 10 close images: dist [[ 0.09757 -0.11132 -0.0049 -0.00319 0.05245]] roi (8, 14, 1905, 1058)
• Dec 14, 50 close images: dist [[ 0.16167 -0.38233 -0.00299 -0.00002 0.21451]] roi (9, 5, 1901, 1067)
• Dec 14, 100 close images: dist [[ 0.17922 -0.45282 -0.00315 -0.00019 0.29558]] roi (8, 4, 1903, 1069)
• Dec 14, 200 close images: dist [[ 0.18474 -0.46364 -0.00253 -0.00078 0.30383]] roi (7, 5, 1904, 1068)
• Dec 14, 287 close images: dist [[ 0.18519 -0.46466 -0.00253 -0.00083 0.30463]] roi (7, 5, 1904, 1068)