I am using OpenCV 2.4.11 to undistort the signal from a depth camera. It does a reasonable job, but I still notice that some straight edges do not appear as lines in the calibrated image. I would estimate that in the worst case, if I draw a line from one end of the straight edge to the other, I see a roughly 2 pixel deviation from the line in a 320x240 image. In my use case, this translates to 3 inches of error at the far operating range of my depth camera. I am looking to get 1 inch accuracy.
I have tried the checkerboard pattern and the asymmetric circles grid. I have found the asymmetric circles grid to produce much more correct and stable results, as well as better final distortion coefficients and camera matrices. I have used 40 images covering the FOV of the camera and at various poses. I tried using more, but at 90 images OpenCV consistently crashes (due to the debugging environment I am in, I can't currently say what is causing the crash).
Here is a picture with the original image and the undistorted image. Note that the bottom edge of the poster I am holding is straighter, but not completely straight in the undistorted image.
Is this the best I can expect, or should I expect pixel-level accuracy? If I can do better, how can I do better?
Here is a folder with the calibration images and some ancilary files : https://drive.google.com/open?id=1_2IOCX4GjYGDKYaPDQzR7I0kuKoUICC3