cv::stereoCalibrate high RMS, roll and pitch of subordinate camera not quite correct

edit

Hi all,

I am working on getting a stereo calibration process running in our app, and whilst it roughly works using cv::stereoCalibrate() the results with higher calibration pair count are off due to the rotation matrix returned not being quite correct. I have a few questions on the process and the API in general to try and narrow down the reasons why my calibrations are not great.

• I am using two Kinect 4 Azures, on tripods with zero roll or pitch.
• I am using a calibration board, high quality printed and mounted on wooden board, with a 9 x 6 internal point size with a square size of 48mm.
• I use an "auto calibration" method of capturing frames (once a second) until a maximum calibration pair count is reached (currently targetting 40 calibration pairs)
• The process is to move the calibration board freely over the shared frustum of the cameras and each second a new frame is captured, I move and rotate the board to give as much variance as possible.
• Version of OpenCV I have integrated is 4.2.
• The cameras are set up roughly 2.5m apart, roughly 80 degrees between them.

For the OpenCV side of things, I am using the calibration coefficients from the sensors' factory calibrations for the sensor calibration intrinsics, including the camera matrix (these have been verified to be sensible and the camera matrix not transposed, sensible focal point and principle point etc).

After using cv::findChessboardCorners() for each image, the calibration point orientation has been verified to be correct each time with visualization of the points. These points are fed into cv::stereoCalibrate(), any capture attempt of a pair with points not in both views will fail and it will wait for the next valid pair up until we get 40 pairs.

After a 40 pair capture and stereo calibration, I see the subordinate camera as having a slight roll and pitch resulting in a projected pointcloud off by 50mm-100mm in x and y, although physically there is 0 roll and pitch. The Depth seems to be generally fine so this leads me to think projection of the pointcloud is not incorrect or offset in any way.

The RMS error from cv::stereoCalibrate() is always quite high (1.5-2.0), and seems to get linearly worse, the more calibration images I capture for the calibration. It should go the other way right?

Questions:

1. What could cause the high RMS error? I understand from reading other posts that a calibration error of < 0.5 or so acceptable? And why does the RMS error increase with more calibration images?
2. I have tried doing an undistortion of the calibration images before we pass them to stereoCalibrate() but nothing improves, does cv::stereoCalibrate() require undistorted points or does it already do that internally given that it takes the undistortion coefficients for the stereo calibration anyway?
3. Is there any reported faults in OpenCV 4.2 with stereo calibration? (though i'd ask for sanity!). I ...