I run cv2.calibrateCamera on 67 images with 19x3 chessboard corner detections in each. I find reprojection error around 0.3. When I undistort images, they look great. They had little distortion before, and appear to have basically no distortion afterward.
Later, I use these camera calibrations (and those from the other camera in a stereo setup) as input to stereoCalibrate, which returns error about 0.3. When I try to stereorectify and remap, I get nonsense.
This led me back to the original camera matrices. My distortion coefficients from the camera calibration are
array([[-1.44561083e-01, -3.21620254e+01, 0.00000000e+00,
0.00000000e+00, 1.72034253e+03]])
K3 seems very large! Any idea what is going on here? Could this be ruining my stereo rectification later on?