Something is wrong with the calibration

asked 2018-03-04 06:17:58 -0500

Vahagn gravatar image

updated 2018-03-14 01:21:52 -0500

I am trying to stereo calibrate the camera pair image description.

I created a set of stereo images using the chessboard image on my home TV. The images can be found at https://drive.google.com/drive/folder...

The code for detecting the intrinsic and extrinsic parameters looks as follows:

def calibrate_stereo_pair(image_folder : str,
                      coefficients_folder: str,
                      dist_coefficient_pattern = 'dist_c%s.npy',
                      camera_matrix_pattern = 'cammat_%s.npy',
                      win_names : List[str] = ['left', 'right'],
                      nrows = 9,
                      ncols = 6) -> bool:

    N = len(win_names)
    if N != 2:
        raise NotImplementedError("")


    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    objp = np.zeros((nrows * ncols, 3), np.float32)
    objp[:, :2] = np.mgrid[0:nrows, 0:ncols].T.reshape(-1, 2)

    objpoints = []  # 3d point in real world space
    imgpoints = []
    for i in range(N):
        imgpoints.append([])
    index = 1
    while True:
        filenames = [os.path.join(image_folder , win_names[i] + str(index) + '.png') for i in range(N)]
        if not all(os.path.exists(fname) for fname in filenames):
            if index == 1:
                print('No images detected.')
                return False
            else:
                break

        frames = [cv2.imread(fname, 0) for fname in filenames]
        found = [cv2.findChessboardCorners(gray, (nrows, ncols), None) for gray in frames]
        if not all(q[0] for q in found):
            print ('WARNING: Failed to detect chessboard cornesr in the {0}th set. Skipping...'.format(index))
            continue

        objpoints.append(objp)
        for i in range(N):
            cv2.cornerSubPix(frames[i], found[i][1], (11, 11), (-1, -1), criteria)
            imgpoints[i].append(found[i][1])
        index += 1

    if len(objpoints) == 0:
        print('No images found')
        return False

    retval1, cammat1, K1, L11, L12 = cv2.calibrateCamera(objpoints, imgpoints[0], frames[0].shape[::-1], None, None)
    retval2, cammat2, K2, L21, L22 = cv2.calibrateCamera(objpoints, imgpoints[1], frames[1].shape[::-1], None, None)

    #TODO: resize images to the minimal w, length
    retval, cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, R, T, E, F = cv2.stereoCalibrate(objpoints,imgpoints[0], imgpoints[1],cammat1 , K1, cammat2, K2, frames[0].shape[::-1], flags = cv2.CALIB_FIX_FOCAL_LENGTH  | cv2.CALIB_FIX_S1_S2_S3_S4 | cv2.CALIB_FIX_K1 | cv2.CALIB_FIX_K2 | cv2.CALIB_FIX_K3 | cv2.CALIB_FIX_K4 | cv2.CALIB_FIX_K5 | cv2.CALIB_FIX_PRINCIPAL_POINT | cv2.CALIB_FIX_INTRINSIC  | cv2.CALIB_RATIONAL_MODEL, criteria=criteria)


    np.save(os.path.join(coefficients_folder, dist_coefficient_pattern % win_names[0]), distCoeffs1)
    np.save(os.path.join(coefficients_folder, dist_coefficient_pattern % win_names[1]), distCoeffs2)

    np.save(os.path.join(coefficients_folder, camera_matrix_pattern % win_names[0]), cameraMatrix1)
    np.save(os.path.join(coefficients_folder, camera_matrix_pattern % win_names[1]), cameraMatrix2)

    np.save(os.path.join(coefficients_folder, 'R.npy'), R)
    np.save(os.path.join(coefficients_folder, 'T.npy'), T)
    np.save(os.path.join(coefficients_folder, 'E.npy'), E)
    np.save(os.path.join(coefficients_folder, 'F.npy'), F)
    return True

Initially I used the function stereoCalibrate without separately calibrating each camera, but experiments show that this approach produces slightly better results.
Next I used the following code for further rectification and block matching:

cv2.namedWindow('left', cv2.WINDOW_NORMAL)
cv2.namedWindow('right', cv2.WINDOW_NORMAL)

#uses np.load for loading the coefficients saved in the previous step
cam_mat1, cam_mat2, dist1, dist2, R, T, E, F = camera_calibration.load_stereo_pair([1,2],config.coefficient_folder)


caps = [cv2.VideoCapture(i) for i in [1,2]]
frames = [ cv2.cvtColor(cap.read()[1],cv2.COLOR_BGR2GRAY) for ...
(more)
edit retag flag offensive close merge delete

Comments

Your calibration images look not enough varied (always horizontal, always at the same distance, ...). You should also check the reprojection error (should be less than 0.5 px) and calibrate independently each camera first (as you did?). Redo the calibration until you get correct results. Some calibration image examples.

Eduardo gravatar imageEduardo ( 2018-03-04 07:17:03 -0500 )edit

Thanks a lot. Will check in a couple of hours and report back.

Vahagn gravatar imageVahagn ( 2018-03-04 07:58:44 -0500 )edit

Finally got the reprojection errors for all 3 calibrations (clibrateCamera(left), clibrateCamera(right), stereoCalibrate() less than 0.5. To be more precise :

left: 0.11 right: 0.14 stereo: 0.33

I have updated the images in the drive: https://drive.google.com/drive/folder...

However, the dense matching is again noisy. Am I missing something?

Vahagn gravatar imageVahagn ( 2018-03-09 08:44:41 -0500 )edit

The calibration images look good for me and the reprojection errors you get are good also.

What do you mean by "the dense matching is noisy"? Do you mean the disparity map computed by the semi-global block matching (SGBM) is noisy? There are some parameters to tune, see for example #1 or #2.

You should update your question with the disparity map you get.

Eduardo gravatar imageEduardo ( 2018-03-09 10:20:39 -0500 )edit

@Eduardo, thank you so much. I have updated the initial post with the new image for sift match. I have a couple of questions:

  1. Shouldn't the matching lines be horizontal after the rectification?
  2. Will I get better results if I fix my cameras to be more parallel to each other?
Vahagn gravatar imageVahagn ( 2018-03-11 01:38:19 -0500 )edit

Yes, the matching lines should be horizontal. Ideally, the cameras should be fixed to have only a translation in X between the left and right camera frames (rotation is almost identity, same height).

In your case, I don't know where the issue comes from. Be sure that the cameras are rigidly attached and cannot move when you handle, move the stereo rig.

If you look at the estimated transformation between the left and right cameras, you should check that the values make sense (translation in X and Y should be easy to check).

There are also calibration.cpp and stereo_calib.cpp sample files that can be used.

Eduardo gravatar imageEduardo ( 2018-03-13 10:43:10 -0500 )edit