Getting huge error on distance calculated from stereo vision

edit

I have two cameras that I calibrated using OpenCV’s stereoCalibration functions.

The calibration is done on a 19 pairs of left and right camera images, the chessboard grid is 9*6 and I have set the square size to 2.4 (in centimetres) :

Calibration of left camera gave an error of 0.229
Calibration of left camera gave an error of 0.256
Stereo Calibration gave an error of 0.313

The rectified images look like this and seem to be correct.

I then took few photos of me standing at certain distances to check whether it is working properly or not. Here is one pair of images that i took at about 600 cms away from camera:

The images on top row are original images and the images on the second row are rectified:

On running stereoSGBM I get the following result:

I calculated the distance of white dot on the disparity image and the result was 272. Considering I calibrated the cameras using chessboard square size as 2.4cms the result is in cms (i.e 272 cms) This is a huge error (600 – 272 = 328 cms). Any help that would help me solve this issue will be appreciated. Here are some relevant parts of my code, if this code doesnt provide enough information leave a comment and ill add more snippets.

SINGLE CAMERA CALIBRATION CODE:

def calibration(PATH_TO_FOLDER,PATTERN_SIZE,  visualize, output_file):

    '''
    1.Detect chessboard corners.
    2.Camera Calibration    
    Calibrated data stored at @output_file
    '''

    ### STEP 1 : DETECTING CHESSBOARD CORNERS
    x_pattern, y_pattern = PATTERN_SIZE
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

    # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
    objp = np.zeros((y_pattern*x_pattern,3), np.float32)
    objp[:,:2] = np.mgrid[0:x_pattern,0:y_pattern].T.reshape(-1,2)
    objp *= SQUARE_SIZE_FOR_CALIBRATION

    # Arrays to store object points and image points from all the images.
    objpoints = [] # 3d point in real world space
    imgpoints = [] # 2d points in image plane.

    for filename in os.listdir(PATH_TO_FOLDER):
        if filename[-3:] == IMAGE_FORMAT:
            img = cv2.imread(os.path.join(PATH_TO_FOLDER,filename))
            if img is not None:
                gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

                # Find the chess board corners
                chessboardFlags = cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE | cv2.CALIB_CB_FILTER_QUADS
                ret, corners = cv2.findChessboardCorners(gray, PATTERN_SIZE,chessboardFlags)

                # If found, add object points, image points (after refining them)
                if ret == True:
                    #Add the corresponding object coordinate system points
                    objpoints.append(objp)
                    #Refine image coordinates
                    corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
                    #Add corresponding image coordinates
                    imgpoints.append(corners2)

                    # Draw and display the corners
                    if visualize:
                        img = cv2.drawChessboardCorners(img, PATTERN_SIZE, corners2,ret)
                        cv2.imshow('img',img)
                        cv2.waitKey(500)
    cv2.destroyAllWindows()

    ### STEP 2 : CAMERA CALIBRATION
    ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None ,cv2.CALIB_RATIONAL_MODEL)

STEREO CAMERA CALIBRATION CODE

def stereocalibrate(PATH_TO_LEFT_IMAGES, PATH_TO_RIGHT_IMAGES, NUM_IMAGES, PATTERN_SIZE, VISUALIZATION, RECALIBRATE_INDIVIDUAL_CAMS):
    '''
    1.Detecting chessboard corners.
    2.Stereo calibration of cameras.(Geometry between cams)
    3.Stereo rectification of cameras.(Row aligned images)
    '''

    if(RECALIBRATE_INDIVIDUAL_CAMS):
        left_camera = calibration(PATH_TO_LEFT_IMAGES,PATTERN_SIZE, VISUALIZATION, open(LEFT_CAMERA_DATA_FILE ...