Virtually moving cameras before cv::stereoRectify

edit

Hello,

I need some help on doing the openCV stereo rectification.

We have a working system. The stereo rectification is working fine. But because of the fact, that the second camera is mounted 45 degrees behind the first camera we get something like a perspective distortion in the rectification result images. After my analysis the function cv::stereoRectify uses the cameras translation vector to calculate the cameras output rotation matrices.

My idea is to move the second camera "virtually" in the direction of the cameras viewing axis into the first cameras image plane (the translation vector z value should be zero) and do then the cv::stereoRectify.

The assumption is that I have to calculate the new translation vector and rescale the cameraMatrix for the second camera. But the epipolar lines in the result images doesn't match. Here is my code:

       double _zDirectionCam1[3] = {0.0, 0.0, 1.0};
        cv::Mat zDirectionCam1   = cv::Mat(3, 1, CV_64F, _zDirectionCam1 );
        cv::Mat  zDirectionCam2 = mCameraPairRotation * zDirectionCam1;

        const double deltaZ = mCameraPairTranslation.at<double>( 2, 0 ) / zDirectionCam2.at<double>(2, 0);


        const unsigned int CAMERA_ARRAY_ID = MASTER_CAMERA_ARRAY_ID;

        const double scaleX = 1 / (1.0 + deltaZ / mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 0, 0 ));
        const double scaleY = 1 / (1.0 + deltaZ / mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 1, 1 ));

        qDebug()    << "deltaZ = " <<  deltaZ
                    << "scaleX = " <<  scaleX
                    << "scaleY = " <<  scaleY;


        mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 0, 0 ) *= scaleX;
        mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 1, 1 ) *= scaleY;
        mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 0, 2 ) *= scaleX;
        mCameraMatrixCtn[ CAMERA_ARRAY_ID ].at< double >( 1, 2 ) *= scaleY;


        mCameraPairTranslation = mCameraPairTranslation - deltaZ * zDirectionCam2;

        qDebug() << "Tranlation Manipulated: x= " <<  mCameraPairTranslation.at< double >( 0, 0 )
                         << " y= " << mCameraPairTranslation.at< double >( 1, 0 )
                         << " z= " << mCameraPairTranslation.at< double >( 2, 0 );

    }

    cv::stereoRectify( mCameraMatrixCtn[ SLAVE_CAMERA_ARRAY_ID ],
                       mDistCoeffsCtn[ SLAVE_CAMERA_ARRAY_ID ],
                       mCameraMatrixCtn[ MASTER_CAMERA_ARRAY_ID ],
                       mDistCoeffsCtn[ MASTER_CAMERA_ARRAY_ID ],
                       mInputImageSize,
                       mCameraPairRotation,
                       mCameraPairTranslation,
                       mR[ SLAVE_CAMERA_ARRAY_ID ],
                       mR[ MASTER_CAMERA_ARRAY_ID ],
                       mP[ SLAVE_CAMERA_ARRAY_ID ],
                       mP[ MASTER_CAMERA_ARRAY_ID ],
                       qMat,
                       /*cv::CALIB_ZERO_DISPARITY */ 0,
                       -1,
                       mResultImageSize,
                       &mROICtn[ SLAVE_CAMERA_ARRAY_ID ],
                       &mROICtn[ MASTER_CAMERA_ARRAY_ID ] );

Is there something wrong in my assumption. If the assumption is OK, what is wrong in my calculations?