Calculate extrinsics between two cameras using FindHomography?

asked 2019-08-12 10:26:21 -0600

antithing gravatar image

updated 2019-12-10 01:17:22 -0600

Akhil Patel gravatar image

I have two cameras with different lenses and resolutions. I have images, with chessboards visible in both.

I am trying to calculate very accurate extrinsic information between the two cameras, and then have the method repeatable when i change the lens on one camera, and the extrinsic values stay the same across multiple lens calibrations.

I have tried lens calibration and charuco boards, but although this gives me acceptable translation values, the rotation values vary up to a few degrees on each run, depending on the calibration.

I have tried chessboard corners and solvePnP also, but the results vary wildly.

I have tried cv::findEssentialMat on the chessboard corners, then cv::recoverPose, but the results are even worse (I have read that this method does not like planar points).

So, are there any other methods that I can use to find an accurate rotation extrinsic value between two cameras?

Can I use FindHomography somehow to get the relative pose between images?

The code I am using for the solvepnp approach is:

void main()
    cv::Mat K1, D1, K2, D2;

    bool readOk = readCameraParameters("data/zed1920.xml", K1, D1);
    if (!readOk) {
        cerr << "Invalid camera file" << endl;

    readOk = readCameraParameters("data/sdi.xml", K2, D2);
    if (!readOk) {
        cerr << "Invalid camera file" << endl;

    //set up board
    int grid_size_x = 7;
    int grid_size_y = 5;
    double squareSize = 4.5;

    std::vector<cv::Point3d> object_points_;
    for (int y = 0; y < grid_size_y; y++)
        for (int x = 0; x < grid_size_x; x++)
            object_points_.push_back(cv::Point3d(x * squareSize, y * squareSize, 0));

    cv::Mat imageGray, imageGray2;

    //pair 1
    cv::Mat frame1_2 = cv::imread("frames/frame1_1.jpg");
    cv::Mat frame1_1 = cv::imread("frames/frame2_1.jpg");

    cv::cvtColor(frame1_1, imageGray, cv::COLOR_BGR2GRAY);
    cv::cvtColor(frame1_2, imageGray2, cv::COLOR_BGR2GRAY);
    std::vector<cv::Point2f> corners1, corners2;

    //find board in image
    bool found_chessboard = cv::findChessboardCorners(imageGray, cv::Size(grid_size_x, grid_size_y), corners1, cv::CALIB_CB_ADAPTIVE_THRESH);// cv::CALIB_CB_ADAPTIVE_THRESH + cv::CALIB_CB_NORMALIZE_IMAGE + cv::CALIB_CB_FAST_CHECK);
    bool found_chessboard2 = cv::findChessboardCorners(imageGray2, cv::Size(grid_size_x, grid_size_y), corners2, cv::CALIB_CB_ADAPTIVE_THRESH);//   cv::CALIB_CB_ADAPTIVE_THRESH + cv::CALIB_CB_NORMALIZE_IMAGE + cv::CALIB_CB_FAST_CHECK);

    if (found_chessboard && found_chessboard2)

            cv::cornerSubPix(imageGray, corners1, cv::Size(3, 3), cv::Size(-1, -1), cv::TermCriteria(cv::TermCriteria::MAX_ITER + cv::TermCriteria::EPS, 50, 0.01));
            cv::cornerSubPix(imageGray2, corners2, cv::Size(3, 3), cv::Size(-1, -1), cv::TermCriteria(cv::TermCriteria::MAX_ITER + cv::TermCriteria::EPS, 50, 0.01));

            if (corners1.size() != 0 && corners2.size() != 0)

                cv::Mat rvec1, tvec1;
                cv::solvePnP(object_points_, corners1, K1, D1, rvec1, tvec1, false, cv::SOLVEPNP_ITERATIVE);
                cv::Mat rvec2, tvec2;
                cv::solvePnP(object_points_, corners2, K2, D2, rvec2, tvec2, false, cv::SOLVEPNP_ITERATIVE);

                cv::Mat R1, R2;

                cv::Rodrigues(rvec1, R1);
                cv::Rodrigues(rvec2, R2);
                cv::Mat R_1to2 = R2 * R1.t();
                cv::Mat tvec_1to2 = R2 * (-R1.t()*tvec1) + tvec2;

                std::cout << tvec_1to2 << std::endl;

                Eigen::Matrix<double, 3, 3> Reigen2;
                cv2eigen(R_1to2, Reigen2);

                Eigen::Quaterniond quaternion2(Reigen2);
                auto euler2 = quaternion2.toRotationMatrix().eulerAngles(0, 1, 2);

                euler2 = euler2 * 180 / M_PI;

                std::cout << " rotation diff pair 1 is: " << euler2.x() << " " << euler2.y() << " " << euler2.z() << std::endl ...
edit retag flag offensive close merge delete


Can you guarantee that when you change a lens the camera maintains its position and orientation to a fraction of a mm and a degree despite screwing a lens out and in? Perhaps you subtly tilt the camera during this operation which explains why translation is acceptable but orientation differs up to a few degrees?

Witek gravatar imageWitek ( 2019-08-13 02:59:07 -0600 )edit

Hi, thanks for your reply. My default setup, which I am having trouble getting a good result with, is camera A at 1920 x 1080, with a 25mm lens. Camera B at 1280 x 720, with a 7mm lens. I am thinking that the FOV difference might be the issue here. If i add a center crop to the wider image and run stereo calibrate, will that still give me accurate extrinsics?

antithing gravatar imageantithing ( 2019-08-13 03:34:19 -0600 )edit

I don't think a crop would do anything. I understand that both cameras can see a chessboard that is placed somewhere within the FOV of both cameras and nothing is moved between the shots (neither the chessboard nor the cameras) and you use SolvePnP to get the extrinsics of the cameras, right? And your goal is to get the transformation between the cameras, correct?

Witek gravatar imageWitek ( 2019-08-14 06:20:20 -0600 )edit

Hi, that is correct. i have updated the question with more code to show what I am doing. the relative rotation and translation that I get vary wildly depending on the angle of the board and distance from cameras. Is solvePnP the most accurate way to do this? Is decomposing the H matrix an option?

antithing gravatar imageantithing ( 2019-08-14 06:36:20 -0600 )edit

Can you also provide the K1, K2, D1, D2 and both images. I don't know if I can help but I am curious to run it myself and see the results.

Witek gravatar imageWitek ( 2019-08-14 07:05:20 -0600 )edit

Sure, you can grab the intrinsics and frames here:

I have updated the question code to a main function that will work with this data so you can see what i mean! Thank you!

antithing gravatar imageantithing ( 2019-08-14 07:24:23 -0600 )edit

Hi, I just noticed an error in the code I uploaded, the read should be:

    cv::Mat frame1_2 = cv::imread("frames/frame1_1.jpg");
    cv::Mat frame1_1 = cv::imread("frames/frame2_1.jpg");

//pair 2
    frame1_2 = cv::imread("frames/frame1_2.jpg");
    frame1_1 = cv::imread("frames/frame2_2.jpg");

//pair 3
    frame1_2 = cv::imread("frames/frame1_3.jpg");
    frame1_1 = cv::imread("frames/frame2_3.jpg");
antithing gravatar imageantithing ( 2019-08-14 10:02:35 -0600 )edit

First impressions after seeing the images: they are blurry due to chessboard being moved - especially frame1_x. This makes corner detection less precise. Another issue is that if the chessboard is being moved the cameras take images at different chessboard locations (I assume that you just capture from one camera and then from the other one without precise synchronization) although I might be wrong. My suggestion is to fix the chessboard at a few locations and take sharp pictures - and see if it helps. Additionally, if you have similarly blurry images for camera calibration, you might want to recalibrate the cameras minimizing the blur (brighter light and no backlight would also be a good idea).

Witek gravatar imageWitek ( 2019-08-15 17:22:59 -0600 )edit

Thank you! I cannot sync them, it's true. I will try to use a static chessboard, and adjust the focus / lighting conditions as you suggest. Thank you again for your time.

antithing gravatar imageantithing ( 2019-08-16 06:45:54 -0600 )edit

Please let us know if that helped. One other thing is that there are more than one pnp method in OpenCV and their provide slightly different results. It would be interesting to see which are the most consistent. And last, but not least, your chessboard has odd number of rows and columns which makes its orientation ambiguous (+-180 deg) and shifts its origin which is confusing when comparing results.

Witek gravatar imageWitek ( 2019-08-19 14:22:41 -0600 )edit