Revision history [back]

Hey all,

I'm trying to use the GoPro Hero2 stereo setup (the cameras use fisheye lenses with a 170 degree field of view) and finished calibrating the cameras using the caltech toolbox. My problem is that OpenCV somehow doesn't crop the valid region in both images during rectification but rather includes wrapped regions in the corners and blank regions at the borders (see http://i.imgur.com/cWVnD.jpg for an example of the rectification result).

The problem associated with this is that the region of interest returned by stereoRectify is of size (0,0) even though it's supposed to mark the region of valid pixels - or at least some region, but certainly not nothing? At the moment I'm using an alpha value of -1 as this provides the most acceptable output so far. When I set alpha to 0, I only receive valid pixels, but the images aren't properly rectified any longer (i.e. the optical image centers aren't at the same y-coordinate .. I also tried setting cv::CALIB_ZERO_DISPARITY but that didn't fix this problem either).

I've also tried sending the raw images + calibration parameters through the ROS pipeline and have the stereo_image_proc node do the rectification, but that led to the same result.

This is the code I'm using to rectify my images:

// <load calibration parameters etc. >

int frame = 0;
cv::Mat mapx, mapy;
while(true)
{


    //read frame from video
    cv::Mat img;
    if(!vidStream.read(img))
        break;


    if(frame == 0)
    {
        //build undistortion + rectification map
        cv::Mat rect1, rect2, proj1, proj2;
        cv::Mat Q; // disparity-to-depth mapping matrix
        double alpha = -1;
        cv::Size imgSize(img.cols, img.rows);
        cv::Size newImgSize = imgSize;
        cv::Rect roi_left, roi_right;
        cv::stereoRectify(M_left, D_left, M_right, D_right, imgSize, R, T, rect1, rect2, proj1, proj2, 
                          Q, 0/*cv::CALIB_ZERO_DISPARITY*/, alpha, newImgSize, &roi_left, &roi_right);

        if(useLeft)
            cv::initUndistortRectifyMap(M_left, D_left, rect1, proj1, newImgSize, CV_16SC2, mapx, mapy);  
        else
            cv::initUndistortRectifyMap(M_right, D_right, rect2, proj2, newImgSize, CV_16SC2, mapx, mapy);

    }

    //undistort and rectify image
    cv::Mat imgRect;
    cv::remap(img, imgRect, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0));

    // < do stuff >
}

Thanks,
Thomas

Hey all,

I'm trying to use the GoPro Hero2 stereo setup (the cameras use fisheye lenses with a 170 degree field of view) and finished calibrating the cameras using the caltech toolbox. toolbox (using 5 distortion parameters). My problem is that OpenCV somehow doesn't crop the valid region in both images during rectification but rather includes wrapped regions in the corners and blank regions at the borders (see http://i.imgur.com/cWVnD.jpg for an example of the rectification result).

The problem associated with this is that the region of interest returned by stereoRectify is of size (0,0) even though it's supposed to mark the region of valid pixels - or at least some region, but certainly not nothing? At the moment I'm using an alpha value of -1 as this provides the most acceptable output so far. When I set alpha to 0, I only receive valid pixels, but the images aren't properly rectified any longer (i.e. the optical image centers aren't at the same y-coordinate .. I also tried setting cv::CALIB_ZERO_DISPARITY but that didn't fix this problem either).

I've also tried sending the raw images + calibration parameters through the ROS pipeline and have the stereo_image_proc node do the rectification, but that led to the same result.

This is the code I'm using to rectify my images:

// <load calibration parameters etc. >

int frame = 0;
cv::Mat mapx, mapy;
while(true)
{


    //read frame from video
    cv::Mat img;
    if(!vidStream.read(img))
        break;


    if(frame == 0)
    {
        //build undistortion + rectification map
        cv::Mat rect1, rect2, proj1, proj2;
        cv::Mat Q; // disparity-to-depth mapping matrix
        double alpha = -1;
        cv::Size imgSize(img.cols, img.rows);
        cv::Size newImgSize = imgSize;
        cv::Rect roi_left, roi_right;
        cv::stereoRectify(M_left, D_left, M_right, D_right, imgSize, R, T, rect1, rect2, proj1, proj2, 
                          Q, 0/*cv::CALIB_ZERO_DISPARITY*/, alpha, newImgSize, &roi_left, &roi_right);

        if(useLeft)
            cv::initUndistortRectifyMap(M_left, D_left, rect1, proj1, newImgSize, CV_16SC2, mapx, mapy);  
        else
            cv::initUndistortRectifyMap(M_right, D_right, rect2, proj2, newImgSize, CV_16SC2, mapx, mapy);

    }

    //undistort and rectify image
    cv::Mat imgRect;
    cv::remap(img, imgRect, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0));

    // < do stuff >
}

Thanks,
Thomas

Hey all,

I'm trying to use the GoPro Hero2 stereo setup (the cameras use fisheye lenses with a 170 degree field of view) and finished calibrating the cameras using the caltech toolbox (using 5 distortion parameters). My problem is that OpenCV somehow doesn't crop the valid region in both images during rectification but rather includes wrapped regions in the corners and blank regions at the borders (see http://i.imgur.com/cWVnD.jpg for an example of the rectification result).

The problem associated with this is that the region of interest returned by stereoRectify is of size (0,0) even though it's supposed to mark the region of valid pixels - or at least some region, but certainly not nothing? At the moment I'm using an alpha value of -1 as this provides the most acceptable output so far. When I set alpha to 0, I only receive valid pixels, but the images aren't properly rectified any longer (i.e. the optical image centers aren't at the same y-coordinate .. I also tried setting cv::CALIB_ZERO_DISPARITY but that didn't fix this problem either).

I've also tried sending the raw images + calibration parameters through the ROS pipeline and have the stereo_image_proc node do the rectification, but that led to the same result.

This is the code I'm using to rectify my images:

// <load calibration parameters etc. >

int frame = 0;
cv::Mat mapx, mapy;
while(true)
{


    //read frame from video
    cv::Mat img;
    if(!vidStream.read(img))
        break;


    if(frame == 0)
    {
        //build undistortion + rectification map
        cv::Mat rect1, rect2, proj1, proj2;
        cv::Mat Q; // disparity-to-depth mapping matrix
        double alpha = -1;
        cv::Size imgSize(img.cols, img.rows);
        cv::Size newImgSize = imgSize;
        cv::Rect roi_left, roi_right;
        cv::stereoRectify(M_left, D_left, M_right, D_right, imgSize, R, T, rect1, rect2, proj1, proj2, 
                          Q, 0/*cv::CALIB_ZERO_DISPARITY*/, alpha, newImgSize, &roi_left, &roi_right);

        if(useLeft)
            cv::initUndistortRectifyMap(M_left, D_left, rect1, proj1, newImgSize, CV_16SC2, mapx, mapy);  
        else
            cv::initUndistortRectifyMap(M_right, D_right, rect2, proj2, newImgSize, CV_16SC2, mapx, mapy);

    }

    //undistort and rectify image
    cv::Mat imgRect;
    cv::remap(img, imgRect, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0));

    // < do stuff >
}

EDIT: I've started trying to debug the opencv code to figure out what's going on and here is an image showing the inner (green) and outer (red) rectangles which are calculated inside stereoRectify using the icvGetRectangles function to get the regions of interest (which is the intersection between inner and outer rectangle): http://i.imgur.com/tUKHe.png Other than those regions (or especially the green one) not making a lot of sense the problem also is that the inner rectangle always has a negative width: rect inner: offset:(1331.3,268.081), size:(-546.046,253.081) rect outer: offset:(185.031,-5162.95), size:(10238.4,5971.77)

Thanks,
Thomas

Hey all,

I'm trying to use the GoPro Hero2 stereo setup (the cameras use fisheye lenses with a 170 degree field of view) and finished calibrating the cameras using the caltech toolbox (using 5 distortion parameters). My problem is that OpenCV somehow doesn't crop the valid region in both images during rectification but rather includes wrapped regions in the corners and blank regions at the borders (see http://i.imgur.com/cWVnD.jpg for an example of the rectification result).

The problem associated with this is that the region of interest returned by stereoRectify is of size (0,0) even though it's supposed to mark the region of valid pixels - or at least some region, but certainly not nothing? At the moment I'm using an alpha value of -1 as this provides the most acceptable output so far. When I set alpha to 0, I only receive valid pixels, but the images aren't properly rectified any longer (i.e. the optical image centers aren't at the same y-coordinate .. I also tried setting cv::CALIB_ZERO_DISPARITY but that didn't fix this problem either).

I've also tried sending the raw images + calibration parameters through the ROS pipeline and have the stereo_image_proc node do the rectification, but that led to the same result.

This is the code I'm using to rectify my images:

// <load calibration parameters etc. >

int frame = 0;
cv::Mat mapx, mapy;
while(true)
{


    //read frame from video
    cv::Mat img;
    if(!vidStream.read(img))
        break;


    if(frame == 0)
    {
        //build undistortion + rectification map
        cv::Mat rect1, rect2, proj1, proj2;
        cv::Mat Q; // disparity-to-depth mapping matrix
        double alpha = -1;
        cv::Size imgSize(img.cols, img.rows);
        cv::Size newImgSize = imgSize;
        cv::Rect roi_left, roi_right;
        cv::stereoRectify(M_left, D_left, M_right, D_right, imgSize, R, T, rect1, rect2, proj1, proj2, 
                          Q, 0/*cv::CALIB_ZERO_DISPARITY*/, alpha, newImgSize, &roi_left, &roi_right);

        if(useLeft)
            cv::initUndistortRectifyMap(M_left, D_left, rect1, proj1, newImgSize, CV_16SC2, mapx, mapy);  
        else
            cv::initUndistortRectifyMap(M_right, D_right, rect2, proj2, newImgSize, CV_16SC2, mapx, mapy);

    }

    //undistort and rectify image
    cv::Mat imgRect;
    cv::remap(img, imgRect, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0));

    // < do stuff >
}

EDIT: I've started trying to debug the opencv code to figure out what's going on and here is an image showing the inner (green) and outer (red) rectangles which are calculated inside stereoRectify using the icvGetRectangles function to get the regions of interest (which is the intersection between inner and outer rectangle): http://i.imgur.com/tUKHe.png http://i.imgur.com/tUKHe.png
Other than those regions (or especially the green inner one) not making a lot of sense the problem also is that the inner rectangle always has a negative width: rect inner: offset:(1331.3,268.081), size:(-546.046,253.081) rect outer: offset:(185.031,-5162.95), size:(10238.4,5971.77)

Thanks,
Thomas