Revision history [back]

I ran in to the same problem. Opening up the source code revealed the problem - The cv::stereoRectify function assumes most of the input matrices are in double precision format (CV_64F). Converting the rotation matrix (rotation_ in your example) to a double precision cv::Mat solves the problem.

cv::Mat r;
rotation_.convertTo(r, CV_64F);
cv::stereoRectify(ints_left_, ints_right_, dtcs_left_, dtcs_right_, 
          cv::Size(640, 480), r, translation_, 
          homography_left_, homography_right_, 
          rppm_left_, rppm_right_, qmat_);

Hope this clarifies things.