Stereo rectification is not row aligned

edit

I try to rectify two images taken with a Samsung Galaxy S to calculate a depth map. They seem to be correctly rectified, but they are not row aligned, that is the drawn epipolar lines are horizontal, but have different y-values. I rectify the images with stereoRectify(), call initUndistortRectifyMap/remap to rectify the two images.

The rotation/translation is calculated with the fundamental matrix (using manually specified correspondences). It is converted to the essential matrix using intrinsic parameters. Then rotation/translation can be calculated using the formulas specified in Hartley&Zisserman's Multiple View Geometry, "9.6.2 Extraction of cameras from the essential matrix", p. 258

Question: Can you give me a hint to align the rows? In the few tutorials/examples that I found, the images are row-aligned after stereoRectify/initUndistortRectifyMap/remap

I calibrated the camera using a 6x8 checkerboard pattern. The colored lines represent the epipolar lines

If you want to rectify the images by yourself:

/* to view the two rectified images side by side,
   insert this source code in samples/cpp/stereo_match.cpp, after the following lines:
img1 = img1r;
img2 = img2r;
*/

cv::Mat imgboth(img1.rows, img1.cols*2, img1.type());
img1.copyTo(cv::Mat(imgboth, cv::Rect(0,0, img1.cols, img1.rows)));
img2.copyTo(cv::Mat(imgboth, cv::Rect(img1.cols, 0, img2.cols, img2.rows)));

cv::namedWindow("rectified", cv::WINDOW_NORMAL);
cv::imshow("rectified", imgboth);
while(cv::waitKey(0) != 'q');

Here the intrinsic and extrinsic parameters.

Here the original images for comparison: screen0.jpg screen1.jpg