stereoRectify doesn't work sometimes

edit

I have many shots of object on aruco board:

All images in the folder are undistorted, you can download them here: https://drive.google.com/drive/folder...

For each picture i have xml file with t_vec and r_vec of the aruco board. I draw blue line over border and it fits perfectly, so these vectors are correct:

For each pair of images I use this code to calculate R and T from one viewport to another:

pair<Mat,Mat> calcRT(View&v1, View&v2) {
    Mat R1,R2,R,T;
    Rodrigues(v1.r, R1);
    Rodrigues(v2.r, R2);
    R = R2*R1.inv();
    Mat RT1 = R*v1.t.t();
    T = v2.t.t() - RT1;
    return make_pair(R, T);
}

And then I call rectify this way:

auto RT = calcRT(v1, v2);
Mat R1, R2, P1, P2, Q, RT_d;
RT.first.convertTo(RT_d, CV_64F);
Mat tt = RT.second;
tt.convertTo(tt, CV_64F);
auto imageSize = Size(v1.img.cols, v1.img.rows);
Rect validROI[2];
stereoRectify(cm, Mat(), cm, Mat(), imageSize,
    RT_d, tt, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY*0,.999, imageSize, &validROI[0], &validROI[1]);
bool isVerticalStereo = fabs(P2.at<double>(1, 3)) > fabs(P2.at<double>(0, 3));
Mat rmap[2][2];
initUndistortRectifyMap(cm, Mat(), R1, P1, imageSize, CV_32FC1, rmap[0][0], rmap[0][1]);
initUndistortRectifyMap(cm, Mat(), R2, P2, imageSize, CV_32FC1, rmap[1][0], rmap[1][1]);
Mat canvas;
double sf = 1;
int w, h;
if (isVerticalStereo) {return;} // I never get here
cout << "horisontal stereo" << endl;
w = cvRound(imageSize.width*sf);
h = cvRound(imageSize.height*sf);
canvas.create(h, w * 2, CV_8UC3);
String file;
Mat rimg1, rimg2;
remap(v1.img, rimg1, rmap[0][0], rmap[0][1], INTER_LINEAR);
remap(v2.img, rimg2, rmap[1][0], rmap[1][1], INTER_LINEAR);

As a result some of pairs are perfectly correct:

But the rest are either black or broken:

I always get vertical stereo as result of stereoRectify. I tried various combinations of flags and alpha parameters. But most of the pairs don't look correct. What am I doing wrong?

Here is a source code of the program: https://gist.github.com/stiv-yakovenk...