Stitching: camera matrices (R and t) from P=K[R|T]

edit

Hello, I have a process which makes a sparse point cloud from a set of 2D images, basically SfM. The output of this algorithm contains K, R and T matrices for triangulation of 3D points and reprojecting them back into the 2D image.

I now want to produce an orthomosaic as well, so I started to use the stitcher, which uses its own feature detector and bundle adjuster and using a transverse mercator warper I already got quite a decent result.

I would now like to see if I can improve the results by applying the R|T matrix from the sparse point cloud on the stitcher cameras instead. The R|t matrices of that camera isn't the same though, as mentioned in this post:

http://answers.opencv.org/question/26821/a-question-about-relation-of-k-r-t-h/

My question is... how does one convert/map an R|T matrix from a 3D SfM process into a R|t matrix used in the stitcher in opencv code, or is that very involved? The rotation in 3D space (x/y/z) of this sparse point cloud is arbitrary.

The motion estimator.cpp code contains this fragment:

    Mat R = K_from.inv() * pairwise_matches[pair_idx].H.inv() * K_to;
    cameras[edge.to].R = cameras[edge.from].R * R;

which effectively assigns a rotation to a camera with respect to the projection plane. H defines the rotation between two images, so the R for the new camera is a concatenation of the rotation so far concatenated with the new rotation defined by the homography.

The r and t here are from the P=K[R|T] camera model, which is accurately determined. I'm eventually looking for the R and t matrices that are used in the stitcher, which seem different from the R|T above.

25-04-2014 HYPOTHESIS: I figured out a different way for my problem, but would still like to see this question answered. From Robert Collins, CSE486 Penn state, "Projection of Points on Planar Surface":

[x]    [f 0 0 0] [r11 r12 r13 tx] [p]
[y] ~  [0 f 0 0] [r21 r22 r23 ty] [q]
[1]    [0 0 1 0] [r31 r32 r33 tz] [0]
                 [ 0   0   0  1 ] [1]

[x]    [f 0 0 0] [r11 r12 tx] [p]
[y] ~  [0 f 0 0] [r21 r22 ty] [q]
[1]    [0 0 1 0] [r31 r32 tz] [1]
                 [ 0   0  1 ]

[x]    [f 0 0] [r11 r12 tx] [p]
[y] ~  [0 f 0] [r21 r22 ty] [q]
[1]    [0 0 1] [r31 r32 tz] [1]

[x]    [fr11 fr12 ftx] [p]
[y] ~  [fr21 fr22 fty] [q]
[1]    [r31   r32  tz] [1]

[x]    [h11 h12 h13] [p]
[y] ~  [h21 h22 h23] [q]
[1]    [h31 h32 h33] [1]

Which apparently defines the homography.

Which would make this in code:

Mat_<double> H = Mat::eye(3, 3, CV_64F);
H.at<double>(0,0) = f * R.at<double>(0,0);
H.at<double>(0,1) = f * R.at<double>(0,1);
H.at<double>(1,0) = f * R.at<double>(1,0);
H.at ...