Hi, I have a stereo camera setup for which I am doing relative pose estimation of one camera w.r.t the other using the essential matrix. Once I computed the pose, I want to refine it; so I was thinking of triangulating the matched points into the 3D space, reprojecting them backwards and computing the reprojection error. Eventually I would also like to use solvePnP to refine my pose (I was trying bundle adjustment, but the results were awry: possibly because my extrinsics change over time)
The thing I am confused by is what R and t is used by the triangulatePoints() and projectPoints() functions. When I am passing the projection matrices to triangulatePoints(), do I just pass K1[I|0] for the first camera and K2[R|t] for the second? And does projectPoints() require the R and t between the camera and the object or camera1 to camera2? Any suggestions would be helpful.
Thanks!