What I need is very simple! Given a 2D point in image coordinates and a calibrated camera, I need the 3D projective ray corresponding to this image point. I know the procedure, mathematically at least, but I'm sure I'm missing something very stupid.
My camera is calibrated using calibrateCamera(). As a result, I have the intrinsics K, rvec, tvec and the distortion coefficients. I'm stuck at creating the camera projection matrix P (which is 3x4) so that I can back-project my 2d points, using the pseudo-inverse of P (as mentioned in Multiple View Geometry by Hartley and Zisserman).
How can I correctly create P using the values obtained from calibrateCamera() call? Is there an internal OpenCV function that I can use for creating the camera projection matrix, or better yet, back-projection?
What I have tried:
I tried manually creating the matrix P using the formulation P = K[R | T]. Where R was obtained using cv::Rodrigues(rvec) and T was set to tvec. However, that P matrix was wrong because it produced wrong results even for forward projection of 3D points to 2D coordinates :(
I guess the other confusing thing for me is that I don't know what I'm supposed to do with rvec and tvec. I know they are the camera pose information which convert world coordinates to camera coordinate frame. However, the 3D points that I have are already in camera coordinate frame because they were obtained using a Kinect! So I don't think any additional transformation is needed.
Any help is greatly appreciated.