When obtaining the camera calibration parameters using calibrateCamera I get back two C++ vectors of matrices tvecs and rvecs. It seems that these are the extrinsic parameters of the camera, i.e. the parameters that describe the location and rotation of the camera. If I'm understanding it correctly, the rotation matrix rotates the scene so that the optical axis of the camera is aligned parallel to the Z-axis and the translation vector then moves the scene by the negative camera position vector, so that it is located at the origin. Am I understanding this correctly?
Bonus question: Is it possible to reproject (u, v) points from the camera image back to the space? It seems to be impossible, because cameraMatrix performs a central projection so that the depth information is lost. I could merely try to reproject the intersection of the optical axis and the test pattern, correct? Like this:
Rodrigues(rvec)^(-1) · Mat(tvec)^{-1} · cameraMatrix^{-1} · (c_x, c_y, 1)