reprojectImageTo3D coordinate system
I found a bunch of questions related to this and I haven't found a fully satisfactory answer. So, my question is:
"In which coordinate system is the point cloud resulting from reprojectImageTo3D?"
My guess is that as its origin is located at the principal point of the left rectified camera since I can map the rectified left image perfectly onto the point cloud. How can I then express the point cloud with respect to the original unrectified camera? My underlying goal being that I want to ask for a given pixel in the original image the corresponding 3d point and for a given 3d point how can I project it into the original image.
cv::remap(),cv::undistort(), andcv::undistortPoints()aren't helpful to you? What you want is an inverse mapping from rectified/undistorted to unrectified/distorted? I think you should be able to write your own function to do that given the rectification/undistortion maps. Better yet, make a pull request? Maybe look at this question.Yep, I think I'll work with the remap functions and stay within the virtual coordinate system. That is for going from a pixel in the image to depth, I'll first undistort/rectify it, then having one-to-one correspondence. Going the other way should also be ok by inversing the map.