Ask Your Question

image coordinate to world coordinate opencv

asked 2015-05-28 04:32:38 -0500

mefmef gravatar image

updated 2015-05-28 04:32:57 -0500

I calibrated my mono camera using opencv. Now I know the camera intrinsic matrix and distortion coefs [K1, K2, P1 ,P2,K3 ,K4, K5, K6] of my camera. Assuming that camera is place in [x, y, z] with [Roll, Pitch, Yaw] rotations. how can I get each pixel in world coordinate when the camera is looking on the floor [z=0].

image description

edit retag flag offensive close merge delete


I'd be interested in this as well. I need to know which ray each point on the image corresponds to. Basically I would apply the answer to your question twice for the front and back plane of the box I look at the connect them. Your question should be answerable with a bit of linear algebra if one neglects the curvature. I'll play around in Mathematica in the meanwhile.

Theoretiker gravatar imageTheoretiker ( 2015-05-28 07:52:44 -0500 )edit

hints: search for pose estimation, solvePnP(), Rodrigues() in addition to image coordinates to world coordinates and opencv...and be prepared for a lot of reading.

theodore gravatar imagetheodore ( 2015-05-28 09:08:40 -0500 )edit

@theodore: As far as I have understood one needs solvePnP() to get the position and angle of the camera in world coordinates. Once one has this, how does this function help? I am currently thinking about taking a random point on the plane and project it to the image and then iteratively see which points is closest to the given image point.

Theoretiker gravatar imageTheoretiker ( 2015-05-28 12:11:01 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2015-05-28 13:56:46 -0500

theodore gravatar image

updated 2015-05-28 13:59:07 -0500

update: I made it an answer since it was not fitting in a comment, and multiple comments with code were ugly and not helping to understand

@Theoretiker you need to obtain the tvec, rvec and rotationMatrix. the first two you get them through the solvePnP() function and the latter through the Rodrigues() function. Once you have this information in addition to the camera calibration coefficients you can transform the pixel point to world coordinates point. The code should be something like that (it is taken from an old project of mine):

Mat intrinsics, distCoeffs;

cv::solvePnP(worldPlane, imagePlane, intrinsics, distCoeffs, rvec, tvec);

cv::Mat uvPoint = cv::Mat::ones(3,1,cv::DataType<double>::type); //u,v,1

// image point<double>(0,0) = 3.; //got this point using mouse callback<double>(1,0) = 134.;

cv::Mat tempMat, tempMat2;
double s, zConst = 0;
tempMat = rotationMatrix.inv() * intrinsics.inv() * uvPoint;
tempMat2 = rotationMatrix.inv() * tvec;
s = zConst +<double>(2,0);
s /=<double>(2,0);
cv::Mat wcPoint = rotationMatrix.inv() * (s * intrinsics.inv() * uvPoint - tvec);

Point3f realPoint(<double>(0, 0),<double>(1, 0),<double>(2, 0)); // point in world coordinates
edit flag offensive delete link more



So you basically invert the pinhole camera equation and find out s such that the ray intersects with the z = 0 plane?

Theoretiker gravatar imageTheoretiker ( 2015-05-28 15:32:58 -0500 )edit

Thanks for your complete answer. Is it possible that instead of using solvepnp I fill rvec and tvec? Based on IMU data?

I have the rotation [Roll, Pitch, Yaw] and transition from the Z=0 plane which is [X, Y, Z].

If it is possible could you briefly tell me how can i fill rvec and tvec?

mefmef gravatar imagemefmef ( 2015-05-29 04:46:39 -0500 )edit

@Theoretiker, something like that. @mefmef I do not know/remember that, the project that I worked with it was 2 years ago, and since then there was not need to work with it again so some of the information/knowledge faded out. But try to search with the keywords that I gave you at the comment in your opening question. I still remember that there was a lot of material, with examples as well, to read about it when I was searching at that point.

theodore gravatar imagetheodore ( 2015-05-29 06:29:42 -0500 )edit

What is these worldPlaneimagePlane two parameters for? And how to determine?

joyyang gravatar imagejoyyang ( 2017-12-04 03:00:32 -0500 )edit


I did this, and also this related article on StackOverflow:

with that same image (its the image described there), and I can't get the positions (realPoints) right!

And of course, when I proyected them back, they are well off.

Also, in :

s = zConst +<double>(2,0);

s /=<double>(2,0);

The "2" is because is z-constant? its the z-pos in the matrix? being z the vertical axis, zeroed at floor?

MikeSZ gravatar imageMikeSZ ( 2018-03-09 14:10:30 -0500 )edit

Question Tools



Asked: 2015-05-28 04:32:38 -0500

Seen: 14,148 times

Last updated: May 28 '15