**update:** I made it an answer since it was not fitting in a comment, and multiple comments with code were ugly and not helping to understand

@Theoretiker you need to obtain the `tvec`

, `rvec`

and `rotationMatrix`

. the first two you get them through the `solvePnP()`

function and the latter through the `Rodrigues()`

function. Once you have this information in addition to the camera calibration coefficients you can transform the pixel point to world coordinates point. The code should be something like that (it is taken from an old project of mine):

```
Mat intrinsics, distCoeffs;
rvec.create(1,3,cv::DataType<double>::type);
tvec.create(1,3,cv::DataType<double>::type);
rotationMatrix.create(3,3,cv::DataType<double>::type);
cv::solvePnP(worldPlane, imagePlane, intrinsics, distCoeffs, rvec, tvec);
cv::Rodrigues(rvec,rotationMatrix);
cv::Mat uvPoint = cv::Mat::ones(3,1,cv::DataType<double>::type); //u,v,1
// image point
uvPoint.at<double>(0,0) = 3.; //got this point using mouse callback
uvPoint.at<double>(1,0) = 134.;
cv::Mat tempMat, tempMat2;
double s, zConst = 0;
tempMat = rotationMatrix.inv() * intrinsics.inv() * uvPoint;
tempMat2 = rotationMatrix.inv() * tvec;
s = zConst + tempMat2.at<double>(2,0);
s /= tempMat.at<double>(2,0);
cv::Mat wcPoint = rotationMatrix.inv() * (s * intrinsics.inv() * uvPoint - tvec);
Point3f realPoint(wcPoint.at<double>(0, 0), wcPoint.at<double>(1, 0), wcPoint.at<double>(2, 0)); // point in world coordinates
```

I'd be interested in this as well. I need to know which ray each point on the image corresponds to. Basically I would apply the answer to your question twice for the front and back plane of the box I look at the connect them. Your question should be answerable with a bit of linear algebra if one neglects the curvature. I'll play around in Mathematica in the meanwhile.

hints:search for pose estimation, solvePnP(), Rodrigues() in addition to image coordinates to world coordinates and opencv...and be prepared for a lot of reading.@theodore: As far as I have understood one needs

`solvePnP()`

to get the position and angle of the camera in world coordinates. Once one has this, how does this function help? I am currently thinking about taking a random point on the plane and project it to the image and then iteratively see which points is closest to the given image point.