I'm using OpenCV and have a ray as a vector in world coordinates. I want to figure out the point in world coordinates where this ray intersects my computer screen. How do I figure out the location of my computer screen in world coordinates?
I'm moving this to an answer so I can write longer.
Here is the setup you should be using. With the "real" chessboard being flatter than that one, I just grabbed it. Oh, and the picture on the screen should be shown in full screen mode so the corner of the board and screen match.
The reason you need to do it this way is because when you find the location of the chessboard, it finds the position of the top left corner of the chessboard and the orientation of the plane of the chessboard. Since you are trying to find the top left corner of the screen and the plane of the screen, you need them to match up. So you put the screen chessboard flat on the screen with it's corner in the corner of the screen.
So you have two cameras and two chessboards. You should take a picture from both cameras with both chessboards in the same spot. For better results, move the secondary camera (the one taking this picture) around and average the results.
From these pictures you will get three transformations.
What you want to find is from primary camera to screen chessboard. So you need to reverse the transformation from secondary camera to real chessboard, and now you have the three you need. primary->real->secondary->screen. And from that, obviously, you can get the inverse, if you need it.
EDIT: Code I used.
//Secondary Camera to Space Chessboard
<Snip Calculation>
Rodrigues(rvecs[0], R2);
tvecs[0].copyTo(T2);
//Secondary Camera to Screen Chessboard
Rodrigues(rvecs[0], R3);
R3 = R3.t();
T3 = -R3*tvecs[0];
//Primary Camera to Space Chessboard
flip(image, image, 1);
Rodrigues(rvecs[0], R1);
R1 = R1.t();
T1 = -R1*tvecs[0];
Mat translation(3, 1, CV_64F);
translation.setTo(0);
translation = R3*translation + T3;
translation = R2*translation + T2;
translation = R1*translation + T1;
This gives this, which is a little off, but probably because I only used one image for secondary camera and got bad camera matrix and distortion. The primary camera needs more images too, but I mentioned that below.
[-96.78414539354907;
-471.6594258771706;
16.61072172695242]
Thank you for that detailed answer, I am close and just have two small questions before I can finish this calibration.
findChessBoardCorners() only gets the internal corners of the screen. If the corner of the chessboard and the screen should match, how can I get that top right corner?
Once I've rotated and translated to get the first corner of the screen, what's the most precise way of getting the other 2 corners? I know the length in mm of a side of one of the chessboard squares. Do I divide that by the length in pixels to get a ratio, and then multiply it by 2880? (my screen is 2880x1800)
You know the size of a square, subtract one.
And yes, that is how I would find the other corners.
Thank you for that, hopefully this is my last question about the rotation and translation.
I'm confused as to how to apply the rvec and tvec to get the transformation. Do I just do (0,0,0)*rvec + tvec. If this is the case, how do I invert the transformation, as I'll need this for something else as well?
After doing research online I realized I may have to use the Rodrigues() function to get the rotation matrix from rvec, augment that with the tvec, and then add a row of (0,0,0,1) to get a 4x4 transformation matrix. This way, I would be able to get the inverse. However, If that is the case, how do I multiply that by (0,0,0)? Am I just supposed to do (0,0,0,1)*(rotm|tvec, 0,0,0,1)?
I'm not sure which of ...
Invert like this:
Rodrigues(rvec, R);
R = R.t();
t = (-R * tvec);
Same results, easier to understand and use.
I’ve performed the calculations, and even after verifying the math independently, my results are quite far off.
Using a tape measure, I obtained (-165,5,0) as my upper left screen corner (in mm with origin as camera).
If I stay consistent with chessboard units (i.e. passing (0,0,0), (1,0,0) etc)) into calibrateCamera() as objectPoints, feeding (0,0,0) into the transform yields (-17,18,100) as the top left corner.
If I use mm as objectPoints for calibrateCamera, noting that my real chessboard has a square length 50 mm and screen chessboard has length 20 mm, it yields (-32,846,2114) as the top left corner.
Ok, so, step one for debugging. Take each image, find the rvecs and tvecs for each chessboard and use drawAxis from the aruco module to verify that worked.
Secondly, use the code from that tutorial to draw the detected points on the image. Make sure it's finding the right ones, and it's got the right corner as (0,0).
Thirdly, both chessboards need to be in the same units. So if one is in mm, the other needs to be in mm also. Otherwise your coordinates aren't going to match up.
I've kept the units consistent and done as you've said. The code finds the chessboard points, and draws the axes correct. Although I'm not sure about the blue one since I'm not familiar with the drawAxis function, maybe you can verify.
If you have any idea what might be wrong, or how to proceed, please let me know. Thank you!
All right, I see the problem. For your first and second pics of the axes, you can see that the axes are not representing the same thing. They are both in a different place, and rotated differently. So when you combine your transformations, you are assuming a 90 degree rotation and a step to the side that doesn't exist, so of course you get the wrong results.
Two ways to fix it. One: Alter your second picture's rvecs and tvecs so it aligns with the first. This will involve adding PI/2 to one of the rvec elements and 6*square mm to one of the tvecs (I think, may need to rotate that first).
Two: re-arrange the world points so that they line up the same as in the first picture. It's patterned, so it should be pretty easy to re-arrange.
I made sure all 3 chessboards had the same axes, but actually rotated the first and third to match the second, as the second had x and y axes that matched my world system x and y, and its origin was closest to the top left corner of the chessboard.
However, my results are still far off from the tape measured (-165,5,0), as I’m getting (294.49, -60.72, 2030.44) after passing (0,0,0) and translating to the top left corner.
Any ideas what might be off? Is it the axes? The z axis of my world system is the opposite of that on the pics, but I don’t think that should affect the results like this. I’ve included updated pics below.
Is one of those mirrored? The chessboard on the chair is backwards in the two pictures.
Also, both of the cameras are calibrated with camera matrix and distortion matrix?
Asked: 2016-04-05 17:59:29 -0600
Seen: 1,894 times
Last updated: Jun 12 '16
Could you add a little more information? Do you mean you are displaying an image and what to know where in the image the ray is? Or is the ray coming from something in front of the screen like an eye? Or what?
I have a video of a person and a gaze vector originating from that person's eye. The center of the eye and gaze vector are in 3d world coordinates. I want to figure out where the user is looking, so consequently where the gaze ray intersects with the computer screen..
Alright, first step, do you know the coordinates of the corners of the screen in world coordinates? That's step one.
No, that's where I'm struggling. I'm trying to figure out how to get those so that I can construct a plane and intersect the gaze vector (or ray) with it. Any ideas/pointers would be much appreciated!
Ok, Where is the origin of your world system? The camera? There are a couple of ways to do this. One is to carefully align the camera with the screen and measure the distance to the corners.
If you have another camera, you can do something more precise. Place a chessboard at a forty five degree angle in front of your primary camera and find the transformation from world coordinates to chessboard coordinates. Then use the second camera to take a picture (or pictures) with the first chessboard, and a chessboard pattern on the screen. You can then find the location of the screen in the chessboard coordinates and translate that to world coordinates.
Are you comfortable enough with the transformations to do that?
The camera is the origin of my world system. If I'm using a laptop with a built in webcam, the z coordinate of the screen corners would be 0, correct?
For the second method, I'm assuming you're referring to the findChessBoardCorners() function OpenCV has. But I'm confused as to how the second camera's picture would help determine the location of the screen.
If you're using a laptop, then yes, you can likely assume zero. At least for the top corners. If the focal plane of the camera is tilted with respect to the screen the bottom corners could be pretty far off.
The calibrateCamera function returns the rvec and tvec of the camera with respect to the chessboard. So once you use multiple cameras and a chessboard in space and on the screen you have Primary Camera -> Chessboard in Space and Secondary Camera -> Chessboard in Space and Secondary Camera -> Chessboard on screen. Just invert the second one, and then apply them all in sequence to get Primary Camera -> Chessboard on Screen.
Thanks! Could you just clarify how I would get the bottom corners using the first method? I have the intrinsic camera matrix.
Honestly, I can't think of a way without simply carefully measuring.
Wait, I have an idea. Get a mirror and set it up so that the camera is looking straight back into itself. Then show a chessboard on the screen and get the rotation and translation. Then measure the distance from mirror to camera, and you have everything you need. This is a bit hard to hold the mirror in place, but not impossible. Set the laptop in front of the bathroom mirror and carefully arrange the screen.
How would getting the rvecs and tvecs help with the screen corners? I'm still a bit confused
One other point came up. I have fx, fy in pixel units. Is there a conversion to mm? I've looked this up but the answers have been inconsistent.