3D reconstruction with known camera parameters

edit

Thanks for the link, sounds interesting!

Dear @Tetragram thanks for your work on the mapping3d module. I find it very interesting. But, I still cannot reach to find a way to make it working, expecially the calcObjectPosition function. Could you please paste an example of usage? In particular, what are the shapes and details of the input parameters? It seems that following the doc is not useful, I still get into some errors (different shapes of inputs, Mat types, etc.) Please, it will be very helpful!

What you need to know is that you have N cameras with known parameters, and M points on an object.

• imagePointsPerView is a vector with M elements, each of which is a vector with N cv::Point2f. So imagePointsPerView[m][n] is the pixel location of the mth point in the nth camera.
• objectPoints is a vector of M cv::Point3f. objectPoints[m] is the 3d location of the mth object point, in a reference model. Just like you would input to solvePnP to find the R|t of the camera, except this function finds the R|t of the object now, relative to the model.
• cameraMatrices is either a vector of N cameraMatrices, or a single camera matrix, just like you get out of calibrate camera. You can use a single camera matrix if your cameras are all the same, or actually the same.

• distortionMatrices is the same as cameraMatrices, just the distortion matrix.
  • tvecs is a vector of N cv::Mat, the location of the cameras, in the same coordinate system as the model. Just like you get from solvePnP
  • rvecs is the same as tvecs.
  • tvecObject and rvecObject are the outputs, both cv::Mat, just like you get from solvePnP

A trivial example is to set up your cameras and calibrate them (camera matrix and distortion), and place a Charuco board in the center of your space. Use estimatePoseCharucoBoard to get all the camera tvecs and rvecs.

You'll need to do a bit of work to get the objectPoints vector, but it's there in charuco.cpp for an example.

Then when you move the board, you detect the corners the same way you do for estimatePoseCharucoBoard.

The result is the rotation and translation of the board relative to where you set it originally.

Dear @Tetragramm, thanks for the detailed explanation. It is actually what I was trying to do, my settings are exactly the ones you detailed. imagePointsPerView is a NxM vector<vector<cv::point2f>>, objectPoints is a M vector<cv::point3f> and so on. But it continues to go into OpenCV Error: Sizes of input arguments do not match. It is basically due to wrong sizes of both tvec or the object model point. moreover I had to comment the line 196 in positionCalc.cpp, because it continued to run in the cv assertion error CV_Assert( ( tvecs.size() == pts.checkVector( 2, CV_64F, true ) ) ); but my elements in tvecs (1 for each camera pose) are both CV_64F elements arranged as 3x1 cv::Mat. So I cannot figure out while this assertion returns false.

Line 196 is, I believe, correct. It verifies that there are as many elements of the vector<cv::mat> named tvecs as there are of pts.

Which shows me where my comment was incorrect. I will post the correction here, then edit the original to help anyone who finds this with a search.

• imagePointsPerView is a vector with M elements, each of which is a vector with N cv::Point2f. So imagePointsPerView[m][n] is the pixel location of the mth point in the nth camera. Negative valued points are ignored, as they cannot be real pixels.

This reverses n and m from what I originally said. My apologies. I did this to make it easier to discard model points if they don't have enough views.

Dear @Tetragramm thanks for your R&D and contributions. Now a few years later, I just wanted to check in with you on whether you see your mapping3D package as still the best way to go here. Basically, aiming to do enable 3D reconstruction from ChArUco coordinates. I also see that a pipeline could be developed that uses ChArUco sub-pixel localizations as input to the OpenCV Camera Motion module in the Structure from Motion branch. I think both packages could potentially deliver the matrices needed to be able to transform point clouds from camera coordinate system into a joint shared world coordinate system, but maybe you could give an update / comment on any tradeoffs + (e.g.) if mapping3d works w/ Open CV 4?