Projecting point cloud to image plane in order to estimate camera pose

asked 2020-11-27 05:36:50 -0600

sigmoid90 gravatar image

updated 2020-11-30 03:48:20 -0600

I am working on a task of image registration in point cloud of scene. I have an image from camera, 3D point cloud of scene and camera calibration data (i.e. focal length). The image and point cloud of scene share the same space. Here is a snapshot of my point cloud of scene

image description

and query image

image description

From my previous question here I have learnt that the SolvePnP method from OpenCV would work in this case but there is no obvious way to find robust correspondences btw query image and scene point cloud.

My intuition now is to project point cloud to image plane, match keypoints calculated on it against ones calculated on query image and use robust point correspondences in the SolvePnP method to get camera pose. I have found here a way to project point cloud to image plane.

My questions is could it work and is it possible to preserve the transformation btw the resultant image of scene and the original point cloud?

edit retag flag offensive close merge delete


to speed this up a bit, do you have any other means to render the cloud to an image, like meshlab, blender, etc ?

berak gravatar imageberak ( 2020-11-27 10:52:31 -0600 )edit

Do you have pointcloud information with intensity (pcl::PointXYZI) or RGB (e.g. pcl::PointXYZRGB) data?

If you have at least intensity information, indeed you should be able to:

  • project the pointcloud using an intrinsics matrix
  • match keypoints between camera image and "pointcloud image", use something like the Lowe's ratio test to filter bad matches
  • use solvePnP to estimate the relative transformation ([R | t]) between the color camera and the depth camera (2D is from the color camera, 3D from the pointcloud)
Eduardo gravatar imageEduardo ( 2020-11-28 04:00:54 -0600 )edit

Without at least intensity information for the pointcloud, the only common information between image and pointcloud should be object contour information since the pointcloud encodes spatial structure of the scene.

Keypoints would not be useful here because they need texture information and your pointcloud don't have that (when you project your raw pointcloud into the image plane, the intensity is then the depth value).

Eduardo gravatar imageEduardo ( 2020-11-28 04:23:25 -0600 )edit

@Eduardo, see the other question for a better pointcloud description

berak gravatar imageberak ( 2020-11-28 04:59:19 -0600 )edit

@berak I have Meshlab. Is it possible to render the cloud to an image using meshlab?

sigmoid90 gravatar imagesigmoid90 ( 2020-11-29 16:37:20 -0600 )edit

file -> save snapshot

just try and add it to your question along with your other 2d image

in the meantime, we can work out a better way to read your ply into opencv

berak gravatar imageberak ( 2020-11-30 02:46:10 -0600 )edit

I added a snapshot of my scene point cloud and other 2d image.

sigmoid90 gravatar imagesigmoid90 ( 2020-11-30 03:49:10 -0600 )edit

^^ well done, thanks ;)

unfortunately, this already shows some problems:

  • the 3d point cloud contains far more, than your 2d image, e.g. the back wall. this might even obscure the view you have in 2d (you could probably move the camera to the same viewpoint, your 2d image was taken from, but then you have one more transformation in the way)

  • it is probably far too sparse, to be rendered into an image useful for 2d feature matching (yea, you could try to use larger points for rendering, but hmmm, a lot of "holes" in there)

even if you can read in the point cloud from opencv, and do your own projection / rendering -- the problems will be the same.

btw, that's a real nice cloud, how did you acquire that ?

and btw, what's the purpose of all of this ?

berak gravatar imageberak ( 2020-11-30 04:53:54 -0600 )edit

Thank you for answer. It's a reconstructed point cloud out of photos from phone camera. My task is to recover the camera pose of capturing the query image. I can ask for another scene cloud in obj format if it helps. Maybe use PCL library to obtain image projection of the point cloud? Or something else?

sigmoid90 gravatar imagesigmoid90 ( 2020-11-30 05:32:02 -0600 )edit

It's a reconstructed point cloud out of photos from phone camera.

so, some "structure from motion" approach ? tell us more, this is important (or at least, interesting) !

Maybe use PCL library to obtain image projection of the point cloud?

sure PCL has support for rendering images (via VTK) also, opencv_contrib has a VIZ(also vtk based) and an oviz(using ogre) module to render point clouds (but i never used either)

berak gravatar imageberak ( 2020-11-30 05:54:57 -0600 )edit