Ask Your Question

Error "Assertion failed (scn == 3 || scn == 4) in cvtColor" when calculating ORB features on point cloud

asked 2020-11-25 16:33:29 -0500

sigmoid90 gravatar image

updated 2020-11-27 13:01:42 -0500

crackwitz gravatar image

My task is to register image from camera to point cloud of scene using C++. My intuition here is to calculate ORB descriptors on both camera image and point cloud of scene so I can obtain point correspondences btw them and apply SolvePnP function to estimate the camera pose. I following this tutorial. I load the point cloud of scene in PLY format to cv::Mat and try to calculate ORB decriptors on it. Resultant Mat object has single channel. Here is my code:

Mat scene_mat = loadPLYSimple("a_cloud.ply", 1);

vector<KeyPoint> kp1, kp2;
Mat des1, des2;
Ptr<Feature2D> orb = ORB::create();

orb->detectAndCompute(scene_mat, Mat(), kp2, des2);

When run this code I get error

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /home/vladimir/opencv/modules/imgproc/src/color.cpp, line 7564 terminate called after throwing an instance of 'cv::Exception'

What is the problem with this code? Is this correct way to calculate ORB descriptors on point cloud?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2020-11-26 03:11:15 -0500

berak gravatar image

updated 2020-11-26 08:06:40 -0500

bad news is:

you cannot use opencv's feature2d classes with point clouds AT ALL, they expect a dense 2d textured neighbourhood image, not 3d points.

while the pcl library has something for this, you cannot use those features for 2d images

edit flag offensive delete link more


Thank you for reply. But how I can solve this problem of registering a new image to a point cloud of scene? I see that I should use SolvePnP function from OpenCV.

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 04:06:44 -0500 )edit

yea, that's quite difficult. solvePnP only works nicely with known correspondances, e.g. facial landmarks or grid patterns. maybe you can add some markers (e.g. aruco) to your scene ? maybe you can select points manually ?

what's in your point cloud ? does it have texture ? can you render a 2d image from it ?

berak gravatar imageberak ( 2020-11-26 05:12:13 -0500 )edit

I see. I can't add any markers to the scene. I only given point cloud of scene and image from camera. My goal is to obtain camera pose of this image relating to the scene point cloud. My point cloud of scene is textured. What about this tutorial? Seems like they find robust point correspondenses btw 3D model and image and use SolvePnP for pose estimation.

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 06:15:50 -0500 )edit

sure you could try that idea, unfortunately, the tutorial does a little cheat there, the descriptors for the 3d model are taken from a file (cookies_ORB.yml), and it's unclear, how those were obtained / computed

berak gravatar imageberak ( 2020-11-26 07:52:57 -0500 )edit

ok. Is there a way to obtain camera pose for an image if I have camera calibration data?

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 08:07:32 -0500 )edit

what do you mean now, exactly ?

berak gravatar imageberak ( 2020-11-26 09:58:38 -0500 )edit

I have image from camera, 3D point cloud of scene and camera calibration data (i.e. focal length, principal center points of the image etc). The image and point cloud of scene share the same space. I need to find the pose of that image.

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 10:49:58 -0500 )edit

well you need the calibration data to use solvePnP(), still, all previous problems remain the same

berak gravatar imageberak ( 2020-11-26 11:09:30 -0500 )edit

Thank you. I see this approach is the most suitable in my case and I will try to overcome the issues you remind above.

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 11:54:50 -0500 )edit

@berak I have an idea to project point cloud of scene to a plane image and match the query image against that image. Could it work?

sigmoid90 gravatar imagesigmoid90 ( 2020-11-26 14:59:22 -0500 )edit

Question Tools

1 follower


Asked: 2020-11-25 16:33:29 -0500

Seen: 214 times

Last updated: Nov 26 '20