Error "Assertion failed (scn == 3 || scn == 4) in cvtColor" when calculating ORB features on point cloud

asked 2020-11-25 16:33:29 -0600

sigmoid90
16 ●1 ●4

updated 2020-11-27 13:01:42 -0600

crackwitz
186 ●2 ●8

My task is to register image from camera to point cloud of scene using C++. My intuition here is to calculate ORB descriptors on both camera image and point cloud of scene so I can obtain point correspondences btw them and apply SolvePnP function to estimate the camera pose. I following this tutorial. I load the point cloud of scene in PLY format to cv::Mat and try to calculate ORB decriptors on it. Resultant Mat object has single channel. Here is my code:

Mat scene_mat = loadPLYSimple("a_cloud.ply", 1);

vector<KeyPoint> kp1, kp2;
Mat des1, des2;
Ptr<Feature2D> orb = ORB::create();

orb->detectAndCompute(scene_mat, Mat(), kp2, des2);

When run this code I get error

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /home/vladimir/opencv/modules/imgproc/src/color.cpp, line 7564 terminate called after throwing an instance of 'cv::Exception'

What is the problem with this code? Is this correct way to calculate ORB descriptors on point cloud?

edit retag flag offensive close merge delete

add a comment

answered 2020-11-26 03:11:15 -0600

berak
32993 ●7 ●81 ●312

updated 2020-11-26 08:06:40 -0600

bad news is:

you cannot use opencv's feature2d classes with point clouds AT ALL, they expect a dense 2d textured neighbourhood image, not 3d points.

while the pcl library has something for this, you cannot use those features for 2d images

edit flag offensive delete link

Comments

Thank you for reply. But how I can solve this problem of registering a new image to a point cloud of scene? I see that I should use SolvePnP function from OpenCV.

sigmoid90 ( 2020-11-26 04:06:44 -0600 )edit

yea, that's quite difficult. solvePnP only works nicely with known correspondances, e.g. facial landmarks or grid patterns. maybe you can add some markers (e.g. aruco) to your scene ? maybe you can select points manually ?

what's in your point cloud ? does it have texture ? can you render a 2d image from it ?

https://github.com/Awesome-Image-Regi...

berak ( 2020-11-26 05:12:13 -0600 )edit

I see. I can't add any markers to the scene. I only given point cloud of scene and image from camera. My goal is to obtain camera pose of this image relating to the scene point cloud. My point cloud of scene is textured. What about this tutorial? Seems like they find robust point correspondenses btw 3D model and image and use SolvePnP for pose estimation.

sigmoid90 ( 2020-11-26 06:15:50 -0600 )edit

sure you could try that idea, unfortunately, the tutorial does a little cheat there, the descriptors for the 3d model are taken from a file (cookies_ORB.yml), and it's unclear, how those were obtained / computed

berak ( 2020-11-26 07:52:57 -0600 )edit

ok. Is there a way to obtain camera pose for an image if I have camera calibration data?

sigmoid90 ( 2020-11-26 08:07:32 -0600 )edit

what do you mean now, exactly ?

berak ( 2020-11-26 09:58:38 -0600 )edit

I have image from camera, 3D point cloud of scene and camera calibration data (i.e. focal length, principal center points of the image etc). The image and point cloud of scene share the same space. I need to find the pose of that image.

sigmoid90 ( 2020-11-26 10:49:58 -0600 )edit

well you need the calibration data to use solvePnP(), still, all previous problems remain the same

berak ( 2020-11-26 11:09:30 -0600 )edit

Thank you. I see this approach is the most suitable in my case and I will try to overcome the issues you remind above.

sigmoid90 ( 2020-11-26 11:54:50 -0600 )edit

@berak I have an idea to project point cloud of scene to a plane image and match the query image against that image. Could it work?

sigmoid90 ( 2020-11-26 14:59:22 -0600 )edit

see more comments

Error "Assertion failed (scn == 3 || scn == 4) in cvtColor" when calculating ORB features on point cloud

1 answer

Comments

Links

Question Tools

Stats

Related questions

Error "Assertion failed (scn == 3 || scn == 4) in cvtColor" when calculating ORB features on point cloud edit

1 answer

Comments

Links

Question Tools

Stats

Related questions

Error "Assertion failed (scn == 3 || scn == 4) in cvtColor" when calculating ORB features on point cloud