solvePnP fails in a specific case

asked 2016-02-19 21:21:42 -0600

saihv gravatar image

I am using OpenCV's solvePnPRansac function to estimate the pose of my camera given a pointcloud made from tracked features. My pipeline starts with forming a point cloud from matched features between two cameras, and use that as a reference to estimate the pose of one of the cameras as it starts moving. I have tested this in multiple settings and it works as long as there are enough features to track while the camera is in motion.

Strangely, during a test I did today, I encountered a failure case where solvePnP would just return junk values all the time. What's confusing here is that in this data set, my point cloud is much denser, it's reconstructed pretty accurately from the two views, the tracked number of points (currently visible features vs. features in the point cloud) at any given time was much higher than what I usually have, so theoretically it should have been a breeze for solvePnP, yet it fails terribly. I tried with CV_ITERATIVE, CV_EPNP and even the non RANSAC version of solvePnP. I was just wondering if I am missing something basic here? The scene I am looking at can be seen in these images (image 1 is the scene and feature matches between two perspectives, image 2 is the point cloud for reference)

Feature matches

Point cloud

edit retag flag offensive close merge delete

Comments

From my limited knowledge on PnP problem, failure cases could be due to:

  • singular points configuration (as there are lot of points, this should not be the problem I think)
  • convergence to a local minima as it is a non linear problem

What I would do is:

  • test with cvPosit (see here)
  • test with an initial guess (useExtrinsicGuess parameter) for the pose with CV_ITERATIVE and see if it converges at some point
  • compare the OpenCV EPnP with the original one, to spot an issue in OpenCV
  • test with other methods / implementations: OpenGV, to spot a bug or a bad handling of singular cases in OpenCV
Eduardo gravatar imageEduardo ( 2016-02-20 08:56:02 -0600 )edit

Thanks for the ideas, Eduardo. I am trying them out and will let you know if I have any luck. Just one omore question, you mentioned that the singular points config. would be a failure case, which I am assuming is due to most of them being coplanar? When I was first developing this pipeline, I was running some tests using a simulator, where I could place models of buildings and look at it through simulated cameras. In that case, the cloud was almost flat, yet PNP worked perfectly even without initial estimates. Although the number of points I had in those point clouds were upwards of a 1000. Do you see anything strange in that case?

saihv gravatar imagesaihv ( 2016-02-20 14:27:55 -0600 )edit

Coplanar 3D object points are not a problem since we can detect this configuration and use the appropriate method (for example coplanar posit). You can easily estimate the pose of a chessboard pattern with solvePnP for example or when you calibrate a camera.

An example of singular configuration would be a pose estimation with 4 points and on these 4 points, 3 points are collinear in the image plane.

It could be great if you could check if the pose estimation fails because of the point configuration / a limitation in the methods or an issue in OpenCV.

Eduardo gravatar imageEduardo ( 2016-02-20 18:57:35 -0600 )edit

An update:

  1. I think I got confused because I was trying a bunch of different things when I mentioned it in the question, CV_EPNP did work when I tried now with solvePnPRansac. The accuracy is not as good as what CV_ITERATIVE used to give me with other datasets though. I have a Kalman filter running alongside, so I am trying to see if I can tweak it to make the estimates better.

  2. I grabbed a few more datasets today of the same scene with my reference at different baselines etc. CV_ITERATIVE still fails, RANSAC or not. If I give it an initial estimate of the extrinsics, it just returns [0 0 0]. POSIT fails too.

  3. Right next to where I was, there's a big set of solar panels like a ceiling(picture linked under). If I use that scene, ITERATIVE works again.

http://imgur.com/a/FP6Y

saihv gravatar imagesaihv ( 2016-02-20 19:07:58 -0600 )edit
add a comment