Ask Your Question

OpenCV real_time_pose_estimation not working

asked 2018-12-07 18:29:50 -0600

I am trying to use OpenCV for real-time pose estimation. I found this tutorial and tried to follow it. When running the example_tutorial_pnp_detection, it did not work. There were no error messages, but there was no detection either.

This is my result for the registration step: image description

along with the cookies_ORB.yml file (Whether or not it is accurate I have no clue).

But this is what I got when running the pnp_detection with the cookies_ORB.yml file, the box.ply file, and box.mp4, all of which came with the sample code: image description

What am I doing wrong? I know this question has been asked before with In real_time_pose_estimation( samples given in opencv 3.1.0) no errors but fails to detect the object and How to use real_time_pose_estimation, but neither of those were even answered. If this has already been answered somewhere I'd be more than glad to have the link to the forum.

I am using OpenCV 3.4.3.

edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted

answered 2018-12-08 19:31:06 -0600

Eduardo gravatar image

updated 2018-12-08 19:32:21 -0600

I will try to add more information about the concepts behind the tutorial. This way, you will be able to tweak the code and adapt it to your needs. Probably the tutorial code needs some improvements.

Pose estimation (quick theory)

The main concept is pose estimation. You can find some references in the OpenCV doc:

But the main idea is to estimate the transformation between the object frame and the camera frame:

image description

That is:

image description

The PnP problem involves to solve for the rotation and the translation that minimize the back-projection error between the set of 3D points expressed in the object frame and the corresponding set of projected 2D image points, assuming a calibrated camera.

Pose estimation (application)

Since camera pose estimation needs a set of 3D object points, we need to find a way to compute these coordinates. This is what is done in example_tutorial_pnp_registration:

  • perform a training/learning step
    • load a mesh file of the desired object, that is simply a CAD model file
    • compute the camera pose by manually clicking on some reference points (this is solvePnP()) with the 2D/3D correspondences done manually
    • detect some keypoints on the object and compute the descriptors
    • compute the corresponding 3D object points from the 2D image coordinates, the camera pose and the CAD model of the object (see Möller–Trumbore intersection algorithm)
    • store in a file the 2D coordinates, the keypoints, the descriptors and the corresponding 3D object coordinates

In example_tutorial_pnp_detection, the first steps are roughly:

  • detect the keypoints in the current image and compute the descriptors
  • match the current keypoints with those detected during the training/learning step
  • now we have the 2D/3D correspondences and the camera pose can be estimated with solvePnP()

The tutorial code does also the following things:

  • PnP is computed in a robust RANSAC way
  • pose is filtered with a Kalman filter (when not enough keypoints are matched to perform solvePnP(), the pose can be predicted)
  • and + some other things

Run the tutorial code

I have observed some issues when the ORB features are detected on multiple levels with solvePnP(). I don't know why but this can be workaround by using only one pyramid level:

  Ptr<Feature2D> orb = ORB::create();

This should be done both in example_tutorial_pnp_registration and in example_tutorial_pnp_detection.

You can extract an image of the video and launch example_tutorial_pnp_registration on this image to have a new cookies_ORB.yml file. If you launch example_tutorial_pnp_detection with the newly cookies_ORB.yml file, you should see that it works more or less, depending on the current object view. Indeed, probably when the current object view is too different than the one used during the training/learning step, the matching is not good ... (more)

edit flag offensive delete link more


@Eduardo Thank you very much. When I tried to add


to main_detection and I ran it with the source YAML file cookies_ORB.yml it worked rather well. Now, however, when I added it to main_registration, generated a new YAML file and executed example_tutorial_pnp_detection it was as screwed up as before. What should I do there?

RobotMan gravatar imageRobotMan ( 2018-12-09 11:28:10 -0600 )edit

It should work if you use an image of the cookies box extracted from the video. Normally, with setNLevels(1), ORB features are no more detected on a pyramid scale, which should degrades the matching performance. But this is the only hack I have managed to make it work with solvePnP().

Or maybe it is just that default image is too different?

Some suggestions:

  • to debug, try to draw the keypoints matching as in my linked video, if the matching is too bad, most likeley that solvePnPRansac() will still return garbage
  • try to change the parameters?
  • tweak the code to be able to train/learn on multiple images
  • experiment with other types of features?
Eduardo gravatar imageEduardo ( 2018-12-10 04:57:17 -0600 )edit

Question Tools

1 follower


Asked: 2018-12-07 17:57:46 -0600

Seen: 1,275 times

Last updated: Dec 08 '18