Ask Your Question

feature matching video stream?

asked 2015-06-23 04:15:09 -0500

stillNovice gravatar image

I am implementing a PnP camera pose estimation. It gives me back coordinates, but on each frame, they jump around a LOT. I believe that this is because the features I use are re-ordering/shifting.

So I have added feature matching between consecutive frames, but I feel like I am missing something.

My workflow:

Store previousFrame. Get currentFrame.

Run feature matching.

Use 2d features of previousFrame to get 3d features, and run PnP.

copy currentFrame to previousFrame and loop again.

I see a similar result, with the coordinates jumping crazily each frame. Is this the right workflow for feature matching? What can I do to improve the stability of my coordinates?


edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2015-06-23 07:43:25 -0500

R.Saracchini gravatar image

I believe, from your posts that you are trying to implement a sort of SLAM solution. This is not an easy task, and I recommend you to read the works of LSD-SLAM (semi-dense approach), and PTAMM (feature-based approach) to have an idea of two very robust (albeit distinct) solutions.

Regarding this particular issue:

Determining the camera pose by feature-matching is extremely jittery. A small amount of camera noise can make a calibration feature "disappear" and affect the solvePNP. This is even aggravated if you are using RANSAC. Keep in mind that RANSAC is non-deterministic algorithm. It will find the solution with least error from a a subset of your input. This means that if a small group of features with a certain error provides a solution with small error (but wrong because their positions are contaminated with noise), your camera pose will "jump".

I fast solution that come to my mind is perform some feature matching and then pass the estimated position of the features as initial guess of an optical flow algorithm. Note that optical flow algorithms are not robust to occlusion, so you will need to make a further check if the tracker feature matches with the original one. Then you can use the input to solvePNP. A better yet solution would use a weighted motion estimator instead solvePNP, so features with low confidence would affect the camera calibration less than properly matched points, but keeping all the information into account. This isn`t implemented in OpenCV, but there is a complete description in a the thesis "Visual Tracking for Augmented Reality" from Georg Klein (Univ. of Cambridge), about how to implement one.

I hope that this helps.

edit flag offensive delete link more


I am indeed! Thanks for the info. I have looked into LSD-SLAM and PTAMM(as well as many others), but need to use a stereo setup so that the world scale is correct. Most of the source that is available is using ROS on linux, and I am on windows, without ROS, so I have started from scratch. It most definitely is not an easy task.

I have implemented optical flow just now, and use the tracked features for the solvePnp function, but I still see too much noise...

I would love to use FoVis

but I cannot get it to run on windows.

stillNovice gravatar imagestillNovice ( 2015-06-23 08:52:03 -0500 )edit

Sadly a lot of good algorithms have a ROS dependency tacked on. ..It is a pain to port such algorithms (I did remove recently the ROS dependency of LSD-SLAM, took one week). Anyway, several of such algorithms have dependencies here and there that makes it not worthy to you just add as library into your code as it will be a huge "Frankenstein" code... for example, the PTAMM has a lot of functionality that is present in OpenCV implemented by a bunch of uncommon libs. It takes a long time, but pays off read the papers and if possible the code, and write the thing yourself.

I really recommend read the thesis above if you have time. It have a very good explanation how to compute the Motion Estimator. I hope that this helps you in your project

R.Saracchini gravatar imageR.Saracchini ( 2015-06-24 09:17:37 -0500 )edit

Hi again, I am just getting back to this project now. Sorry if this is a cheeky request, but I don't suppose you would share your ROS-less LSD-SLAM code? No problem if you don't want to, I just thought I would ask. :) thanks!

stillNovice gravatar imagestillNovice ( 2015-07-02 13:56:57 -0500 )edit

Hi. I'm sorry for the late answer ! I'm quite overwhelmed these months. For sure I would like to share it, I would need to strip out some libraries which I attached, that we use in my research group, and replace with a simple version of it. Still, I'm not sure if you would be able to compile it in a non-Linux platform, and surely making it compile in Visual Studio would be a feat. The thing is tangled with a bunch of boost mutexes and shared pointers...

R.Saracchini gravatar imageR.Saracchini ( 2015-07-15 03:53:42 -0500 )edit

Hi, in your post you mentioned you freed LSD slam from ROS. I am struggling with it for 1 month .. It is really very complicated for me and I am not so experienced with programming.. I would be very very very greatful if you could share the code. Thank you :)

S.Azhar gravatar imageS.Azhar ( 2015-07-22 02:37:59 -0500 )edit

Question Tools



Asked: 2015-06-23 04:15:09 -0500

Seen: 1,075 times

Last updated: Jun 23 '15