Ask Your Question

Evaluation of binary descriptors

asked 2013-06-09 07:30:20 -0500


I'm using the project (EXAMPLE) detector_descriptor_matcher_evaluation to make a comparison of the performance of binary descriptors.

The results that I get basically show that in cases of blur, exposure and jpeg compression SIFT > BRIEF > ORB > BRISK > FREAK > SURF and in viewpoint changes and rotation+zoom SIFR>FREAK>BRIEF>BRISK>SURF>ORB where ">" denotes "outperforms".

Those results are a bit contradicting of my intuition. ORB is an "improved" version of BRIEF where learned pairs and rotation invariance is added, so how come BRIEF outperforms ORB?

In addition, I'm getting very bad results for the "zoom and rotation tests" (bark and boat datasets") – lower than 2% precision for all the image pairs other than (im1,im2). Should I expect such low precision?

Thanks in advance,


edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted

answered 2013-06-09 08:16:37 -0500

Notas gravatar image

How much is each of those algorithms better than the ones before? Might be interesting to know. Anyway, the big problem is the matching. It's ineffective. VERY!

First off, use KNN-matching with two nearest neighbours. Then do the ratiotest, proposed by Lowe, to eliminate possible bad matches. Then use the symmetrytest / crosscheck to further remove bad matches. Lastly, use the epipolargeometry to find the best possible matches. For code, see the OpenCV-cookbook, the class with the code of the tests I wrote about in there is called RobustMatcher.

Simply finding the best match for each keypoint without regarding the geometric relation of the two images and the matches might work if you have VERY distinct keypoints and therefore descriptors, but if you add noise and other changes, it gets hard. And then a simple "find the best match for each keypoint" is not enough.

edit flag offensive delete link more



Thanks for the response. I think that since I'm performing an evaluation of the descriptors distinctiveness ability, I shouldn't use the geometric relation. Of course that it is vital in order to get good correspondences, but that is not the purpose of the evaluation. However, I will use the other suggestion you made.

Thanks again! Gil.

GilLevi gravatar imageGilLevi ( 2013-06-09 10:26:41 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2013-06-09 07:30:20 -0500

Seen: 291 times

Last updated: Jun 09 '13