Keypoint and descriptor ordering - C++ vs Android

asked 2014-10-20 15:52:27 -0600

I'm trying to understand differences I'm seeing between the C++ and Android implementations of FeatureDetector and DescriptorExtractor in OpenCV 2.4.9.

Fundamentally I'm trying to generate a descriptor matrix for an image in C++, encode it in JSON, pass it to an Android app, and use that descriptor matrix in the DescriptorMatcher.match() method for image recognition.

I'm using ORB for both feature detection and descriptor extraction in both C++ and Android.

What I'm finding is that the C++ descriptor extractor generates the same descriptor matrix as the Android implementation but with the order of the rows completely scrambled. When decoding this matrix from JSON in Android the match() method doesn't find matches.

I also looked at the keypoints generated by the FeatureDetector on both sides. Again, the actual values of the rows in the generated Mat structure are the same, but the rows are in different order. And this order seems to drive the DescriptorExtractor to generate a different order for the descriptor matrix rows.

First question: does the order of the rows in a descriptor matrix fundamentally matter to the DescriptorMatcher.match() method? It appears it does, but I can't find any documentation that says it should.

Second question: does the order of keypoints from the FeatureDetector matter to the DescriptorExtractor.compute() method? Again, empirically, it seems so but I can't find any documentation that states it explicitly.

Final question: Why would the C++ feature detector return keypoints in a different order for the exact same image? I've verified that the image, in Mat structure, is exactly the same in both implementations.

In both C++ and Android I'm using the default create() methods for the detectors and extractors so I would expect the objects to be initialized the same way. Is this a faulty assumption? I can't seem to directly inspect the detector or extractor in Android to verify this.

If I take the descriptor matrix generated in Android and encode it into JSON (using Base64 encoding for the data element), and pass that into the Android app instead of the JSON-encoded descriptor matrix created by C++, the descriptor matcher works. This tells me the issue isn't with the JSON or Base64 encoding.

If anyone has suggestions on where to look next I'd be much obliged.

Thanks,

Chris

edit retag flag offensive close merge delete

add a comment

answered 2014-10-21 04:24:10 -0600

StevenPuttemans

20029 ●16 ●82 ●207 http://stevenputtemans...

First of all let me tell you that it seems there is something going wrong in the wrapping of the C++ interface. Indeed one would expect the complete interface to output the exact same results, else there is no reason in creating wrapper functionality. A big chance thus that this is indeed a bug. To answer your questions:

The order of rows, so basically the order of points should not interfere with the end result as long as you perform a brute force matching of your keypoints against eachother which is a 1 versus all approach. Looking at the description of the match function I would expect that this is the case, since each point given is checked against the known descriptor database of the training set.

Same as above. If you have an identical image or apply the featuredetector multiple times on the same image, the order of features should be identical. However if a slight movement is in place, then it is possible the order is changed, but by a brute force matching, this shouldn't create any problem at all.

As I said, I am guessing a bug in how the function gets wrapped. However, you need someome with Java and Android experience for that. @berak, you maybe got a clue on what is happening here?

edit flag offensive delete link

Comments

Thanks @StevenPuttemans for correcting me , well as final question is concern isn't it depend on the size of the image ? like if you have same two images but one size is bigger than other then it may return keypoint in different order ?

FLY ( 2014-10-21 05:18:42 -0600 )edit

Size of image will also add a different order of keypoints. That is normal because you have more information available.

StevenPuttemans ( 2014-10-21 07:02:31 -0600 )edit

Thanks @StevenPuttemans for the information. The exact same image, in OpenCV Mat format, is passed to the FeatureDetector in both C++ and Android, and the exact same set of keypoints and descriptors is generated.

We are using the BRUTEFORCE_HAMMING DescriptorMatcher, so based on what you said it would seem that even with the descriptors in a different order the matches should be made between the training image and the test image.

@berak, if you have any suggestions on the Android side for debugging this I'd be very appreciative.

testflyjets ( 2014-10-21 12:08:57 -0600 )edit

I did a small experiment and took the descriptor matrix generated on Android and shuffled all the rows randomly, then used that matrix to try to register the target image. It doesn't work, just as the descriptor matrix generated in C++ has a different row ordering and also doesn't work.

So it seems the order of rows in the descriptor matrix does matter in terms of calling .match().

I also noted that there is a 1:1 correspondence between the rows in the keypoints and the generated descriptor matrix rows. That is, for each keypoint generated on Android, the same feature descriptor row is generated on Android and C++. So if the keypoints are in a different order in C++, the descriptor matrix rows will be in the same, different order.

testflyjets ( 2014-10-21 16:47:35 -0600 )edit

I did another experiment on Android. I wrote a native method using JNI to call the FeatureDetector.detect() method and examined the keypoints generated. They are the same keypoints generated by the Android Java method, so at least those two are consistent.

testflyjets ( 2014-10-21 18:12:59 -0600 )edit

add a comment

Keypoint and descriptor ordering - C++ vs Android

1 answer

Comments

Links

Question Tools

Stats

Related questions

Keypoint and descriptor ordering - C++ vs Android edit

1 answer

Comments

Links

Question Tools

Stats

Related questions

Keypoint and descriptor ordering - C++ vs Android