Ask Your Question
0

cv2.BOWImgDescriptorExtractor works with SIFT but not with ORB

asked 2017-11-28 02:59:52 -0500

kama gravatar image

Dear Community

I hope you can help me out with the following question. I trained a vocabulary using cv2.BOWKMeansTrainer. When I try to get the visual word for a descriptor, using cv2.BOWImgDescriptorExtractor, this works fine for SIFT but crashes for ORB.

Relevant Code for SIFT (WORING)

def get_visual_word_histogram_of_images(vocabulary):
sift = cv2.xfeatures2d.SIFT_create()
bow_ext = cv2.BOWImgDescriptorExtractor(sift, cv2.BFMatcher(cv2.NORM_L2))
bow_ext.setVocabulary(vocabulary)

bow_list = []

for filename in listdir(PATH_TO_IMAGES):
    path_to_image = "{}{}".format(PATH_TO_IMAGES, filename)
    image = cv2.imread(path_to_image, 0)
    kp, des = sift.detectAndCompute(image, None)
    histogram = bow_ext.compute(image, kp)[0]
    bow_list.append((histogram, path_to_image))

return bow_list

Relevant Code for ORB (CRASHING)

def get_visual_word_histogram_of_images(vocabulary):
orb = cv2.ORB_create()
bow_ext = cv2.BOWImgDescriptorExtractor(orb, cv2.BFMatcher(cv2.NORM_HAMMING))
# bow_ext = cv2.BOWImgDescriptorExtractor(orb, cv2.BFMatcher(cv2.NORM_L2))
bow_ext.setVocabulary(vocabulary)

bow_list = []

for filename in listdir(PATH_TO_IMAGES):
    path_to_image = "{}{}".format(PATH_TO_IMAGES, filename)
    image = cv2.imread(path_to_image, 1)
    kp, des = orb.detectAndCompute(image, None)
    histogram = bow_ext.compute(image, kp)[0]
    bow_list.append((histogram, path_to_image))

return bow_list

Error Message produced by the ORB Code:

/home/kama/bin/anaconda3/envs/genistat/bin/python /home/kama/genistat/visual_scene_fingerprint/experiments/orb_bow_similarity.py
OpenCV Error: Assertion failed (_queryDescriptors.type() == trainDescType) in knnMatchImpl, file /feedstock_root/build_artefacts/opencv_1506439690741/work/opencv-3.3.0/modules/features2d/src/matchers.cpp, line 744
Traceback (most recent call last):
  File "/home/kama/genistat/visual_scene_fingerprint/experiments/orb_bow_similarity.py", line 89, in <module>
    run()
  File "/home/kama/genistat/visual_scene_fingerprint/experiments/orb_bow_similarity.py", line 75, in run
    bow_list = get_visual_word_histogram_of_images(vocabulary)
  File "/home/kama/genistat/visual_scene_fingerprint/experiments/orb_bow_similarity.py", line 49, in get_visual_word_histogram_of_images
    histogram = bow_ext.compute(image, kp)[0]
cv2.error: /feedstock_root/build_artefacts/opencv_1506439690741/work/opencv-3.3.0/modules/features2d/src/matchers.cpp:744: error: (-215) _queryDescriptors.type() == trainDescType in function knnMatchImpl

Where line 49 is the histogram = bow_ext.compute(image, kp)[0] statement.

The error message indicates, that I used another descriptor type for training. Which I did not... See the following code for training:

def train_vocabulary():
    k_means_trainer = cv2.BOWKMeansTrainer(100)
    for filename in listdir(PATH_TO_IMAGES):
        path_to_image = "{}{}".format(PATH_TO_IMAGES, filename)
        img = cv2.imread(path_to_image, cv2.IMREAD_GRAYSCALE)
        orb = cv2.ORB_create()
        key_points, descriptors = orb.detectAndCompute(img, None)

        if len(key_points) <= 0:
            continue
        descriptors = np.float32(descriptors)

        k_means_trainer.add(descriptors)
    vocabulary = k_means_trainer.cluster()
    return vocabulary

Any Ideas?

Installation: OS: Ubuntu 16.04 Python: 3.6.2 opencv 3.3.0 opencv3 3.1.0

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2017-11-28 03:09:25 -0500

berak gravatar image

updated 2017-11-28 03:40:43 -0500

you CANNOT use binary descriptors, like ORB,BRISK or BRIEF with BoW, it's only possible with float descriptors like SIFT,SURF or KAZE.

descriptors = np.float32(descriptors)

you tried to "cheat" here, by converting the descriptors to float for the BowKMeansTrainer, but think of it: it does not make any sense to convert bitstrings to float for kmeans clustering, they simply do not represent numbers, that can be compared with L2 distance.

this also leads to the error you get, the train descriptors indeed don't have the same type as your test descriptors

to use ORB or similar, it would need KMajority or KMedian clustering, not implemented in opencv.

again, if you want to use a "free" descriptor for this, try again with KAZE (and the FlannBasedMatcher)

edit flag offensive delete link more

Comments

Thanks a lot for your profound answer!

I will do some research before I decide what I will use for my implementation.

kama gravatar imagekama ( 2017-11-28 04:41:45 -0500 )edit

I have the same problem and I would like to know if It is possible to construct a BOVW starting from BRIEF or ORB descriptors with Skitlearn. Thank you in advance for the answer.

rory gravatar imagerory ( 2018-01-29 07:27:15 -0500 )edit

@rory, we cannot really help with scikit here, but you'll either have to find an alternative to kmeans clustering (something, that works with HAMMING, not L2), or use float descriptors.

berak gravatar imageberak ( 2018-01-29 07:38:08 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-28 02:59:52 -0500

Seen: 1,004 times

Last updated: Nov 28 '17