Revision history [back]

This result is not strange at all. SIFT and SURF are not working for binary images, there you'd need to compare the shapes, see e.g. http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?highlight=shape#matchshapes, see also http://answers.opencv.org/question/10763/what-kind-of-features-that-can-be-extracted-from/#10774 for other possibilities. If you have also the grayscale versions of your images then your approach would work fine. For comparing a larger database a BoW-approach could also be useful, see http://answers.opencv.org/question/8677/image-comparison-with-a-database/#8686

EDIT (Answering to your two new questions)

Steven already mentioned it: You may have detected three matches in your first image and 6 images in your second one, now you match all your features of the first one to those of the second -> you'll get the three closest matches. These are the ones you have drawn.
Each classifier needs the training feature vectors in one matrix and also the classes in a response-matrix. So, create first one large matrix (let's say you have 10 images and take always the 20 best features -> 200 rows and each feature has the dimension 128 -> Matrix of 200x128), compute now for each image the features, select the 20 best ones, copy them in your big matrix. This big matrix you can now save/load via FileStorage. The response-matrix has in our example the dimension 200x1, where every feature-row get's its class-label.

Note: This approach of taking the 20 best features won't give you good results! There you have to unify your features somehow. This you can do either with a Bag-of-Words-approach (see my link above) or switching to texture-features (which are basically more primitive features but applied on the whole image (or grid of your image) and typically stored in histograms). Then you pass the BoW-Descriptors or your texture-features to your classifier as described above.