I'm not well-versed in OpenCV, but I have a working gender classification program, and I am using exactly the same method as you. I still have it at around 75% accuracy, but it can all be improved afterthat. Like what you said, I believe it is returning the most matching face too, but that doesn't mean it can't be believed.
The thing about it is to have a more diverse set of face database, it'll be good to increase the number of faces you have for both genders, with different races, different emotions (or void of any). It'll be good to include a few faces that is unique, for example, faces of male that look feminine, or faces of female that looks boyish. The larger your database, the more diverse it is, I believe it'll return better results.
20 faces for each class is pretty small, many others recommend to have a few hundreds.
There are definitely many other ways to do it, or even better ways, but what you're doing is not wrong either, and there are people who reached up to over 95% of accuracy, so if you don't have a lot of time to learn about a total new algorithm or method, I think it'll be good to stick with Fisherfaces and just add on to your face database. It'll take a lot of time too, but a more diverse and large set of training data would help in any approach you're taking in gender classification I believe.
if you want my 2 cents, i'd say this is the correct way.
what are you unsure about ?
@berak I think it is returning the most matching face. so the result can not be believed. assume I have male face which is matching with any of the female face of training faces up to some extent but not any of the male faces. so it will return female.(This problem is arising for me cz it has recognized many of the male images to female.) I am just asking what modifications should I apply to get better results.
oh, you said before : "it gives correct answer.". what now ? does it predict correctly or not ?
initially i tested with few images and it gave correct answer for many of them but when it was tested with large amount of test images, it gave many wrong results. I am just asking the correct way to detect gender. please suggest the better ways or improvements.