How to get better results with OpenCV face recognition

edit

I'm trying to use OpenCV's face recognition module to recognize 2 faces from a video. I cropped 20 face images of the first subject and 20 face images of the second subject from the video and I use these as my training set.

I've tested Fisherfaces but I'm not getting good results in this program. Sometimes the first subject is classified as the second subject and vice-verse (I mean sometimes it detects the correct lable sometimes it predict incorrectly ), sometimes false detections are classified as one of the two subjects and sometimes other people in the video are classified as one of the two subjects.

How can I improve performance?

I think increasing the face samples may not solve it completely!So what I should do here:

here is a part of my code which check the condition of labels:

if (label == 2){
                    //string text = format("Person is  = %d", label);
                    Pname = "Mehdi";
                }
                else if(label == 3)
                {
                    Pname = "Kourosh";
                }
                else{
                    Pname = "unknown";
                }