Unable to predict the face - false positive scenarios

edit

I want to recognize human face using OpenCV library. I have set of users in the system, to start with I have single image of most of the users. I want to first train the system with the set of images. Then I would give some images to predict the given user. Some input users are existing & it should recognize them. Also it should not recognize the user for which it is not trained so far.

I am using Open CV - C# 3.4.1

Following is my sample code.

// Predict the given user, problem is it identifies user for which it is even not trained for with the high threshold value :( :(
public void Predict(string imagePath)
        {
            Console.WriteLine("-----------------------------");
            Image<Gray, byte> detect_image = new Image<Gray, byte>(imagePath);
            PredictionResult result = recognizer.Predict(detect_image);
            Console.WriteLine(Path.GetFileName(imagePath) +" : " + result.Distance + "************" + result.Label);

            Console.WriteLine("-----------------------------");
        }

         public void Train()
        {
            // I tried all three recognizer one by one, FisherFaceRecognizer is the closest one still it give false positive results

            //recognizer = new LBPHFaceRecognizer(1, 8, 8, 8, 100);
            recognizer = new FisherFaceRecognizer(0, 3000);
            //recognizer = new EigenFaceRecognizer(80, double.PositiveInfinity);

            List<Image<Gray, byte>> sampleImages = new List<Image<Gray, byte>>();

            string baseDirectory = @"\data\trainingdata\";
            string[] files = Directory.GetFiles(baseDirectory, "*.bmp");

            int count = 1;
            List<Image<Gray, byte>> images = new List<Image<Gray, byte>>();
            List<int> labels = new List<int>();

            foreach (string file in files)
            {
                string[] str = Path.GetFileNameWithoutExtension(file).Split('-');

                int label = Int32.Parse(str[2]);

                images.Add(new Image<Gray, byte>(file));
                labels.Add(label);

                Console.WriteLine(count++ + " : Adding Label for " + Path.GetFileName(file) + " - " + label);
            }

            recognizer.Train(images.ToArray(), labels.ToArray());

        }

I used following code to create the training set.

     public static List<Rectangle> FindFaces(IInputArray image)

{ List<rectangle> faces = null; using (InputArray iaImage = image.GetInputArray()) { faces = new List<rectangle>();

    //Read the HaarCascade objects
    using (CascadeClassifier face = new CascadeClassifier(HaarcascadeFrontalFaceDefaultPath))
    {
        using (UMat ugray = new UMat())
        {
            CvInvoke.CvtColor(image, ugray, Emgu.CV.CvEnum.ColorConversion.Bgr2Gray);

            //normalizes brightness and increases contrast of the image
            CvInvoke.EqualizeHist(ugray, ugray);

            Rectangle[] facesDetected = face.DetectMultiScale(
               ugray,
               1.2,
               10,
               new Size(20, 20));

            Image<Bgr, Byte> currentFrame = new Image<Bgr, Byte>(iaImage.GetSize());
            currentFrame.Bytes = iaImage.GetUMat().Bytes; //your byte array

            //Action for each element detected
            for (int i = 0; i < facesDetected.Length; i++)// (Rectangle face_found in facesDetected)
            {
                Image<Gray, byte> result = currentFrame.Copy(facesDetected[i]).Convert<Gray, byte>().Resize(100, 100, Emgu.CV.CvEnum.Inter.Cubic);
                result._EqualizeHist();
                result.ToBitmap().Save(destinationLocation + "-" + i + ".jpg");
            }

            faces.AddRange(facesDetected);
            Console.WriteLine("Total face detected : " + faces.Count);

        }
    }
}
return faces;

}

User inputs: There are five users user1,user2,user3,user4,user5, I successfully detected the their individual faces using FindFaces method. The used the same to predict it. I manually changed the out file name to train the system. Now when invoke Predict() method with set of images then sometime it gives false positive and false negative results.

Can anyone please tell me what is wrong with the above code? how to increase the recognition accuracy. Why it is reporting false positive cases.