Head pose estimation with OpenCV java

asked 2017-05-25 05:07:49 -0600

myCode gravatar image

I am trying to do head pose estimation with OpenCV java. I am trying to follow http://free-d.org/3d-head-tracking-in... tutorial

now for far i can track the face but how can i use the algorithm once the face is tracked

here are my codes

main:

public class App {

    private JFrame frame;
    private JLabel label;


    private void mainGUI(){
        frame= new JFrame("The Camera");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setSize(300,300);
        label=new JLabel();
        frame.add(label);
        frame.setVisible(true);
    }


    /**** Run main Loop  for face detection and all other good stuff ****/
    private void runLoop(){

        String frontFacePath= "/opencv/data/haarcascades/haarcascade_frontalface_alt.xml";


        CascadeClassifier faceCascade = new CascadeClassifier(frontFacePath);


        Mat webCamImg = new Mat();
        Image tmpImg;
        VideoCapture capture = new VideoCapture(0);

        double fps = capture.get(Videoio.CAP_PROP_FPS);
        System.out.println(fps);

        capture.set(Videoio.CAP_PROP_FRAME_WIDTH,300);
        capture.set(Videoio.CAP_PROP_FRAME_HEIGHT,300);
        ConvertImage convertImage = new ConvertImage();


        if (capture.isOpened()){
            while (true){
                capture.read(webCamImg);

                MatOfRect faceRect = new MatOfRect();
                MatOfRect eyeRect = new MatOfRect();
                MatOfRect faceProfile = new MatOfRect();

                faceCascade.detectMultiScale(webCamImg,faceRect);
                eyeCascade.detectMultiScale(webCamImg,eyeRect);
                profileCascade.detectMultiScale(webCamImg,faceProfile);

                for (Rect rect:faceRect.toArray()){
                    Imgproc.putText(webCamImg,"face",
                            new Point(rect.x,rect.y-5),1,1,
                            new Scalar(0,0,225));

                    Imgproc camShow = new Imgproc();

                    camShow.rectangle(webCamImg,
                            new Point(rect.x,rect.y),
                            new Point(rect.x+rect.width,rect.y+rect.height),
                            new Scalar(0,100,200),5);




                    if (capture.grab()){

                        System.out.println("Frame Grabbed ");

                        /*** Calll the Head Pose estimation method    **/
                        HeadPoseEstimationMain hpE = new HeadPoseEstimationMain();
                        //hpE.HPE(webCamImg);
                    }


                    System.out.println("Face");

                }


                if (!webCamImg.empty()){
                    tmpImg= convertImage.convertMatToImg(webCamImg);
                    ImageIcon imageIcon = new ImageIcon(tmpImg,"Captured");
                    label.setIcon(imageIcon);
                    frame.pack();
                }
                else {
                    System.out.println("no frame");
                    break;
                }
            }
        }
        else {
            System.out.println("nothing ");
        }
    }

    /**---------------------------- The main --------------------------**/

    public static void main(String[] args) {
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        App app = new App();
        app.mainGUI();
        app.runLoop();

    }
}

The Heap pose estimation class

public class HeadPoseEstimationMain {

    public void HPE(Mat matOfImg){

        int MaxCount= 10;
        TermCriteria termCriteria = new TermCriteria(TermCriteria.EPS | TermCriteria.COUNT,20,0.3);

        List<MatOfPoint2f> matOfPointList = new ArrayList<MatOfPoint2f>();
        Size subPixWinSize = new Size(10,10);
        Size winSize = new Size (21,21);

        Imgproc.goodFeaturesToTrack(matOfImg, (MatOfPoint) matOfPointList,MaxCount,0.01,10, null,3,true,0.04);
        Imgproc.cornerSubPix(matOfImg, (MatOfPoint2f) matOfPointList,subPixWinSize,new Size(-1,-1),termCriteria);

    }


}

Please help me out

edit retag flag offensive close merge delete

Comments

imho, your tutorial there is quite bad. it does not show the 2d ->3d mapping (solvePnP), you probably should use some real facial landmarks, and a real 3d head model.

maybe this or that

berak gravatar imageberak ( 2017-05-25 06:33:41 -0600 )edit

Thanks for the Hint. How can i Add those model points with Java ?

myCode gravatar imagemyCode ( 2017-05-25 08:52:23 -0600 )edit

is it kind of List<point3> point3List = new ArrayList<>();

    point3List.add(new Point3(-36.9522f,39.3518f,47.1217f));
myCode gravatar imagemyCode ( 2017-05-25 08:54:29 -0600 )edit

yea, something like that.

but the gist is: you need some 3d coords corresponding to some known 2d landmarks, then you go and find the landmarks in your image, and use solvePnP with the new 2d positions, and the original 3d points, to get a (3d) transformation matrix - the "pose"..

the corner-tracking / optflow thing in your original post won't lead you anywhere, simply said.

berak gravatar imageberak ( 2017-05-25 09:01:18 -0600 )edit