How to construct a 3d face from 2d images in openCV?

edit

In that case what about the so-called 3d face recognition solutions? I assume there must be some real 3d information contained in the descriptors of the different keypoints, right? Next to this I assume that in some way it must be possible to use voxels, which are 3d-pixels. What do you think?

Which 3D face recognition solutions? Is there a link?

It is possible to use 3D pixels if you have a 3D sensor. If you have a 2D sensor there is no way to magically get 3D data. Feature extraction will further reduce the dimensionality of the data, not increase it.

Just think about it. Imagine a frontal face picture. How could it be ever possible to determine, for example, the lenght of the nose?

Which 3D face recognition solutions? Is there a link?

I am just speaking in general. I suppose the title of this wikipedia article (https://en.wikipedia.org/wiki/Three-d...) wouldn't contain the words 3D if there isn't really some 3d info/data being used.

How could it be ever possible to determine, for example, the lenght of the nose?

By combining multiple images: profile, frontal etc.. You may not have an absolute scale, but a relative scale is good enough for recognition

As I said in the answer, OpenCV willl let you detect faces, eyes, mouth, nose, etc in 2D data.

If having loads of images at different angles of a face is an option for you, why don't you setup a recognition system that computes and matches features on all of these 2D images, instead of having to reconstruct a 3D model to then perform recognition? Seems an useless step honestly.

But, my answer still stands, OpenCV does not provide standard tools for this, you'll have to go through the literature and implement it yourself.

According to the wiki page, going via 3d is much more robust for recognition than simply using 2d images. It states: "It has been shown that 3D face recognition methods can achieve significantly higher accuracy than their 2D counterparts, rivaling fingerprint recognition" Feel free to correct me or the wikipedia article, there are quite some new aspects there for me.

That quote is comparing recognizing in single 2D image vs single 3D "image".

If you can use loads of images to build 3D model with software math calculations, and then use it to perform recognition, in the end it would be the same thing as performing recognition right away on all of these loads of 2D images.

So you'd say stitching all the 2d images together (like for a panorama) and afterwards performing recognition using machine learning, would be robust enough? I was thinking about a 3d model that way you can eg have an idea about ones nose length as well. While when just stiching the images together to get a flattened face you are -I think- unable to know that.

Some people seem to go really very far for a robustface recognition (or smth similar) system: https://github.com/sergeytulyakov/Fac... I guess the best way after having stitched the images together using matched keypoints would be to create my own machine learning based classifier that picks good/strong features (ie non matter whether he wears sunglasses a beard etc...) to do recognition?

@linengmiao, Look carefully at that github source. They do it using 3D sensor (microsoft kinect), precisely as I said.

As to my previous idea, there is no need for panorama.. just read carefully:

If you have a set of 2D images which can be transformed to a single 3D image, and then you use that 3D model features to do recognition, wouldn't it be exactly the same as just extracting features from the set of 2D images right away and do recognition? Why go about reconstructing?

If you use all these 2D images to extract relevant features for recognition, indirectly your machine learning method will include nose lenght and any features that are actually relevant for differentiating a sample from other.

Indeed you're making a fair point. If "panorama" is not the way to go, how would you combine those different images to get "global" descriptors? I mean by that one could calculate the descriptors for 2 profile pictures and afterwards separately the descriptors for a frontal picture.The subject would then have to show its face to the camera under multiple angles again in order to be able to fully recognize him. I don't see where feature matching comes into play. What I initially had in mind: when signing up we take 3 pictures of the subject (2 profiles and one frontal), create a 3d model with those. And then when logging in the user would just have to face the camera (no need to be face completely straight as we have a sort of speak +/- 360 view of his head in the database) to be recognized

Recognition means that you have a database somewhere with the faces you want to recognize stored. So, in run time, you application will always compute features and compare them to this database in order to recognize whose face it is.

First you compute features from your set of 2D images, and then you go compare those features to all the subjects on your database and figure out which subject returns the highest match score.

As to your system design, I know nothing about it so I don't know how it should be done.