Ask Your Question
9

how can I align Face Images

asked 2013-11-27 17:49:17 -0600

xaffeine gravatar image

updated 2020-11-30 00:53:13 -0600

I see that there's a nice little script in the facerec tutorial that aligns faces in images given positions for the eyes.

I need to get the coordinates for the eyes, though, somehow. Would it work to use the CascadeClassifier with the haarcascade_mcs_eyepair_big model?

I understand that there are many ways to do it (including the patented ASEF technique). I'm just wondering if this is a good way to get started.

edit retag flag offensive close merge delete

Comments

1

i'm wondering about this, too. did you get any results, so far ?

applying AAM or flandmark to find eye position might be another idea.

this is very complex/difficult, but gets the job done

berak gravatar imageberak ( 2013-11-28 04:26:20 -0600 )edit

6 answers

Sort by ยป oldest newest most voted
18

answered 2013-12-02 05:23:55 -0600

updated 2014-05-23 03:15:29 -0600

I'm going to put the code I perform in order to pre-process a face. I do the following:

//Step 1: detect landmarks over the detected face
vector<cv::Point2d> landmarks = landmark_detector->detectLandmarks(img_gray,Rect(r.x,r.y,r.width,r.height));
//Step 2: align face
Mat aligned_image;
vector<cv::Point2d> aligned_landmarks;
aligner->align(img_gray,aligned_image,landmarks,aligned_landmarks);
//Step 3: normalize region
Mat normalized_region = normalizer->process(aligned_image,Rect(r.x,r.y,r.width,r.height),aligned_landmarks);
//Step 4: tan&&triggs
normalized_region = ((FlandMarkFaceAlign *)normalizer)->tan_triggs_preprocessing(normalized_region, gamma_correct,dog,contrast_eq);

Step 3 tries to get only "face region", and step 4 performs tang&&triggs.

I'm going to put the code for step 1 and 2.

STEP 1

Step 1 calls a landmark detector. Flandmark detector is a good one (better than the one based on cascades for example). Flandmark detector returns the position of the landmarks in the image calling flandmarks, but also does an additional task:

I create a line based on linear regresion using the four points for the eyes (LEFT_EYE_OUTER, LEFT_EYE_INNER, RIGHT_EYE_INNER, RIGHT_EYE_OUTER). Then i calculate two points based on this linear regrestion (the blue ones). (These points are called pp1 and pp2 in the code). I used this linear regresion class.

I USED THESE TWO POINTS TO ROTATE THE FACE --> THESE POINTS ARE MORE ROBUST THAN USING FOR EXAMPLE LEFT_EYE_OUTER/RIGHT_EYE_OUTER OR LEFT_EYE_INNER/RIGHT_EYE_INNER

So, with the first step i get the following results:

image description image description

The code is as follows:

vector<cv::Point2d> FlandmarkLandmarkDetection::detectLandmarks(const Mat & image, const Rect & face){

vector<Point2d> landmarks;

int bbox[4] = { face.x, face.y, face.x + face.width, face.y + face.height };
double *points = new double[2 * this->model->data.options.M];

    //http://cmp.felk.cvut.cz/~uricamic/flandmark/
if(flandmark_detect(new IplImage(image), bbox, this->model,points) > 0){
    return landmarks;
}

for (int i = 0; i < this->model->data.options.M; i++) {
    landmarks.push_back(Point2f(points[2 * i], points[2 * i + 1]));
}

LinearRegression lr; 
lr.addPoint(Point2D(landmarks[LEFT_EYE_OUTER].x,landmarks[LEFT_EYE_OUTER].y));
lr.addPoint(Point2D(landmarks[LEFT_EYE_INNER].x,landmarks[LEFT_EYE_INNER].y));
lr.addPoint(Point2D(landmarks[RIGHT_EYE_INNER].x,landmarks[RIGHT_EYE_INNER].y));
lr.addPoint(Point2D(landmarks[RIGHT_EYE_OUTER].x,landmarks[RIGHT_EYE_OUTER].y));

double coef_determination = lr.getCoefDeterm();
double coef_correlation = lr.getCoefCorrel();
double standar_error_estimate = lr.getStdErrorEst();

double a = lr.getA();
double b = lr.getB();

cv::Point pp1(landmarks[LEFT_EYE_OUTER].x, landmarks[LEFT_EYE_OUTER].x*b+a);
cv::Point pp2(landmarks[RIGHT_EYE_OUTER].x, landmarks[RIGHT_EYE_OUTER].x*b+a);

landmarks.push_back(pp1); //landmarks[LEFT_EYE_ALIGN]
landmarks.push_back(pp2); //landmarks[RIGHT_EYE_ALIGN]

delete[] points;
points = 0;
return landmarks;
}

STEP 2

Step 2 aligns the face. It is based also on https://github.com/MasteringOpenCV/code/blob/master/Chapter8_FaceRecognition/preprocessFace.cpp

The code is as follows:

double FlandMarkFaceAlign::align(const Mat &image, Mat &dst_image, vector<Point2d> &landmarks, vector<Point2d> &dst_landmarks){
const double DESIRED_LEFT_EYE_X = 0.27;     // Controls how much of the face is visible after preprocessing.
const double DESIRED_LEFT_EYE_Y = 0.4;

int desiredFaceWidth = 144;
int desiredFaceHeight = desiredFaceWidth;

Point2d leftEye = landmarks[LEFT_EYE_ALIGN];
Point2d rightEye = landmarks[RIGHT_EYE_ALIGN];

// Get the center between the 2 ...
(more)
edit flag offensive delete link more

Comments

This is a very nice answer. You should consider transforming this into a guide that can be attached with the face recognizer guide. You up for it?

StevenPuttemans gravatar imageStevenPuttemans ( 2013-12-02 08:42:53 -0600 )edit

I can try, but i don't know how to do it...

  1. where is the face recognizer guide?
  2. how can i edit the face recognizer guide?
  3. My "english" needs checking Thanks!!
albertofernandez gravatar imagealbertofernandez ( 2013-12-02 09:24:45 -0600 )edit
1

Well let me give you some guidelines:

  1. An overview of guides on the face recognition algorithm can be found here, with especially interesting for you this and this one.

  2. Guidelines on how to contribute can be found on the how to contribute page.

  3. The documentation can be found inside the rst files that are around in source code.

  4. If you need help on your spell checking, just put @StevenPuttemans in your GitHub pull

StevenPuttemans gravatar imageStevenPuttemans ( 2013-12-03 02:55:16 -0600 )edit
1

sorry for my lack of knowledge, but i have another question. If for example, I modify a rst file (for example opencv/modules/contrib/doc/facerec/index.rst to include a link to my documentation file srt, is there a way to see if it has errors, something like a srtToPDF convertor or something like this?

albertofernandez gravatar imagealbertofernandez ( 2013-12-03 03:57:24 -0600 )edit
2
1

You could use the tool provided by @Daniil Osokin indeed. What you could also do is build documentation through CMAKE but that requires some extra packages like a LaTeX framework and so on. Basically I saw there is an approach already for alignment, what you could do is split up the tutorial and make two links to both approaches, both openening in a new page. Haven't made drastic improvements like this myself, so maybe taking a look at this could help.

StevenPuttemans gravatar imageStevenPuttemans ( 2013-12-03 06:36:58 -0600 )edit
1

Thanks for all the tips. I came across https://notex.ch/ It can handle complete projects and you can export to PDF, HTML, latex, epub,.... (http://wiki.typo3.org/Editors_(reST))

albertofernandez gravatar imagealbertofernandez ( 2013-12-03 10:31:44 -0600 )edit
1

hi alberto, thanks for your extremely useful post.

i'm just giving it a try, can you give me a hint, which one is the LEFT_EYE in your notation ?

s1/s5 ? or s2/s6 ?

const double DESIRED_LEFT_EYE_X = 0.27;

const double DESIRED_LEFT_EYE_Y = 0.4;

if that's the desired position for the left eye, it sounds like it's the reverse of the image above

berak gravatar imageberak ( 2013-12-05 11:33:33 -0600 )edit

I feel a new module/functionality rising up @berak? :D

StevenPuttemans gravatar imageStevenPuttemans ( 2013-12-06 02:09:59 -0600 )edit
5

answered 2013-11-29 14:33:39 -0600

xaffeine gravatar image

I discovered Emami's chapter 8 of the Mastering OpenCV book. It gives a few variations on how to use eye detection for face alignment.

I don't know yet how well it will work. I like the congealing approach berak referred to, but I'll try the simpler stuff first.

edit flag offensive delete link more
1

answered 2014-05-22 08:22:51 -0600

molina gravatar image

Hello, I can not to do work this source code posted. Somebody expert could post the complete source code for help me ? I need to align an face but I'm newbie in this subject, I'm wasted few days research and I have really difficulty in reproduce "alignment face". Help me please.

edit flag offensive delete link more

Comments

1

hi molina. please start a new question ( and refer to this one ), delete this answer, show what you tried, and where you got stuck,

berak gravatar imageberak ( 2014-05-22 08:31:08 -0600 )edit
1

I have updated the answer with the source code.

albertofernandez gravatar imagealbertofernandez ( 2014-05-23 03:16:33 -0600 )edit

It is still actual question. Am I brutal if ask you to updated code to latest version all used libraries (flandmark now clandmark for example, etc.)? Thanks

vinnitu gravatar imagevinnitu ( 2015-05-22 05:07:46 -0600 )edit
0

answered 2015-06-16 03:56:41 -0600

Mohanraj gravatar image

updated 2015-06-16 03:57:39 -0600

I have done step 1 and step 2 completely and also i got proper output . Anyone please help me how to step 3 for normalizing the face region.

edit flag offensive delete link more

Comments

2

I am going to update my github account with this procedure. Meanwhile, you can check this procedure in the paper "An experimental comparison of gender classification methods". See Table 1 (The algorithm for face normalization)

An experimental comparison of gender classification methods

albertofernandez gravatar imagealbertofernandez ( 2015-06-16 03:59:38 -0600 )edit

i have used the same algorithm and calculated d0 but i don't know how to calculate dt. if possible please mail me the step 3 part alone it will be very helpful to carry out my research work. [email protected]

Mohanraj gravatar imageMohanraj ( 2015-06-16 05:43:30 -0600 )edit
0

answered 2015-09-02 03:22:51 -0600

Hi Alberto, thanks for the detailed procedure. I use 0.3 MP laptop cam images to train, and then later recognize faces. I used the same except for the flandmark: I used eye cascade of OpenCV to detect eye positions.

I have been getting confidence/distance values coming more than 80 with LBPH face recognizer, sometimes giving incorrect recognition. Even with Tan triggs preprocessing, I dont see any improvement in the confidence, or correct recognition. Similar with the face alignment.

My query is: - have you seen substantial improvement in correct face recognition percentage with Tan-triggs and face alignment using eye position: so much as to use it for commercial applications? - The distance measures come as more than 70, is there any way to normalize, say 0-1 the distance measures, as these values seem quite dependent on the LBPH parameters.

I have not observed any improvement with Opencv eyecascade and Tan-triggs. I am looking for Clandmark windows8 build to test out with it. Will report if I see substantial improvement with that.

edit flag offensive delete link more
-1

answered 2018-02-21 11:35:55 -0600

ManuVISION gravatar image

I have implemented it using DLib and OpenCV https://github.com/ManuBN786/Face-Ali...

It corrects alignments in any face.

edit flag offensive delete link more

Comments

sorry to say so, but a "look i dunnit", link-only answer is never really welcome.

don't get me wrong, i know, you can do better than this ...

berak gravatar imageberak ( 2018-02-21 12:19:55 -0600 )edit

dlib has its own alignment feature.

sturkmen gravatar imagesturkmen ( 2018-02-21 12:24:11 -0600 )edit

Question Tools

6 followers

Stats

Asked: 2013-11-27 17:49:17 -0600

Seen: 19,370 times

Last updated: Feb 21 '18