What is the best method to match face features against a database?
The challenge
I'm trying to replicate the experience from this campaign, but in realtime:
What I have now
- I'm able to detect faces and (somewhat naively) mark the regions I want: right and left eyes, cheeks, forehead, mouth, nose and so on.
- I run the same algorithm against a database of face images (5k images from Getty Images) and store image slices for each region.
- I'm just converting to gray and storing some metadata for each face slice. I suspect I could get better matches applying some kind of filter.
- I made a web app using Django Rest Framework that captures a snapshot using the web cam, locate your face and try to match each slice against slices of the same part of the face in the database.
The current problem
The matching is not very good. Sometimes, the best match for a face slice is a false positive from the face recognition algorithm (I got a low percentage of false positives and I'm curating the database by hand). I've tried the following algorithms in order to find the best match for each face feature from skimage.measure:
- compare_ssim
- compare_psnr
- compare_mse
- compare_nrmse
Best results are from Structural Similarity, but still far from ideal. Bellow are an example of the app returning a false positive as the best match for my mouth:
As you may have guessed I don't have a clue about what I'm doing - unable to figure out why it is not picking from hundred of mouths in the database.
how is tis related to opencv at all ?
if you want to compare face regions, imho,
you have to align the input images, so they're frontal, and the eyes are on a horizontal line (and in a fixed position)
try features different from plain pixels, LBPH, gabor wavelets, dense SIFT, etc.
I got the images doing a search for "face man front" so most of the database are frontal and horizontal. After a couple hours curating the database, results are starting to get better. I will try other algorithms a see if I can make it fast enough. Thanks!
also, maybe, instead of having "fixed" search locations (and aligning your images to that) you could try to use facial landmarks, and adjust your search region to "where the nose really is"
Oh, that would be great because I then I could transform the image. I just made a Google search about "face landmark detection", looks like it will be very useful. Thanks again, this is a good tip.