Eye detection is "jumping". [closed]
I am working on a system that tracks the eye and log data like eye blinks, rate, etc. But I have a problem that the cv::Rect, returned after eye detection (haar detection), is jumping around so that the iris of the person appear on different positions in the matrix. in a 640x480 web cam frame, where the face is about 200px height the cv::Rect of the eye can jum about 40px each time I do the tracking. It jumps so far out that the eye sometimes is outside the rectangle representing the eye!
I am running this detection about 2 times per second, and in between I trace the eye by template mathcing. The template matching works great but sometimes the template slides of and after that keep track on the wrong position of the face. For example during a fast movement of the face the template can get stuck on the nose, or mouth.
I've tried 3 of the four eye classsifiers, (_eye, _mcs, _lefteye_2split). These three, according to the book "Master openCV..." is the best. But I cant se any big difference (more than the _mcs classifier is much slower)
It doesnt matter which one I use, So i was thinking that I might do something wrong!
Someone that is running eye detection multiple times with a good result?
My detector: eyeCascade.detectMultiScale(webcamFrame(ROI), detEye, 1.1, 3, CV_HAAR_FIND_BIGGEST_OBJECT|CV_HAAR_DO_CANNY_PRUNING|CV_HAAR_SCALE_IMAGE, cv::Size(20,20
ROI = centerpoint of last eye position, double width, double height. (4x sixe of cv::Rect( eye position))
edit: There are different classifiers for left and right eye. I force the system to find only one eye. (persons right, on the screen the left one.) Which classifier is the right one?
kalman filter ?
No filtering of the position yet. Do you think that is necessary?, I mean... The cv::Rect output from detection algorithm misses the eye abou 1/3 of all the times. Half the pupil is inside the rectangle. It feels wrong. Head is still, image is good.
"Do you think that is necessary ?"
ah, no idea, just mindstorming
Why using haar classifiers? Did you try flandmarks for example in order to get the position of the eyes and another landmarks? And this paper "Accurate eye centre localisation by means of gradients" has a c++ implementation for pupil tracking
I took a look at the flandmarks stuff, after your recommendation, and it sounds really good. Is it possible to use it in real - time you think? Haar vs flandmarks -computation time? Thanks! I will try flandmarks
Haar vs flandmarks computation time i don't know. But if you look at the videos, you can see the fps of flandmarks with different resolutions.
i cant get the flandmarks to work. unresolved external symbol during FLANDMARK_init(); :(
you need the code from the cpp/libflandmark folder in your project:
and later, when you run it, it needs to find flandmark_model.dat ( from the data folder )
There is no cpp folder, but I copied the files from libflandmark into my project. stdint.h isnt there, but a file called msvc-compat.h. I copied 5 files. I include the dlandmark_detecter.h into my main file. I am able to "FLANDMARK * model;", but when I try to do "model = flandmark_init("file"); I get the unresolved external symbol. Where shall I start to look for the error? Thanks for helping me!
oh, right, i got the old sourcefourge version ..
still investigating!
Error LNK2001, and Error LNK1120.
This is the only thing I do:
include "stdafx.h"
include "flandmark_detector.h"
int _tmain(int argc, _TCHAR* argv[]) { FLANDMARK_Model * model = flandmark_init("flandmark_model.dat"); return 0; }
ahh those pesky tchars ..., might be a widechar / unicode / ascii thing then
afaik, if you use _tmain() it assumes all strings being tchars, so the function sig does not match (thus linker error)
try an empty project for now,
include "flandmark_detector.h"
int main(int argc, char * argv[]) { FLANDMARK_Model * model = flandmark_init("flandmark_model.dat"); return 0; }
I created an empty project. copied the files mentioned before. And used the code you wrote. I also included the propertysheey I use so that I can use opencv. But the same error. could it be the propertysheet? Just to be clear. I dont have to run this cmake-thing??!!
" I dont have to run this cmake-thing??!!"
not really. you'll have to add the opencv related includes/libs manually then
This is strange... Too bad there isn't much information available on the internet about flandmark. Thanks for your help!
at least i could reproduce it ;)
haven't found out what it is, but it is related to the propertysheet.
try to skip it as well, add the opencv stuff manually ( easier, if you edit the vcproj from scite or notepad )
By adding it manually, should I create an empty propertysheet and add the opencvstuff? or do you mean by copy all the files into the project folder?
Could it be because I use the debug-files from opencv? (with a 'd' at the end of each filename.
meant creating an empty project, and adding the "additional includes" in the c++ section / libs manually
unfortunately, i only got vs2008 here, so no real idea, how that looks on your side.
not likely. it's something in your project settings
I tried it again. Empty project. In the configuration manager I change to x64, (and copy from x86). I add the 5 files from the folder libflandmark. I add in C/C++ additional directories, opencv/build/include, -opencv & -opencv2. In the linker I add opencv/build/x64/vc10/lib/ and in linker-input I add all opencv_.....d.lib files. (debug files). Thats the only thing. Opencv works, And I still got a linker error from flandmarks... :(