piedar's profile - activity

I am trying to write a HandTracker application to identify and track human hands in a scene. I have tried a number of techniques, which I will detail here. I need your help to figure out the technique or combination of techniques I should use to be successful.

What I have:

1. 640*480 Bayer-encoded color stream
2. 640*480 depth stream wherein each pixel is in the range 0 - 10000 and is registered (aligned) to the corresponding pixel in the color stream.

What I have tried:

1. Cascade classifier using a Haar dataset. I have tried the hand datasets here, here, and here with almost no success. Some datasets produce many false positives and some produce no positives no matter how much I wave my hand about.
2. Descriptor matching with a SURF detector, BRISK extractor, and BFMatcher using knnMatch(). I use a screenshot cropped to contain only my open hand as the first image. I then run the matching scheme on each color frame, keeping only matches that pass a ratio test and a symmetry test. Finally, I compute a bounding polygon by using findHomography() on the matching keypoints and doing a perspectiveTransform(). This is somewhat successful, but is slow and dependent on the scale and orientation of the first image. And even when I get enough keypoints, the bounding polygon is often wildly incorrect.
3. Contour and convex hull search. This finds way too many contours and hulls, and I don't know how to discriminate hands from everything else.
4. Background removal with BackgroundSubtractorMOG2. This helps the accuracy of the contour search, as it makes sure all contours are part of the moving foreground. But it still doesn't help me discriminate between a hand and a neck (for example), and finding accurate bounding geometry still a problem.
5. CamShift. This looks like it could be useful, but I haven't gotten it working. All the examples I've found seem really complicated.

After working on this for a week, I'm starting to get more than a little frustrated. I just want to do something simple like the hand tracking in OpenNI. Unfortunately, the OpenNI 1.5 code is so obfuscated that I can't even find the hand tracking portion.

Does anyone have any ideas for how I can combine and/or refine these techniques on the color stream or incorporate the depth stream to achieve robust hand identification/tracking?

Got it thanks to berak! It seems so obvious after a good night's sleep.

The scale value for cv::Mat::convertTo() should be the maximum possible value of the data type over the maximum possible value of the arbitrary range. So if the values are in the range 0 - 10000, conversion would be done like so to remain in 16 bits:

raw.convertTo(raw, CV_16UC1, 65535.0 / 10000);

and to convert to 8 bits:

raw.convertTo(raw, CV_8UC1, 255.0 / 10000);

Thanks berak, that works. I would answer my own question, but the site won't let me for another 17 hours.

I have a cv::Mat of uint16_t representing a grayscale depth map. The values in this map are in the range 0 - 10000. I construct and display the map like this:

cv::Mat raw(frame.height(), frame.width(), CV_16UC1, frame.data.data());
raw.copyTo(depth_image);
cv::imshow("depth", depth_image);

The problem is that cv::Mat seems to be expecting the full range of possible values for a 16-bit unsigned integer, so I can barely see differences between areas in my image (everything is very dark). Is there any way I can tell cv::Mat to expect grayscale values no larger than 10000?

Maybe using cv::Mat::convertTo()? I don't know what to use for scale though. The obvious guesses (10000, 1/10000) don't work.

raw.convertTo(raw, CV_8UC1, ???);

Thanks! I spend hours on this last night. I can't believe it was just rows, cols instead of width, height. CV_8UC1 was right in this case.

I am new to OpenCV. I am trying to convert a cv::Mat of Bayer data to BGR. The input data is a 640*480 array of uint8_t. My code looks like this:

cv::Mat video_image;
cv::Mat raw(640, 480, CV_8UC1, frame->buffer.data());
cv::cvtColor(raw, video_image, CV_BayerGB2BGR);
cv::imshow("video", video_image)

BONUS QUESTION: Does the Mat constructor make a deep copy of the input data?

But video_image looks like this:

I have tried CV_8UC3 and setting raw.step to 640*3. What am I missing?

PS: I took a screenshot because the highgui save image button had no apparent effect on the filesystem.