Ask Your Question

Using the IR image as a mask for RGB image

asked 2016-06-03 03:23:41 -0500

I'm passing the RGB, Depth and IR image through from libfreenect2 to opencv, and I am having trouble working out how to use the the IR image as a mask over the RGB image to remove everything except humans from a video stream.

I've gotten the images from libfreenect2 and converted them to the following image types (based off this post) :

    cv::Mat(rgb->height, rgb->width, CV_8UC4, rgb->data).copyTo(rgbmat);
    cv::Mat(ir->height, ir->width, CV_32FC1, ir->data).copyTo(irmat);
    cv::Mat(depth->height, depth->width, CV_32FC1, depth->data).copyTo(depthmat);

Thanks in advance!

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2016-06-03 04:01:15 -0500

updated 2016-06-03 04:41:42 -0500

Just pass the IR image as second parameter to copyTo. Only pixels with a non-zero value in the mask are copied:

CopyTo With mask

// Update: As the mask has to be CV_8U, you could use cv::convertTo to change the type.

edit flag offensive delete link more


wait, masks have to be CV_8U, and you probably have to scan the value range of the ir image, and then threshold it at an approp. value)

berak gravatar imageberak ( 2016-06-03 04:11:41 -0500 )edit

Thanks for the hint!

FooBar gravatar imageFooBar ( 2016-06-03 04:41:17 -0500 )edit

Thanks for helping getting me started. So using this source code, it does compile and run but I think there is still an issue with the conversion.

cv::Mat(ir->height, ir->width, CV_32FC1, ir->data).copyTo(irmat);
irmat.convertTo(irmat, CV_8U, 1, 0); 
resize(irmat, irmat, cvSize(1920, 1080); 
cv::Mat(depth->height, depth->width, CV_32FC1, depth->data).copyTo(depthmat);
cv::Mat(rgb->height, rgb->width, CV_8UC4, rgb->data).copyTo(rgbmat, irmat);

When running this program, the IR image and the depth image when doing something similar don't match up properly to the rgb image.

carouselification gravatar imagecarouselification ( 2016-06-03 05:49:31 -0500 )edit

did you find out, in which range your ir (float) values are ?

convertTo() clips/saturates values > 255 to 255, so if your range there is larger, you might need a different scale factor

berak gravatar imageberak ( 2016-06-03 06:30:48 -0500 )edit

Is there an easy way to calculate this scale factor in opencv?

carouselification gravatar imagecarouselification ( 2016-06-04 01:18:54 -0500 )edit

minMaxLoc will give you the minimum/maximum value

berak gravatar imageberak ( 2016-06-04 01:53:08 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2016-06-03 03:23:41 -0500

Seen: 687 times

Last updated: Jun 03 '16