Display part of colour image on Grey Scale image

edit

I have an image that is in HSV, and an image that is in Greyscale.

I want to keep the grayscale image as grayscale, and display the HSV image on top of this one. There are black pixels in the HSV image which are intentionally black and are to be masked out.

I have tried a few things, the most recent approach got me no where.

cv::Mat mImg; // grayscale "background" image
cv::Mat hsvImg; // foreground hsv image

// convert hsv
cv::Mat rgbOutput, greyOutput;
cv::cvtColor(hsvImg, rgbOutput, CV_HSV2BGR);
rgbOutput.convertTo(rgbOutput, CV_8U);

cv::cvtColor(rgbOutput, greyOutput, CV_BGR2GRAY, 1);
greyOutput.convertTo(greyOutput, CV_8U);

// create mask and inverted mask
cv::Mat rgbMask, greyMask;
cv::threshold(rgbOutput, rgbMask, 1, 255, cv::THRESH_BINARY);
cv::threshold(greyOutput, greyMask, 1, 255, cv::THRESH_BINARY_INV);

showImage("mImg", mImg);
showImage("mask", rgbMask);

cv::Mat mNoHSVPixels = greyMask & mImg; // all hsv pixels are now 0

showImage("and", mNoHSVPixels);

// convert the mNoHSVPixels to be in BGR
cv::Mat mNoHSVPixelsRGB;
cv::cvtColor(mNoHSVPixels, mNoHSVPixelsRGB, CV_GRAY2BGR, 3);
showImage("maskedRGB", mNoHSVPixelsRGB);

cv::Mat mOutImg = mNoHSVPixelsRGB + (rgbOutput & rgbMask) ;

showImage("out?", mOutImg);

The images always come out horribly white with a small amount of Red. The input image is a generic image.