How about converting to HSV colorspace (call cv::cvtColor with cv::COLOR_BGR2HSV), then threshold the Saturation channel (perhaps using cv::inRange), to generate the mask? The Saturation channel is what our human eyes perceive as "the rich colorfulness". Tones of grays on the image should produce low values in the Saturation channel. Please let us know how that turns out (especially as the background color changes) by adding the new output images to your question.