The best way to do this is by using the HSV color space, which makes it easier to manipulate colors
First convert color space:
Mat RGBmat;
Mat HSVmat;
cvtColor(RGBmat, HSVmat, CV_RGB2HSV);
Then filter the yellow colors of the image. You can check the HSV color space intervals in this link .
The color component of the HSV color space is mainly controled by the H channel. The S stands for saturation, V stands for value and H stands for hue
To get each channel you have to split the initial Mat:
Mat HSVchannels[3];
cv::split(HSVmat, HSVchannels);
HSVchannels[0] is a Mat with values that range between 0 and 255, and the yellow tones are between 20 and 40 (this was a really rough estimation by looking at the chart in the link above).
You can threshold the HSVchannels[0] to get only the yellow values
Mat aux = (HSVchannels[0] < 40);
Mat yellow = (aux > 20);
Then, you just manipulate the yellow mat as you wish. You can also change the V and S mat values that correspond to the yellow pixels to change lower brightness and saturation.
After manipulation the channels, you have to merge them back into one 3 channel matrix:
Mat newHSV;
Mat newRGB;
cv::merge(HSVchannels, newHSV);
cv::cvtColor(newHSV, newRGB, CV_HSV2RGB); //change it back to RGB
Good luck