How to get those unique lable in a mask with efficient

edit

If I have a such label image

Then I have a such mask

I want to get those unique integer lables in the given mask(same size with the label image). The result is {0,1,2,3,6} in my this example. This is my current method

#include<opencv.hpp>
#include<string>
using namespace std;
using namespace cv;

int main() {
    uchar a[6][7] = { { 0,0,0,0,0,0,0 },{ 0,1,2,2,3,3,0 },{ 0,0,0,0,0,0,0 },{ 0,5,5,6,6,0,0 },{ 0,5,5,6,6,0,0 },{ 0,0,0,0,0,0,0 } };
    Mat label = Mat(6, 7, CV_8UC1, a);
    uchar b[6][7]= { { 0,0,0,0,0,0,0 },{ 0,255,255,255,255,0,0 },{ 255,255,255,0,0,0,0 },{ 0,0,0,255,255,0,0 },{ 0,0,0,255,255,0,0 },{ 0,0,0,0,0,0,0 } };
    Mat mask = Mat(6, 7, CV_8UC1, b);
    vector<int> list;
    for (int i = 0; i < mask.rows; i++) {
        uchar* mask_p = mask.ptr<uchar>(i);
        for (int j = 0; j < mask.cols; j++)
            if (mask_p[j] == 255)
                list.push_back(int(label.at<uchar>(i,j)));
    }
    sort(list.begin(), list.end());
    list.erase(unique(list.begin(), list.end()), list.end());

    return 0;
}

But in my real case, I have just a small ROI in a huge mask image. I think the current method which iterate every pixel in the mask waste my time too much in my case. Can anyone give me some advice?