Difference between output of grabCut between runs

asked 2018-09-28 05:59:40 -0500

Jendker gravatar image

updated 2018-09-28 08:04:55 -0500

Hello everyone,

When running grabCut on an the same input in the loop every time I will get the different output, but if I restart the program, the output, even though different within the loop, will match the result of corresponding iteration from the previous run. I know, that grabCut will use some singletons so that the instance will be there through all the runs until the program will be terminated, but still, why do the separate runs of the algorithm interfere with each other (somehow) and what to do to get around that?

I checked it with the unit test. If on one run I will save the results to disk like this:

TEST(PreprocessingTests, CutAndFilterSave)
{  
    std::string output_directory = "cutAndFilterPreviousRun";
    std::experimental::filesystem::create_directory(output_directory);

    cv::Mat hidden_mouth_image = cv::imread("hidden_mouth.png", cv::IMREAD_GRAYSCALE);
    EXPECT_TRUE(hidden_mouth_image.data);
    cv::Mat whole_face_image = cv::imread("whole_face.png", cv::IMREAD_GRAYSCALE);
    EXPECT_TRUE(whole_face_image.data);

    std::vector<cv::Mat> images_to_save;
    images_to_save.emplace_back(hidden_mouth_image);
    images_to_save.emplace_back(whole_face_image);

    processImagesForComparison(images_to_save);

    cv::imwrite(output_directory + "/hidden_mouth_filtered.png", images_to_save.at(0));
    cv::imwrite(output_directory + "/whole_face_filtered.png", images_to_save.at(1));
}

And on second run I will run this function

TEST(PreprocessingTests, CutAndFilterRead)
{
    std::string output_directory = "cutAndFilterPreviousRun";
    if (std::experimental::filesystem::exists(output_directory))
    {
        cv::Mat hidden_mouth_image_filtered = cv::imread(output_directory + "/hidden_mouth_filtered.png", cv::IMREAD_GRAYSCALE);
        EXPECT_TRUE(hidden_mouth_image_filtered.data);
        cv::Mat whole_face_image_filtered = cv::imread(output_directory + "/whole_face_filtered.png", cv::IMREAD_GRAYSCALE);
        EXPECT_TRUE(whole_face_image_filtered.data);

        std::vector<cv::Mat> previous_run_filtered_images;
        previous_run_filtered_images.emplace_back(hidden_mouth_image_filtered);
        previous_run_filtered_images.emplace_back(whole_face_image_filtered);

        cv::Mat hidden_mouth_image = cv::imread("hidden_mouth.png", cv::IMREAD_GRAYSCALE);
        EXPECT_TRUE(hidden_mouth_image.data);
        cv::Mat whole_face_image = cv::imread("whole_face.png", cv::IMREAD_GRAYSCALE);
        EXPECT_TRUE(whole_face_image.data);

        std::vector<cv::Mat> test_images;
        test_images.emplace_back(hidden_mouth_image);
        test_images.emplace_back(whole_face_image);

        processImagesForComparison(test_images);
        EXPECT_TRUE(areTheSame(test_images, previous_run_filtered_images));
    }
}

Here areTheSame will return true. But if I run it in the loop:

TEST(PreprocessingTests, CutAndFilterTest)
{ 
    cv::Mat hidden_mouth_image = cv::imread("hidden_mouth.png", cv::IMREAD_GRAYSCALE);
    EXPECT_TRUE(hidden_mouth_image.data);
    cv::Mat whole_face_image = cv::imread("whole_face.png", cv::IMREAD_GRAYSCALE);
    EXPECT_TRUE(whole_face_image.data);

    std::vector<cv::Mat> original_images;
    for (int i=0; i < 1; i++)
    {
        original_images.emplace_back(hidden_mouth_image);
        original_images.emplace_back(whole_face_image);
    }

    auto copied_original_images = cloneMatVector(original_images);

    std::vector<cv::Mat> previous_filtered_images;
    for (int run = 0; run < 2; run++)
    {
        std::vector<cv::Mat> run_images = cloneMatVector(original_images);
        processImagesForComparison(run_images);

        if (!previous_filtered_images.empty())
            EXPECT_TRUE(areTheSame(run_images, previous_filtered_images));

        previous_filtered_images = cloneMatVector(run_images);    
    }
    EXPECT_TRUE(areTheSame(original_images, copied_original_images));
}

then areTheSame will fail... In processImagesForComparison my grabCut implementation will be run on each vector element:

cv::Mat grabCutSegmentation(const cv::Mat& input)
{
    cv::Mat bgModel, fgModel;
    cv::Mat mask(input.rows, input.cols, CV_8U);
// let's set all of them to possible background first
    mask.setTo(cv::GC_PR_BGD);

// cut out a small area in the middle of the image
    int m_rows = 0.75 * input.rows;
    int m_cols = 0.6 * input.cols;
// the region of interest
    cv::Mat fg_seed = mask(cv::Range(input.rows/2 - m_rows/2, input.rows/2 + m_rows/2),
                           cv::Range(input.cols/2 - m_cols/2, input.cols/2 + m_cols/2));
// mark it as foreground
    fg_seed.setTo(cv::GC_FGD);

// select last 3 rows of the image as background
    cv::Mat1b bg_seed = mask(cv::Range(mask ...
(more)
edit retag flag offensive close merge delete

Comments

I know, that grabCut will use some singletons so that the instance will be there through all the runs until the program will be terminated

oh, wait, that was about the threadpool memory used in parallel_for_, not about the grabcut algorithm !

berak gravatar imageberak ( 2018-09-28 08:06:37 -0500 )edit

When running grabCut on an the same input in the loop every time I will get the different output, but if I restart the program, the output, even though different within the loop, will match the result of corresponding iteration from the previous run

sorry, but it means you have to dive into the src code now. e.g. the GMM part uses kmeans with KMEANS_PP_CENTERS. you'll have to investigate, how random is used there, if or how the seed is initialized there, etc.

berak gravatar imageberak ( 2018-09-28 08:10:24 -0500 )edit

I don't know... Was it? I am still fighting with the same issue as before, grabCut works nice here for segmentation, wanted to find the source of the problem before deciding to switch to different algorithm, maybe we can solve it somehow.

Jendker gravatar imageJendker ( 2018-09-28 08:11:44 -0500 )edit

Thank you berak, it really seems to be some issue with randomization here, thank you! I will start with kmeans as you recommended.

Jendker gravatar imageJendker ( 2018-09-28 08:13:18 -0500 )edit

But I guess it will not help me in the end, the result is going to be random and algorithm non deterministic. I will have to run it a couple of times to get the mean during result verification. Still I will try to find where the seed is set and make a push request. Thank you for your help berak!

Jendker gravatar imageJendker ( 2018-09-28 08:29:50 -0500 )edit

kmeans.cpp:276 RNG& rng = theRNG();

so this means: the global random generator is used. it is seeded with a fixed number on startup of the program.

but if I restart the program

yea, it will do the same thing as the last time

berak gravatar imageberak ( 2018-09-28 08:57:01 -0500 )edit

Got it, thank you. It wouldn't be difficult to change to my generator with different seed in those functions, but I don't know how far it is helpful. Good to know that there are no memory leaks or similar ;)

Jendker gravatar imageJendker ( 2018-09-28 09:08:21 -0500 )edit

oh, don't try to change it, it would making tests impossible.

berak gravatar imageberak ( 2018-09-28 09:24:55 -0500 )edit