Cut faraway objects based on depth map

asked 2020-11-23 05:31:46 -0500

Ahmed gravatar image

updated 2020-11-24 04:30:48 -0500

berak gravatar image

I would like to do grabcut which uses a depth map that cuts away far objects, that is used in mixed reality application. So I would like to show just the front of what I see and the background as virtual reality scene.

The problem right now I tried to adapt so code and what I get is front which is cut but in black color, the mask actually.

I don't know where is the problem settle.

The input is a depth map from zed camera. here is a picture of the behaviour: enter image description here

image description

 private void convertToGrayScaleValues(Mat mask)
        int width = mask.rows();
        int height = mask.cols();
        byte[] buffer = new byte[width * height];
        mask.get(0, 0, buffer);
        for (int x = 0; x < width; x++)
            for (int y = 0; y < height; y++)
                int value = buffer[y * width + x];

                if (value == Imgproc.GC_BGD)
                    buffer[y * width + x] = 0; // for sure background
                else if (value == Imgproc.GC_PR_BGD)
                    buffer[y * width + x] = 85; // probably background
                else if (value == Imgproc.GC_PR_FGD)
                    buffer[y * width + x] = (byte)170; // probably foreground
                    buffer[y * width + x] = (byte)255; // for sure foreground
        mask.put(0, 0, buffer);

For Each depth frame from Camera:

  Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(4, 4));
        Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7));


        Core.normalize(maskFar, maskFar, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);

        Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);

        Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_BINARY);
        Imgproc.dilate(maskFar, maskFar, erodeElement);
        Imgproc.erode(maskFar, maskFar, dilateElement);

        Mat bgModel = new Mat();
        Mat fgModel = new Mat();

        Imgproc.grabCut(image, maskFar, new OpenCVForUnity.CoreModule.Rect(), bgModel, fgModel, 1, Imgproc.GC_INIT_WITH_MASK);

        convertToGrayScaleValues(maskFar); // back to grayscale values
        Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_TOZERO);

        Mat foreground = new Mat(image.size(), CvType.CV_8UC4, new Scalar(0, 0, 0));
        image.copyTo(foreground, maskFar);

        Utils.fastMatToTexture2D(foreground, texture);
edit retag flag offensive close merge delete


is this really c# (like the tag says) ? it rather looks ike java.

you also probably need to tell us, what should be the correct foreground

berak gravatar imageberak ( 2020-11-23 07:23:12 -0500 )edit

@break thanks, the correct foreground is the one that near the camera like the laptop screen, and other far away pixels are cut out

Ahmed gravatar imageAhmed ( 2020-11-23 07:39:20 -0500 )edit

you probably misunderstood the mask usage.

GC_INIT_WITH_MASK assumes you have it filled with GC_BGD, GC_FGD, PR_BGD,GC_PR_FGD values when calling grabCut, you ssem to have a [0,255] binary image.

then, your convertToGrayScaleValues() function assumes the mask is filled with those values, but it will only get set to those using GC_INIT_WITH_RECT

please read docs, again.

the correct foreground is the one that near the camera like the laptop screen, and other far away pixels are cut out

sorry, still unclear. instead of your screenshot, we probably need to sse your original and depth image,

berak gravatar imageberak ( 2020-11-23 08:14:00 -0500 )edit
  Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);

what is inside your depth image ? this would be wrong for a single channel

berak gravatar imageberak ( 2020-11-23 08:38:13 -0500 )edit

@berak, the depth image is F32_C4, I would like to cut far objects and extract the foreground as shown in intel example from your discussion yes you are correct that's a mistake, the input depth is float, but how would I fix it ? and just cut away far pixels and keep near pixels ?

Ahmed gravatar imageAhmed ( 2020-11-23 09:01:00 -0500 )edit

yes, that intel example looks quite good, you should try it like that !

(again, the way you fill your mask now, seems to be the problem)

berak gravatar imageberak ( 2020-11-23 09:24:27 -0500 )edit

@break How would I fill the incoming depth F32_C4 with the required mask like intel ? I don't know how to convert it to meanfull values, for example they say they have a colorized shader that marks black and white with distance far is black and near is white, don't know how to do that, thanks for your help!

Ahmed gravatar imageAhmed ( 2020-11-23 11:30:24 -0500 )edit