Ask Your Question

temporal effect on depth result in ghosty image

asked 2020-11-01 03:32:53 -0500

Ahmed gravatar image

updated 2020-11-01 03:41:39 -0500

Hi, I'm trying to do temporarl removal on depth image, and I'm getting a ghosty image as shown in the video below

image description and here is the code:

   while (key != 'q') {

        if (zed.grab(runtime_parameters) == ERROR_CODE::SUCCESS) {

            zed.retrieveImage(depth_image_zed_display, VIEW::DEPTH, MEM::CPU, new_image_size);

            zed.retrieveMeasure(depth_image_zed, MEASURE::DEPTH, MEM::CPU, new_image_size);
            cv::Mat dst;
            //cv::boxFilter(depth_image_ocv_display, dst, -1, cv::Size(1, 1));

            cv::Mat result;
            double factor = 0.5;
            cv::addWeighted(depth_image_ocv, 1.0 - factor, last_image, factor, 0.0, result);
            last_image = result;

            cv::imshow("dep", depth_image_ocv_display);

            cv::Mat ucharMat, ucharMatScaled;
            result.convertTo(result, CV_8UC1);
            // scale values from 0..1 to 0..255
            result.convertTo(ucharMatScaled, CV_8UC1, 255, 0);
            cv::imshow("filtered_ dep", ucharMatScaled);

            // Handle key event
            key = cv::waitKey(10);
edit retag flag offensive close merge delete


please rather add a single image here, than a link to a video elsewhere, thank you.

berak gravatar imageberak ( 2020-11-01 03:37:43 -0500 )edit

@break I added a single image

Ahmed gravatar imageAhmed ( 2020-11-01 03:41:49 -0500 )edit

can you take a look at it again, and clarify, where all those Mat's come from, and where they go ?

does your camera move ?

I'm trying to do temporarl removal on depth image

what does it mean ? removal of what ? explain, please.

berak gravatar imageberak ( 2020-11-01 03:58:56 -0500 )edit

@break, I wanna do temporal filtering on that image, I see a lot of spots that appear in the screen. Yes the camera moves, and i see accumlated ghosty image that fades the filtered image into black when I move

Ahmed gravatar imageAhmed ( 2020-11-01 04:33:12 -0500 )edit

i've no idea about the range of your depth images, but this:

 result.convertTo(result, CV_8UC1);

might simply binarize it if it was in [0..1] before

hard to say anything with that incommplete code, please update

what you're producing there is some kind of "echo buffer". if that's not what you wanted, you'll have to explain better what you wanted instead

berak gravatar imageberak ( 2020-11-01 07:41:31 -0500 )edit

@break I removed that line you mentioned, I'm getting an echo buffer still, like the picture is accumlated to black every frame, that's the complete code actually above. My main complaint is to remove temporarl effect, like popping up spots on the depth image, how would I remove that ?

Ahmed gravatar imageAhmed ( 2020-11-01 09:27:32 -0500 )edit

@break the depth image is C1_32F

Ahmed gravatar imageAhmed ( 2020-11-01 09:27:57 -0500 )edit

no, it's not complete, we don't know, what depth_image_ocv or depth_image_ocv_display are

popping up spots on the depth image

i can't see any of it in your image above (so again, please show a "problematic" one !), and it's also unclear, if this is a temporal effect even, or "speckles" from the disparity generation. i also have no idea, how the zed folks generate the depth image

as long as the camera moves, your idea above won't work, simply said.

berak gravatar imageberak ( 2020-11-01 09:35:10 -0500 )edit

@breal I uploaded a video, which shows two things 1. Speckles 2. Ghosting after filtering the depth. Please take a look at it. The above video is done using the above code.

Ahmed gravatar imageAhmed ( 2020-11-01 09:43:06 -0500 )edit

@break ok, lets focus on something, how would I do temporal remove on a depth image ?

Ahmed gravatar imageAhmed ( 2020-11-01 10:01:31 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2020-11-01 11:28:47 -0500

berak gravatar image

updated 2020-11-01 12:25:38 -0500

your current addWeighted() approach seems to consider all images during the runtime of the program, while this probably requires a constant number of images to interpolate

for a fixed camera, you could use a ringbuffer/fifo/lifo to calculate a "moving average over time" like:

std::deque<cv::Mat> queue;
int N=8;
while (1) {
    cv::Mat new_depth_frame = .... // from zed cam
    queue.push_back(new_depth_frame / N);
    cv::Mat avg = Mat::zeros(new_depth_frame.size(), CV_32F); // EDIT!
    if (queue.size() > N) {
        for (cv::Mat q : queue) 
            avg += q;
    if (! avg.empty()) { //queue not full yet
        // now use avg ...
edit flag offensive delete link more


tried your solution, I get access violation on avg+=q, being avg is empty

   while (key != 'q')
        zed.retrieveMeasure(depth_image_zed, MEASURE::DEPTH, MEM::CPU, new_image_size);

        cv::Mat new_depth_frame = depth_image_ocv.clone();
        queue.push_back(new_depth_frame / N);
        cv::Mat avg;
        if (queue.size() > N) {
            for (cv::Mat q : queue)
                avg += q;
        if (!avg.empty()) { //queue not full yet
            // now use avg ...

            cv::Mat ucharMat, ucharMatScaled;
            avg.convertTo(avg, CV_8UC1);
            // scale values from 0..1 to 0..255
            avg.convertTo(ucharMatScaled, CV_8UC1, 255, 0);
            cv::imshow("filtered_ dep", ucharMatScaled);

Ahmed gravatar imageAhmed ( 2020-11-01 11:59:21 -0500 )edit

there was a problem with initializing the cv::avg, well it has the same ghosting effect!!! I'm getting Crazy

Ahmed gravatar imageAhmed ( 2020-11-01 12:15:46 -0500 )edit

apologies, sure, see EDIT

and again, this only makes sense, if the camera does NOT move

berak gravatar imageberak ( 2020-11-01 12:26:03 -0500 )edit

If the camera doesn't move, still I get black lines the are drawing

Ahmed gravatar imageAhmed ( 2020-11-01 13:11:41 -0500 )edit

The real question is, There is no method to do temporal removal at all in OpenCV that work out of the box ? even with moving Camera ?

Ahmed gravatar imageAhmed ( 2020-11-01 13:35:40 -0500 )edit

here is intel article about that topic

Ahmed gravatar imageAhmed ( 2020-11-01 13:37:22 -0500 )edit

Question Tools

1 follower


Asked: 2020-11-01 03:32:53 -0500

Seen: 397 times

Last updated: Nov 01 '20