Time delay in VideoCapture opencv due to capture buffer

asked 2015-10-25 22:10:58 -0600

mrmuto2
51 ●1 ●2 ●6

updated 2015-10-28 14:08:59 -0600

4284 ●7 ●30 ●60

I use a camera to get frames continuesly.In my code,i have use VideoCapture to do that. But when i debug,i find that the frame i got is the old frame. I have asked some guys, they told me that some old frames will store in the buffer and i should use a worker thread to get the latest frame.Also there is a similar question here. Similar question

Here, i have two questions,

If the buffer can store 5 frames and now have stored 5 frames, When does the buffer update?After 5 frames have been got out or just 1 frame? Because i want to know whether i got old frames all the time or i got one new frame then four old frames then change to one lastest frame...

I have debug my code, set a breakpoint at cap >> frame. First frame(loop) Put my hand in and second loop put my hand out of the camera, I found that the image will changed after few frames. Does it mean i got one new frame then four old frames then change to one lastest frame...?

2.Because it's my first time to use thread, i am not sure my code is wrong or not, here is my code,

void task(VideoCapture cap, Mat& frame) {
    while (true) {
          cap >> frame;
    }
}

int main() {
    Mat frame, image;
    VideoCapture cap;
    cap.open(0);
    cap.set(CV_CAP_PROP_FRAME_WIDTH, 1600);
    cap.set(CV_CAP_PROP_FRAME_HEIGHT, 1080);
    cap >> frame;
    thread t(task, cap, frame);
    while (true) {
        initUndistortRectifyMap(
            cameraMatrix,  // computed camera matrix
            distCoeffs,    // computed distortion matrix
            Mat(),     // optional rectification (none) 
            Mat(),     // camera matrix to generate undistorted
            Size(1920 * 1.3, 1080 * 1.3),
            //            image.size(),  // size of undistorted
            CV_32FC1,      // type of output map
            map1, map2);   // the x and y mapping functions
    remap(frame, frame, map1, map2, cv::INTER_LINEAR);
          frame.copyTo(image);
          ...//image processing loop
    }
}

I also have set breakpoint in task() to check the frame, but is the same condition. The image will changed after few frames.I don't know why. Is my code wrong or I can not set breakpoint to check like this as using thread? Can somebody help?Thank you very much.

edit retag flag offensive close merge delete

Comments

I think you have a problem with frame.copyTo(image) You cannot copy frame while acquiring a new image you will need a mutex. In your link that's not exactely same problem because it's IP cam. Answer given is for IEEE 1394 Have you got a such cam?

Add in your thread

 imshow("Image thread",frame);
  waitKey(1);

and in your main

 imshow("Image main",image);
  waitKey(1);

LBerger ( 2015-10-26 02:19:23 -0600 )edit

@LBerger Thank you for your comment.Because It's my first time to use thread, i don't know why i need to use mutex here? frame.copyTo(image) just copy the latest frame to image.Also,can you explain that why i need to use imshow?If i use imshow do i needn't use mutex?

mrmuto2 ( 2015-10-26 03:33:55 -0600 )edit

Problem occur when thread and main try to access data at same time : main start a copy at same time a new image is ready (cap >>frame) What will be data in image? You can have bus error too.

Now I have try to have an error with your code I didn't succeed.

I can set breakpoint using VS2013 in thread and in main without problem. There is no latency

Thanks for your question I have learnt something don't use reference in argument for thread

LBerger ( 2015-10-26 03:59:42 -0600 )edit

@LBerger Thank you for your answer.You mean that my code is right?I haven't get error.But i'm not sure my code is right or not?Also i want to know do i need to use imshow or mutex here?If i use imshow do i needn't use mutex?Sorry for so many question.

mrmuto2 ( 2015-10-27 08:21:40 -0600 )edit

may be like this

std::mutex mtxCam;

using namespace std;
using namespace cv;

void task(VideoCapture *cap, Mat *frame) 
    {
        while (true) 
        {
            mtxCam.lock();
            *cap >> *frame;
            mtxCam.unlock();
            imshow("Image thread",*frame);
            waitKey(1);
        }
    }
int main() {
    Mat frame, image;
    VideoCapture cap;
    cap.open(0);
    cap>>frame;
    thread t(task, &cap, &frame);
    while (true) {
        if (!frame.empty())
        {
            mtxCam.lock();
            frame.copyTo(image);
            mtxCam.unlock();
            imshow("Image main",image);
            waitKey(12);
        }
    }
}

LBerger ( 2015-10-27 11:24:18 -0600 )edit

@Thank you very much!I think this would be help.But in your code when program exits,error happened.It said "mutex destroyed while busy".I tried to solve it,but failed,can you give me some advice?Also, i have question,i have edited my code in my question.When i use remap(), the frame haven't updated.The frame is always the first frame, while i use metux,the result is same.I don't know why

mrmuto2 ( 2015-10-28 05:46:58 -0600 )edit

A small sidenote, we also have a camera here with a buffer of 5 frames. To avoid this issue we simply use the native SDK of the camera to disable camera frame buffering, then all your misery is gone.

StevenPuttemans ( 2015-10-28 09:23:58 -0600 )edit

I think you have to terminate thread before leaving your program. And about update problem I don't know and you should follow @StevenPuttemans comments

LBerger ( 2015-10-28 12:07:57 -0600 )edit

add a comment

answered 2015-10-28 11:23:47 -0600

pklab

4284 ●7 ●30 ●60

updated 2015-12-22 04:24:05 -0600

Thread scheme is too poor because grab and process aren't sync. Depends on grab/process timing you will lost some frames or process same frame twice. You could implements some sync using mutex||semaphores||std::atomic<bool> but it's would be better if you use a thread safe circular buffer with copy constructor, in this case 1 producer 1 consumer: Grab/Push and Process/Pop.

A simple implementation could be done use std:queue but isn't memory effective because you will create a new Mat for each grab (even if STL and Mat optimization will recycle the memory). look at this simple example and try moving the window "Image main"

EDIT: keep a copy of grabbed frame is safer. EDIT2: move grab out of locking. EDIT3: removed useless VideoWriter. Added counter for cv::Mat memory allocation/recycling

#include <thread>
#include <mutex>
#include <queue>
#include <atomic>

std::queue<Mat> buffer;
std::mutex mtxCam;
std::atomic<bool> grabOn; //this is lock free

void GrabThread(VideoCapture *cap)
{
    Mat tmp;

    //To know how many memory blocks will be allocated to store frames in the queue.
    //Even if you grab N frames and create N x Mat in the queue
    //only few real memory blocks will be allocated
    //thanks to std::queue and cv::Mat memory recycling
    std::map<unsigned char*, int> matMemoryCounter;
    uchar * frameMemoryAddr;

    while (grabOn.load() == true) //this is lock free
    {
        //grab will wait for cam FPS
        //keep grab out of lock so that 
        //idle time can be used by other threads
        *cap >> tmp; //this will wait for cam FPS

        if (tmp.empty()) continue;

        //get lock only when we have a frame
        mtxCam.lock();
        //buffer.push(tmp) stores item by reference than avoid
        //this will create a new cv::Mat for each grab
        buffer.push(Mat(tmp.size(), tmp.type()));
        tmp.copyTo(buffer.back());
        frameMemoryAddr = buffer.front().data;
        mtxCam.unlock();
        //count how many times this memory block has been used
        matMemoryCounter[frameMemoryAddr]++; 

        bool show = true;
        if (show)
        {
            int font = CV_FONT_HERSHEY_PLAIN;
            putText(tmp, "THREAD FRAME", Point(10, 10), font, 1, Scalar(0, 255, 0));
            imshow("Image thread", tmp);
            waitKey(1);    //just for imshow
        }
    }
    std::cout << std::endl << "Number of Mat in memory: " << matMemoryCounter.size();
}

void ProcessFrame(const Mat &src)
{
    if(src.empty()) return;
    putText(src, "PROC FRAME", Point(10, 10), CV_FONT_HERSHEY_PLAIN, 1, Scalar(0, 255, 0));
    imshow("Image main", src);
}
int MainProc() {
    Mat frame;
    VideoCapture cap;
    cap.open(0);
    if (!cap.isOpened()) //check if we succeeded
        return -1;

    grabOn.store(true);                //set the grabbing control variable
    thread t(GrabThread, &cap);          //start the grabbing thread
    int bufSize;
    while (true)
    {
        mtxCam.lock();                //lock memory for exclusive access
        bufSize = buffer.size();      //check how many frames are waiting 
        if (bufSize > 0)              //if some 
        {
            //reference to buffer.front() should be valid after 
            //pop because of Mat memory reference counting
            //but if content can change after unlock is better to keep a copy
            //an alternative is to unlock after processing (this will lock grabbing)
            buffer.front().copyTo(frame);   //get the oldest grabbed frame (queue=FIFO)
            buffer.pop ...

(more)

edit flag offensive delete link

Comments

Thank you for your answer. "Depends on grab/process timing you will lost some frames or process same frame twice. But in my case it just process same frame all the time.It did not update.Also in my code i have use mutex which @LBerger show me.

mrmuto2 ( 2015-10-29 00:54:37 -0600 )edit

@mrmuto2 your issue is still present using my code ? It grabs into a queue and this should dump all frames from camera buffer with FIFO logic. or not ?

pklab ( 2015-10-29 03:38:47 -0600 )edit

It works now.Thank you very much.But when i exit the program it says "the mutex destroyed while busy"

mrmuto2 ( 2015-10-29 07:15:03 -0600 )edit

Well, if it works please accept the answer for future reference.

Regards to "the mutex destroyed while busy" you have to ensure a call to mtxCam.unlock(); before leaving your program. In my sample, this is done by grabOn.store(false); t.join(); that are called after that a key is pressed on Image main window.

pklab ( 2015-10-29 10:15:54 -0600 )edit

@pklab I was trying your code above and it works like a charm. Many thanks for it. However, I have a question. What I noticed is that while both grabber and processor are working simultaneously as it is now, in case that there are some frames stored in the buffer the processor after a while manages to process all of them and decrease the bufSize again to 0 reaching again to the point where one frame is grabbed and one frame is processed. However, if I comment the showing part in the grabber, i.e.:

putText(tmp, "THREAD FRAME", Point(10, 10), CV_FONT_HERSHEY_PLAIN, 1, cv::Scalar(0, 255, 0));
imshow("Image thread", tmp);
waitKey(1);    //just for imshow

and store some frames in the buffer then I noticed that the bufSize is not decreasing any more.

theodore ( 2015-12-21 09:08:34 -0600 )edit

Is that mean that now the grabber is faster than the processor (obviously) so the processor does not anymore manages to run faster than the grabber? Also regarding in the previous case and how fast the processor was decreasing the stored frames I wouldn't expect this behavior. Is actually in the first case the processor faster or it skips/flushes some frames and what is happening in the second case? Thanks.

theodore ( 2015-12-21 09:15:18 -0600 )edit

The grabber isn't faster because its timing is related to FPS...do you remember my prev answer ?. As opposite the simple processor is really faster. In this scenario the buffer should be almost empty. So you are right and wrong :=)

The issue is because the lock includes the grab call. Because grab wait for FPS, all idle time is spent while lock is active and the processor can't get the control over the buffer.

As soon as the frame is ready the grabber releases the lock and shows the frame.... at this step the processor can have its token for the buffer. If you remove imshow the lock is enabled immediately so no change will be for the processor. See EDIT2

pklab ( 2015-12-21 10:56:51 -0600 )edit

In real application locks and multiple copyTo can be avoided using better thread safe queue/circular buffer!

pklab ( 2015-12-21 11:03:07 -0600 )edit

@pklab thanks for the feedback it seems that you really know the subject something that I do not :-(, I am really grateful for all the feedback. I will try your new revised code tomorrow at my workstation. I think though that you have a mistake at OpenVideo(vw, tmp.size(), "../img/file2.avi"); line. Moreover, some other thoughts why did you use the VideoWriter() inside the grabber? wouldn't be better to be inside the processor? Also what would be better to use for multiple frames, individual circular buffers or a std::queue<vector<cv::Mat> > buffer;? Actually in order to give you an idea of what I want to do I will open another thread in order not to garbage this one here which was solved. I will pointed you there as soon as it is ready in order to continue there.

theodore ( 2015-12-21 15:57:15 -0600 )edit

@theodore Thank you for your appreciation.... OpenVideo(vw, ...) is my own function. I used it to show that is possible to use 2 parallel instance of cv::VideoWriter. I removed it now to avoid confusion.

As I said in the answer is a simple implementation There are many way to do this... my point of view: multi threading means not deterministic execution, unexpected exceptions, hard debugging. Without a strong background, sleepless night will expect you :)

I'll be happy to answer to your new question If it's in my hands

pklab ( 2015-12-22 02:42:07 -0600 )edit

see more comments

Time delay in VideoCapture opencv due to capture buffer

Comments

1 answer

Comments

Links

Question Tools

Stats

Related questions

Time delay in VideoCapture opencv due to capture buffer edit

Comments

1 answer

Comments

Links

Question Tools

Stats

Related questions

Time delay in VideoCapture opencv due to capture buffer