Ask Your Question
0

sometimes failed to read frame from rtsp with opencv

asked 2018-11-07 10:05:13 -0500

softarts gravatar image

updated 2018-11-08 02:31:51 -0500

https://stackoverflow.com/questions/5...

I got a strange issue. my code is quite simple, just read frame from RTSP and do some processing.

for (;;)
{
    cap >> frame;
    if (frame.empty()) {
        break;
    }
    // heavy_processing(frame) // opencv-dnn face detection stuff
    imshow("cam",frame);
}

the rtsp streaming is a webcam with 1920x1080P, network traffic is arround 4Mb/s

without "heavy_processing()" function, basically it runs good. CPU is 45% (i7-7700)

with "heavy_processing()" function, it will read an empty frame after couple seconds then exit. I can see CPU surge to 95%+ before it exit.

I built opencv(3.4.3) myself with mostly default option.

so far I suspect the videocap device shutdown when there are too much frames in the buffer, but it doesn't make sense.shouldn't it drop the earliest frame? or is it possible the frame is corrupted inside 'heavy_processing()'?

ffmpeg probably is 4.0(built myself), ubuntu 16.04

update:

the heavy_processing() function, suspect it is related.

void detect(Mat frame, Mat blob, Net &net, Size &inpSize, vector<cv::Rect> &face_rects) {
    Mat detectFrame;
    cv::resize(frame, detectFrame, cv::Size(300, 300), 0, 0, cv::INTER_AREA);
    dnn::blobFromImage(detectFrame, blob, 1.0, inpSize, mean, false, false);
    // if I return here is good
    net.setInput(blob);
    Mat detection = net.forward();   // as long as I add the last two line, it will get read frame empty

update:

the detect function will spread over multiple cores(mine is 8 cores i7, total cpu usage is less than 50%), so CPU resource should be enough.

just in case there is race condition, I have added some lock in detection/capture/display part, but actually the code is still in the same thread as capture/display(however net itself might spawn some threads),

the problem is still not resolved yet.

tried opencv 4.0 as well but not help. change 'break' to 'continue' not help because it will keep reading empty frame after that.

edit retag flag offensive close merge delete

Comments

1
sturkmen gravatar imagesturkmen ( 2018-11-07 10:23:56 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-11-08 02:00:14 -0500

berak gravatar image

updated 2018-11-08 02:02:23 -0500

so, it seems you're stalling the underlying ffmpeg library, resulting in invalid frames. things to try:

  • choose a smaller / faster network (e.g. tiny-yolo or squeezedet)
  • since it seems to be a detection network, try a smaller input, than 300x300 (the 1st few layers are the most expensive !)
  • profile your network
  • don't process every frame, but maybe only every 20'th
  • if you have several cores, maybe try to offload the cnn processing to a different thread (but careful ! needs locks / safe queues and whatnot ! and both capture and gui have to stay on the main thread.)
  • don't break out of the loop in case of a missing frame, rather continue and try again..
edit flag offensive delete link more

Comments

thanks, see my update in the question.

softarts gravatar imagesoftarts ( 2018-11-08 02:32:07 -0500 )edit

finally I found a solution: I split these tasks into 3 threads:capture, process, display, I use thread-safe queue to communicate among these threads.the problem was gone now. so I suspect heavy cpu load will cause opencv to lost connection with rtsp streaming thanks a lot

softarts gravatar imagesoftarts ( 2018-11-12 00:23:18 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-07 10:05:13 -0500

Seen: 44 times

Last updated: Nov 08