How to optimize frame grabbing from video stream in OpenCV?

edit

Hello everyone,

I ran into a problem problem of low frame capture efficiency in OpenCV.

Hardware & Software.

• Raspberry Pi 3 (1,2 GHz quad-core ARM) with HDMI Display
• IP camera: LAN connected, RTSP, H264 codec, 1280x720 resolution, 20 fps, 1 GOP, 2500 kB/s VBR bitrate (parameters can be changed).
• OS Raspbian Stretch
• Python 3.5
• OpenCV 4.1
• Gstreamer 1.0

Task.

Get videostream from IP camera, recognize images and display resulting video (with marks and messages).

Important features: real-time processing, HD resolution (1280x720), high frame rate (>20 fps), continuous operation for several hours.

My solution.

General algorithm: source video stream -> decoding and frame grabbing -> work with frames in OpenCV -> assembling the processed frames into a video stream -> display video using a Raspberry Pi GPU

OpenCV output/display method - imshow - does not work well even at low-resolution video. The only library that allows to use a Raspberry Pi GPU to decode and display video is a Gstreamer. I compiled Gstreamer modules (gstreamer1.0-plugins-bad, gstreamer1.0-omx) with OMX support and tested it:

gst-launch-1.0 rtspsrc location='rtsp://web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! glimagesink

It works great, CPU usage is about 9%.

Next I compiled OpenCV with Gstreamer, NEON, VFPV3 support.

I use the following code for testing:

import cv2
import numpy as np

src='rtsp://web_camera_ip'
stream_in = cv2.VideoCapture(src)

pipeline_out = "appsrc ! videoconvert ! video/x-raw, framerate=20/1, format=RGBA ! glimagesink sync=false"
fourcc = cv2.VideoWriter_fourcc(*'H264')

stream_out = cv2.VideoWriter(pipeline_out, cv2.CAP_GSTREAMER, fourcc, 20.0, (1280,720))
while True:
    ret, frame = stream_out.read()
    if ret:
      stream_out.write(frame)
      cv2.waitKey(1)

It also worked, but not so well as Gstreamer itself. CPU usage is about 50%, without stream_out.write(frame) - 35%. At frame rate above 15, there are lags and delays.

How I tried to solve the problem.

A. Use Gstreamer to decode video stream:

pipline_in='rtspsrc location=rtsp://web_camera_ip latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! appsink'
stream_in = cv2.VideoCapture(pipline_in)

It even worsened the situation - the CPU load increased by several percent, the delay has become more.

B. I also tried to optimize the library using methodfrom PyImageSearch.com - threading using WebcamVideoStream from imutils library.

   from threading import Thread
    import cv2
    import numpy as np
    import imutils

    src='rtsp://web_camera_ip'
    stream_in = WebcamVideoStream(src).start()
    pipeline_out = "appsrc ! videoconvert ! video/x-raw, framerate=20/1, format=RGBA ! glimagesink sync=false"
    fourcc = cv2.VideoWriter_fourcc(*'H264')

    stream_out = cv2.VideoWriter(pipeline_out, cv2.CAP_GSTREAMER, fourcc, 20.0, (1280,720))
    while True:
        frame = stream_in.read()
        out.write(frame)
        cv2.waitKey(1)

CPU usage has increased to 70%, the quality of the output video stream has not changed.

C. Сhanging the following parameters does not help: whaitKey(1-50), videostream bitrate (1000-5000 kB/s), videostream GOP (1-20).

Questions.

As I understand, VideoCaputre/Videowritter methods has a very low efficiency. Maybe it's not noticeable on PC, but it is critical for Raspberry Pi 3.

Is it possible to increase the performance of the VideoCaputre (Videowritter)? Is there an alternative ... (more)