# how to stream h264 video with rtsp in opencv- partially answered

I've been struggling with this for a few months now and thought I should consolidate my efforts and "research" into this question all in one place. Mainly I am doing this to help those that have this issue or will face it in the future, so they don't have to waste valuable time.

It appears that others have struggled as well, since OpenCV uses ffmpeg library for rtsp and such, but has decoding errors for at least h264 video streaming (See link1, link2, link3 for some examples of broken images).

I have multiple IP cameras that I connected to a switch, and then need to process the video feeds through the rtsp commands that was given by the camera manufacturer: "rtsp://username:[email protected]:554" (Here, 554 is the RTSP port number assigned to the camera, which may require you to go an adjust the number by logging into the camera and accessing the camera settings)

I wanted to use my OpenCV code but could not just use: VideoCapture cap1("rtsp://username:[email protected]:554"); or, cap1.open("rtsp://username:[email protected]:554"); because of the broken video feeds from FFMPEG as explained above.

Trying to do the TCP protocol instead of UDP may have fixed the issue for others, but this is already incorporated in the latest versions of OpenCV (it was for 2.4.11. and a new update should fix it for 3.0.0; see this bug report) and did not work for me.

edit retag close merge delete

I will shortly post some solutions I've come up with, that will hopefully help others in similar struggles. I just have to wait 2 days according to the site rules.

( 2015-07-09 14:58:59 -0600 )edit

may be a pull request would be better

( 2015-07-09 15:10:27 -0600 )edit

Hi #LBerger, what is a pull request and how do you do one?

( 2015-07-10 14:16:13 -0600 )edit

If your code is really good you can share it here but you can propose that your code will be insert in github (a pull request). That's not very difficult. You can follow this or this post. You can see some new pull request here

( 2015-07-10 14:38:20 -0600 )edit

Thanks LBerger! That is a great option, thanks for letting me know! Mine is how to get all the pieces and integration done, not necessarily a source code to solve it all. The source code that I do use is online, so I reference that, but mostly it's about all the troubleshooting I had to go through to get libVLC installed, integrated, and talking with OpenCV on MS Visual Studio for a Windows machine. Nothing great, but thought it would be helpful for others since it took me many weeks and tons of frustrated days to get to this point. lol I do also like the idea of writing a tutorial maybe in the future for stuff that's more heavy-duty, so thanks for the quick help and advice #LBerger! :)

( 2015-07-10 17:30:34 -0600 )edit

Sort by » oldest newest most voted

After summarizing my question and struggles, here is the answer and solution I've come up with so far. I've posted the question and the solution so others don't have to search through many other posts that don't quite solve the issues I've had, and to share details that I had to figure out which helped me. I am using Visual Studio on MS Windows 7 and 8.1 machines and coding in C++.

The video stream of 2 cameras works with the following minor issues: - I need to figure out how to display them separately instead of alternating on the same window. This is not a problem for my application since I just need to process them together, not display them simultaneously. - About a 1 second delay.

1) It seems clear that FFMPEG library does not work for H264 video streams over RTSP. So I downloaded and installed libVLC along with the VLC Player.

a) libVLC installation: Download the 7zip file (no installer version) so as to get all the library header files, DLL's and source codes. Then link them, almost exactly like you installed and linked OpenCV (I used local method: system paths and environmental variables, after which I had to reboot). - I needed the libvlc.dll, libvlccore.dll, the headers and the .lib files, so you will have to find these and link them.

The player is for testing, since sometimes what you have in the code may be wrong. I recommend testing your URL directly in the player when you find errors you cannot explain. This allows you to test the RTSP or HTTP command you should use exactly in your code (go to Media -> Open Network Stream -> Network -> "Please enter a network URL:" -> and then enter "rtsp://username:[email protected]:554" for example).

2) Sample libVLC + OpenCV code to use to test: I used this which is similar to the VLC = VideoLan example.

• It appears that the "unlock" callback function is one of the places to process the MAT image directly (in libvlc_video_set_callbacks(mediaPlayer, lock, unlock, display, context), for example here)

3) Some troubleshooting I had to do:

a) Undeclared identifiers: This implies missing headers. I needed to include the following for libVLC in addition to OpenCV headers: vlc/libvlc.h vlc/libvlc_media.h vlc/libvlc_media_player.h

b) x86 vs. x64: I kept getting this error since libVLC works for 32 bit and not for 64 bit. You can still use libVLC on a 64 bit machine (which is what I have), but need to change the configuration to 32 bit. I kept getting errors despite changing this in configuration manager, and resolved it by simply creating a new solution from scratch and then copying my code. (See link).

c) Stopped videos: occasionally some errors would show up on the back end, and the first 5 seconds or so were somewhat corrupted for brief periods. After a while, my videos just ...

more

It seems that the latest version of opencv(3.4) has fixed this problem.

I met the same problem while using opencv 2.4.13 on windows. After updating to opencv 3.4, the problem is gone! Thanks to the contributors!

( 2018-01-24 02:46:35 -0600 )edit

@manfeyn Do you mean Opencv 3.4 can capture h264 videos over rtsp streams?

( 2018-03-15 09:52:34 -0600 )edit

I dont think so, I tried installing openCV 3.4 and it couldn't. What this accepted answer has suggested worked perfectly, although I still had to dig around for a while to get it to work.

( 2018-05-10 20:34:34 -0600 )edit

@malharjajoo get streaming h264 did you replace the open CV capture to VLC libvlc_media_new_location?

( 2018-07-29 22:05:51 -0600 )edit

Blockquote

Hi, I have had similar problems (and solved them) as I'll explain below. The approach to solve it, is nearly identical. My O.S is Windows 7 64 bits and I was using VC++2013.

• When streaming H264 video from IP cameras using RTSP with OpenCV , the program crashes after some minutes without a clear reason (tested it with OpenCV 3 and OpenCV 2.4.9...).

• Tried to use FFMPEG, but compiling FFMPEG was quite hard on Windows. I could not compile it correctly following the instructions I've found.

The solution: I tried libVLC and got it working for 32bits EXEs and 64 bits EXEs using:

• DLLs and plugins from VLC player 2.2.1 (if you want to generate a 64 bit EXE use the DLLs from VLC 2.2.1 64 bits, if you want to generate a 32 bit EXE use the DLLs from VLC 2.2.1 32 bits).

• .lib files and .h files for libVLC 32bits and 64 bits can be found here: https://github.com/RSATom/libvlc-sdk

• I created a simple Console project following this tutorial (https://wiki.videolan.org/LibVLC_Samp...) and added some modifications. (See the full code below). Notice the code does not use OpenCV at all, but once you got the frames, you can add OpenCV to run any process on them.

I would love to see a better RTSP support in OpenCV because everything would be easier but this is the only way I've found to solve the crashes. Please, check the code because it is a fast try...it could contain bugs.

#include <iostream>
#include <stdio.h>
#include <stdint.h>
#include <math.h>
#include <stdlib.h>
#include <assert.h>

#include "SDL.h"
#include "SDL_mutex.h"
#include "vlc/vlc.h"

#define WIDTH 800
#define HEIGHT 600

#define VIDEOWIDTH 320
#define VIDEOHEIGHT 240

struct context {
SDL_Renderer *renderer;
SDL_Texture *texture;
SDL_mutex *mutex;
int n;
};

// VLC prepares to render a video frame.
static void *lock(void *data, void **p_pixels) {

struct context *c = (context *)data;

int pitch;
SDL_LockMutex(c->mutex);
SDL_LockTexture(c->texture, NULL, p_pixels, &pitch);

return NULL; // Picture identifier, not needed here.
}

// VLC just rendered a video frame.
static void unlock(void *data, void *id, void *const *p_pixels) {

struct context *c = (context *)data;

uint16_t *pixels = (uint16_t *)*p_pixels;

// We can also render stuff.
int x, y;
for (y = 10; y < 40; y++) {
for (x = 10; x < 40; x++) {
if (x < 13 || y < 13 || x > 36 || y > 36) {
pixels[y * VIDEOWIDTH + x] = 0xffff;
}
else {
// RV16 = 5+6+5 pixels per color, BGR.
pixels[y * VIDEOWIDTH + x] = 0x02ff;
}
}
}

SDL_UnlockTexture(c->texture);
SDL_UnlockMutex(c->mutex);
}

// VLC wants to display a video frame.
static void display(void *data, void *id) {

struct context *c = (context *)data;

SDL_Rect rect;
rect.w = VIDEOWIDTH;
rect.h = VIDEOHEIGHT;

// display the video in a random location
rect.x = (int)((1. + .5 * sin(0.03 * c->n)) * (WIDTH - VIDEOWIDTH) / 2);
rect.y = (int)((1. + .5 * cos(0.03 * c->n)) * (HEIGHT - VIDEOHEIGHT) / 2);

// display the video in the top left corner ...
more

im getting an error unable to link libvlc.dll can you please help me how to link the dll

( 2016-06-07 06:17:28 -0600 )edit

Official site

GitHub

Wiki

Documentation

## Stats

Asked: 2015-07-09 13:51:35 -0600

Seen: 49,506 times

Last updated: Jan 22 '16