Using fisheye correction program works over webcam, but not rtsp stream?

asked 2018-06-22 08:13:56 -0500

tommy61157 gravatar image

So, I'm trying to make a program using openCV to make a fisheye camera have a regular image in real time from an rtsp stream, I started with this code: http://aishack.in/tutorials/calibrati... and am now trying to adapt it so that it will work with an IP camera that I get the video from using rtsp. It works with a webcam the way it says it's supposed to in the link, however, when I put in an rtsp stream, It seems to just freeze on the first frame and not be taking video at all.

// ConsoleApplication2.cpp : Defines the entry point for the console application.

#include <opencv2\videoio.hpp>
#include <opencv2\highgui.hpp>
#include <opencv\cv.hpp>
#include <opencv\cv.h>
#include <iostream>
#include <stdio.h>
#include <chrono>
#include <thread>


using namespace cv;
using namespace std;

int main (){
int numBoards = 0;
int numCornersHor;
int numCornersVer;

printf("Enter number of corners along width: ");
scanf_s("%d", &numCornersHor);

printf("Enter number of corners along height: ");
scanf_s("%d", &numCornersVer);

printf("Enter number of boards: ");
scanf_s("%d", &numBoards);

int numSquares = numCornersHor * numCornersVer;
Size board_sz = Size(numCornersHor, numCornersVer);

VideoCapture capture = VideoCapture("rtsp://172.16.127.27:554/mpeg4"); //Works just fine if value is set to 0 which grabs webcam by default.

vector<vector<Point3f>> object_points;
vector<vector<Point2f>> image_points;

vector<Point2f> corners;
int successes = 0;

Mat image;
Mat gray_image;
capture >> image;

vector<Point3f> obj;
for (int j = 0; j < numSquares; j++)
    obj.push_back(Point3f(j / numCornersHor, j%numCornersHor, 0.0f));

while (successes < numBoards) {
    this_thread::sleep_for(chrono::milliseconds(100)); // Tried this to alleviate it, but it did nothing, still happens with or without this.
    cvtColor(image, gray_image, CV_BGR2GRAY);

    bool found = findChessboardCorners(image, board_sz, corners, CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FILTER_QUADS);

    if (found) {
        cornerSubPix(gray_image, corners, Size(11,11), Size(-1, -1), TermCriteria(CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 30, 0.1));
        drawChessboardCorners(gray_image, board_sz, corners, found);
    }

    imshow("win1", image);
    imshow("win2", gray_image);

    capture >> image;
    int key = waitKey(1);

    if (key == 27)

        return 0;

    if (key == ' ' && found != 0){
        image_points.push_back(corners);
        object_points.push_back(obj);

        printf("Snap stored!");

        successes++;

        if (successes >= numBoards)
            break;
    }
}

Mat intrinsic = Mat(3, 3, CV_32FC1);
Mat distCoeffs;
vector<Mat> rvecs;
vector<Mat> tvecs;

intrinsic.ptr<float>(0)[0] = 1;
intrinsic.ptr<float>(1)[1] = 1;

calibrateCamera(object_points, image_points, image.size(), intrinsic, distCoeffs, rvecs, tvecs);

Mat imageUndistorted;
while (1) {
    capture >> image;
    undistort(image, imageUndistorted, intrinsic, distCoeffs);

    imshow("win1", image);
    imshow("win2", imageUndistorted);
    waitKey(1);
}

capture.release();

return 0;
}

I'm still rather new to OpenCV, so I'm open to any and all suggestions and I'm still trying to fully learn what is going on here, eventually, I want this to translate into just having it take in the video and then spitting it back out modified correctly without the fisheye effect, just having to take this one step at a time since this is proving to be incredibly difficult which is odd because there are cameras that do it by default in their firmware, but those cameras are not always an option for who I'm working for.

edit retag flag offensive close merge delete

Comments

do you have physical access to that ip cam ? (remember, you need to wave chessboards in fronmt of it for the calibration)

then, imho, you have to split the program / problem into 3 parts:

  1. accessing the ip cam in general (try without any calibration or undistort code) if your host is on windows, make sure, it finds opencv_ffmpeg.dll

  2. actually calibrate the camera (must be done only once). and save your intrinsic Mat and the distortion params. if it's a fisheye cam, you also have to use cv::fisheye functions for this, not the pinhole model ones.

  3. for the daily usage later, you need to load the intrinsics/ dist_coeffs, get images from your ip cam, and undistort them.

berak gravatar imageberak ( 2018-06-22 08:49:54 -0500 )edit
1

I do have physical access to it, sorry, thought that was implied. If you look, there is one image that goes through on win1 that isn't edited when it outputs, but I'll still make a separate program for it. 2 and 3 are what I'll do later, just have to get this resolved first. I'll comment again once I try a regular stream without editing.

tommy61157 gravatar imagetommy61157 ( 2018-06-22 09:33:57 -0500 )edit

So, I do have it working on its own, but all this tells me is that something is going wrong in the distorting effects themselves, which I already kinda figured. I'm just wondering if there's something I'm doing improperly or anything else that could be interfering with the effect on an IP camera that wouldn't show up on a webcam, because like I commented in the code, it does work with a webcam. Just not an IP camera.

tommy61157 gravatar imagetommy61157 ( 2018-06-22 09:43:33 -0500 )edit

sorry, but the last comment is a bit garbled. try again ?

also, usually it's easier to calibrate with an imagelist (collect a lot chessboard images before, and sort out the best, try different imagesets), than repeating the "chessboard waving" over and over.

berak gravatar imageberak ( 2018-06-22 09:46:43 -0500 )edit

My point is I'm looking for an actual solution as to what exactly in this code is causing the camera to freeze on my IP camera while it works completely fine on my Webcam since I don't know what would be doing it.

Edit: However, maybe it could have something to do with what you were saying where I may have to use the fisheye effects for it and not the pinhole ones, although I still don't entirely get why this would cause a freeze.

Edit 2: How am I supposed to do multiple when I can't even get one to work since the camera seems to freeze on the first frame it captures?

tommy61157 gravatar imagetommy61157 ( 2018-06-22 09:49:02 -0500 )edit

findChessboardCorners (also with subpixels) has a quite high workload, it's no wonder, that it gets slow.

berak gravatar imageberak ( 2018-06-22 09:51:05 -0500 )edit

That still doesn't answer the question of why my webcam can do it just fine when my IP Camera just completely freezes.

tommy61157 gravatar imagetommy61157 ( 2018-06-22 09:52:19 -0500 )edit

While I do appreciate you showing me that, it is unfortunately unhelpful in providing any information that will help me in this situation.

tommy61157 gravatar imagetommy61157 ( 2018-06-27 14:39:55 -0500 )edit

yea, sorry.

berak gravatar imageberak ( 2018-06-27 14:48:56 -0500 )edit