Ask Your Question
0

accurate method to calculate framerate of input stream

asked 2015-12-09 10:25:36 -0500

theodore gravatar image

updated 2015-12-10 10:16:57 -0500

Related to the latest questions of mine regarding kinect one and opencv I have a loop where I am transforming the images obtained from kinect to opencv matrices and then I am showing these matrices, the procedure is quite similar to the way we are using with the VideoCapture() class. A snippet code can be seen below:

    while( 1 ){
        // Frame

        // get color image
        IColorFrame* pColorFrame = nullptr;
        hResult = pColorReader->AcquireLatestFrame( &pColorFrame );
        if( SUCCEEDED( hResult ) ){
            hResult = pColorFrame->CopyConvertedFrameDataToArray( bufferSizeColor, reinterpret_cast<BYTE*>( colorBufferMat.data ), ColorImageFormat::ColorImageFormat_Bgra );
            // resize for visual reasons
            if( SUCCEEDED( hResult ) ){
                cv::resize( colorBufferMat, colorMat, cv::Size(), 0.5, 0.5 );
            }
        }

            // get depth image
//        // method 1
//        IDepthFrame* pDepthFrame = nullptr;
//        hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame );
//        if( SUCCEEDED( hResult ) ){
//            hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSizeDepth, reinterpret_cast<UINT16**>( &depthBufferMat.data ) );
//            if( SUCCEEDED( hResult ) ){
//                depthBufferMat.convertTo( depthMat, CV_8U, 0.05f/*-255.0f / 8000.0f, 255.0f*/ );
//            }
//        }

        // method 2
        IDepthFrame* pDepthFrame = nullptr;
        hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame );
        if( SUCCEEDED( hResult ) ){
            unsigned int bufferSize = 0;
            unsigned short* buffer = nullptr;
            hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSize, &buffer );
            if( SUCCEEDED( hResult ) ){
                for( int y = 0; y < depthHeight; y++ ){
                    for( int x = 0; x < depthWidth; x++ ){
                        unsigned int index = y * depthWidth + x;
                        BYTE intensity = buffer[ index ] == 0 || buffer[ index ] > 4095/*2047*/ ? 0 : 255 - (BYTE)(((float)buffer[ index ] / 4095.0f/*2047.0f*/) * 255.0f);
                        depthMat.at<unsigned char>( y, x ) = intensity;
                    }
                }
            }
//            memcpy(depth_im.data, buffer, depthHeight*depthWidth*sizeof(UINT16)); // creating an opencv image from the raw data in buffer above with the raw data
        }

        // get a colormap of the depth image
        applyColorMap(depthMat, colorDepthMat, COLORMAP_JET);

        // get infrared image
        IInfraredFrame* pInfraredFrame = nullptr;
        hResult = pInfraredReader->AcquireLatestFrame( &pInfraredFrame );
        if( SUCCEEDED( hResult ) ){
            unsigned int bufferSize = 0;
            unsigned short* buffer = nullptr;
            hResult/*IR*/ = pInfraredFrame->AccessUnderlyingBuffer( &bufferSize, &buffer );
            if( SUCCEEDED( hResult ) ){
                for( int y = 0; y < irHeight; y++ ){
                    for( int x = 0; x < irWidth; x++ ){
                        unsigned int index = y * irWidth + x;
                        infraredMat.at<unsigned char>( y, x ) = buffer[ index ] >> 8;
                    }
                }
            }
        }

        SafeRelease( pColorFrame );
        SafeRelease( pDepthFrame );
        SafeRelease( pInfraredFrame );

        cv::imshow( "Color", colorMat );
        // raw size
//        cv::imshow( "Color", colorBufferMat );
        cv::imshow( "Depth", depthMat );
        cv::imshow( "Color Depth", colorDepthMat );
        cv::imshow( "Infrared", infraredMat );

        int key = waitKey(1);    
        if( key == VK_ESCAPE ){
            break;
        }
    }

the question now is, is there any way to accurate measure the framerate that I get from kinect? I guess it does not have to do strictly related to kinect it could be any normal camera. Searching around I could something clear. Officially it says that kinect supports 30 fps so I would like to see if I get something close to that.

Update

#include <iostream>
#include <time.h>
#include <opencv2/opencv.hpp>

using namespace std;
using namespace cv;

int main()
{    
    VideoCapture cap1(0);
    VideoCapture cap2(1);

    // check if video is loaded successfuly
    if(!cap1.isOpened() || !cap2.isOpened())
    {
        cout << "Problem opening streams!" << endl;
        return -1;
    }

    Mat frame1, frame2;

    VideoWriter rec1("test.avi", -1/*CV_FOURCC('M','J','P','G')*/, 25, Size(640, 480), true); // setting cv_fourcc manually does not work in windows, also do not know why
    VideoWriter rec2("test2.avi", -1/*CV_FOURCC('D','I','V','X')*/, 25, Size(640, 480), true);

    if ( !rec1.isOpened() ) //if not ...
(more)
edit retag flag offensive close merge delete

Comments

Take a look at @pklab's answer here

LorenaGdL gravatar imageLorenaGdL ( 2015-12-09 10:35:48 -0500 )edit

@LorenaGdL thanks for the link. It seems to be what I wanted, how the heck I missed it from the seach. Thanks again ;-).

theodore gravatar imagetheodore ( 2015-12-10 02:27:36 -0500 )edit

Always welcome ;)

LorenaGdL gravatar imageLorenaGdL ( 2015-12-10 02:49:37 -0500 )edit

@LorenaGdL I am see that using the code you pointed out in the link above I get a constant frame rate. However, when I introduce a videowriter in order to save the images in a video the frame rate constantly drops I stopped it at 6 fps, I am sure that if I leave it will go further down. So, I was wondering if you have noticed any similar behavior because I think that it is a bug in videowriter() or something. I would expect to have a short range drop in the fps but not a continues one.

theodore gravatar imagetheodore ( 2015-12-10 07:56:19 -0500 )edit

I don't recall measuring times in code including videowriter. If you post some minimal code to reproduce the issue, I'll check later

LorenaGdL gravatar imageLorenaGdL ( 2015-12-10 08:15:37 -0500 )edit

ok try the code in the update section, for some reason in the beginning before recording Measure time decreases constantly leading to fps increment and after recording (press space for that) Measure time increases constantly leading to less fps. I do not know what is happening, most likely I miss something in the code but I am not sure. Can you have a look and tell me what happens to your computer.

theodore gravatar imagetheodore ( 2015-12-10 10:15:10 -0500 )edit

In the beggining there is a constant decrease of processing times, though I expected so: structures need to be initialized, and the computer is general need to start new processes, new threads and so on. After some few frames the fps is stable, in 30fps in my case. After started the recording, fps is still stable (slightly increased times, but still 30 fps). I let it run for a while, and never got less than 30 fps. So I don't know what's happening in your case. I tested with OpenCV 2.4 and 3.0 in Win7 x64, Visual Studio 2013. There is this issue that points to a similar behaviour, though for Python + Mac. Recommendation is to build with FFMPEG, I don't know if you're using it (I am)

LorenaGdL gravatar imageLorenaGdL ( 2015-12-10 10:59:29 -0500 )edit

@LorenaGdL how to build opencv with FFMPEG in windows because it also seems to be dependent on the codec that I am using. At the cmake phase the FFMEG flag was on before I start the compilation but later I noticed that I couldn't record anything since it couldn't find any codec. After that I installed some codecs from K-lite codecs mega pack and some seems to work some others not. I do not really know what is happening with the ffmpeg in windows, actually is there any guide that describes that for windows because from what I have seen the most guides for windows are missing some information. By the way thanks for your time.

theodore gravatar imagetheodore ( 2015-12-10 11:17:45 -0500 )edit

I actually facing exactly this problem described here

theodore gravatar imagetheodore ( 2015-12-10 11:20:56 -0500 )edit

I know very little about this kind of video issues. Maybe @pklab (sorry if I bother mentioning you) can help you more, I've seen several posts from him about related issues. I had K-lite codecs installed much before I even started using OpenCV, and Cmake always recognizes FFMPEG as installed in my Windows7 when building. So I don't think I can be of any help

LorenaGdL gravatar imageLorenaGdL ( 2015-12-10 11:29:04 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-12-11 13:10:51 -0500

pklab gravatar image

I'm sorry for long but done too fast answer... I hope it's useful.

Your code looks OK (even if should be cleaned), I think your issue is coming from the camera. More in details, maybe your camera has some automatic exposure adjustment. When low light condition AUTO_EXPOSURE increases shutter time or camera gain. Because, in any case your frame rate is max(wantedFps,1/shutter_time) if camera increases shutter time the fps might decrease.

You can easily test this case, pointing a light in front of your camera so it will become saturated. If I'm right your frame rate should increase.

Not all web cams has exposure control. You can test it using CV_CAP_PROP_AUTO_EXPOSURE and/or CV_CAP_PROP_EXPOSURE

It should be (my change from cams and driver)

  • to disable auto exposure: cap.set(CV_CAP_PROP_AUTO_EXPOSURE,-1);
  • to set wanted exposure: cap.set(CV_CAP_PROP_EXPOSURE,divider); where divider reduce in some way the max shutter time. Maybe 0 uses max shutter
  • to open camera driver interface: cap.set(CV_CAP_PROP_SETTINGS, -1); this will pop up a window from driver

Remember that using VideoCapture cap1(CV_CAP_DSHOW + device); you will can the DirectShow driver. DSHOW gives you more control on capture controls param. (Some CV_CAP_PROPs doesn't work with default VFW driver).

At the end keep in mind that the calling cap >> frame you will wait for an available frame. You wont to be surprised if loop time for

while(1){
    cap >> frame;
    if (waitKey(5) >= 0)
        break;
}

has same duration of simpler

while(1){
    cap >> frame;
}

This is because 5ms is short time, in the mean time the camera is preparing the frame so you can spend time to do something (as is in 1st case) instead of waiting for the camera (as in in 2nd case). Exactly you can do something up to 1/fps seconds without loosing !

If you spend more time to do things the readTime will reduce because the image is always ready. This doesn't mean that cam is grabbing faster but just you are loosing frames

here is my function to measure loopTime and readTime. In my case they are both fixed at 30fps

int MeasureFps(int device = CV_CAP_DSHOW + 0)
{
    Mat frame;
    cout << "Opening device " << to_string(device) <<"..."<< endl;
    VideoCapture cap(device);
    if (!cap.isOpened())
    {
        cout << "Problem opening streams!" << endl;
        return -1;
    }
    int maxFrames = 100;
    cout << "Wait while grabbing " << to_string(maxFrames) << "frames..." << endl;

    double capprop;
    capprop = cap.get(CV_CAP_PROP_AUTO_EXPOSURE);
    capprop = cap.get(CV_CAP_PROP_EXPOSURE);
    cap.set(CV_CAP_PROP_AUTO_EXPOSURE,-1); //disable. doesn't works for all cams
    cap.set(CV_CAP_PROP_EXPOSURE,0);       //default. doesn't works for all cams
    cap.set(CV_CAP_PROP_FRAME_WIDTH, 640);
    cap.set(CV_CAP_PROP_FRAME_HEIGHT, 480);
    cap.set(CV_CAP_PROP_FPS, 300000.0);   //very high to catch the maximum
    cap.set(CV_CAP_PROP_SETTINGS, -1);    //pop up the driver window 
    cap >> frame; //grab 1st frame to initialize all chain
    int64 t0, t1, r0, readTicks;
    int tickFreq = cv::getTickFrequency();
    int numberOfFrames = 0;
    readTicks = 0;
    t0 = cv::getTickCount();
    while (numberOfFrames<maxFrames)
    {
        r0 = cv::getTickCount();
        cap >> frame;
        readTicks += (cv::getTickCount() - r0);
        numberOfFrames++;
        /*
        if (frame.empty()) continue;
        imshow("frame ...
(more)
edit flag offensive delete link more

Comments

@pklab I am really grateful to your answer. I will try as soon as I will find some time and I will try to come back. Again many thanks :-).

theodore gravatar imagetheodore ( 2015-12-12 14:12:13 -0500 )edit

you are welcome .. btw you can do an easy test. Run your sw, arrange your room darker, point a flash light in front of your camera. Do FPS goes up ? if yes than degradation of FPS is the outcome of some integration or light auto adjustment done by camera itself.

pklab gravatar imagepklab ( 2015-12-14 04:07:50 -0500 )edit

@theodore did you try the test with light ? I'm curious to know if really some cams/drivers performs auto adjustment so long to delay grabbing FPS.

pklab gravatar imagepklab ( 2015-12-22 04:42:08 -0500 )edit

@pklab not really yet, I was struggling with other stuff (kinect, libfreenect2, recording, finally jump to multithread, etc...) which this question actually was derived from, since I was trying different things and I did not have time. I will try it though as soon as I find some ;-) since it interests me as well.

theodore gravatar imagetheodore ( 2015-12-22 05:01:03 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2015-12-09 10:25:36 -0500

Seen: 1,617 times

Last updated: Dec 11 '15