OpenCV 3.0.0 - process new frame callback

edit

cv::VideoCapture::read returns the frame as soon as it's available. With a simple loop grab+process you'll achieve exactly what you are looking for. Short hints:

• If your processing time is always faster than 1/FPS than a simple grab+process loop works fine without threads too.
• If sometime your processing time is slower than 1/FPS than you can use a grabbing thread, a processing thread and a shared queue (check here for an example)
• If your processing time is always slower than 1/FPS than you have to reduce FPS or improve processing performance.

Details:

On windows, cv::VideoCapture uses internal callback with DSHOW, VFW and Microsoft Media Foundation. This means that cv::VideoCapture::read waits for the next frame as provided camera FPS. It waits (in idle state) for a signal from the callback function. Check the code for DSHOW

Duration of cv::VideoCapture::read can change from close to zero (if a frame is ready) up to 1/FPS (if a frame has just been grabbed).

Results from the code below shows that loop time (grab+process) will not change if processing time < 1/FPS... when processing time increases, grabbing time decreases because grabbing will waits less time for a new frame:

Setting wanted fps to 25.0
Measuring real fps...: 19.2fps
    Max processing time: 1/19.2 = 52.1ms

Start processing time tests...
Processing: 0.0ms   Grabbing: 46.8ms    Total: 46.8ms
Processing: 10.0ms  Grabbing: 35.4ms    Total: 45.4ms
Processing: 20.0ms  Grabbing: 26.8ms    Total: 46.8ms
Processing: 30.0ms  Grabbing: 16.8ms    Total: 46.8ms
Processing: 40.0ms  Grabbing: 6.6ms     Total: 46.6ms
Processing: 50.0ms  Grabbing: 4.1ms     Total: 54.1ms

when processing > 1/fps :

Processing: 60.0ms  Grabbing: 3.6ms Total: 63.6ms  <- we are loosing frames

The test code:

#define SimulateProcessing(_processingTimeMs)  \
    int64 _start= getTickCount();  \
    int64 _end = _start + _processingTimeMs * getTickFrequency() / 1000.;  \
    while (getTickCount() <= _end) {double d = 2*CV_PI;} 

int Main_GrabbingMeasureTimes()
{
    int camId = 0;                //the default camera
    //int driver = CV_CAP_VFW;    //use VFW driver
    //int driver = CV_CAP_MSMF;   //use MSMF driver
    int driver = CV_CAP_DSHOW;    //use DSHOW
    double fps = 25.0;            //wanted fps
    cv::Mat frame;
    int64 grabStart, grabSum, procStart, procSum;
    int cnt, maxCnt = 50;
    double procTestTime,procTestTimeMax;
    double procTime, grabTime;
    std::clog.precision(1);
    clog.setf(std::ios::fixed, std::ios::floatfield);

    //OPEN THE CAMERA
    VideoCapture cap(camId + driver);
    if (!cap.isOpened()) return -1;
    cap >> frame;                    //grab to prepare memory

    //--------------
    //MEASURE REAL FPS
    clog << endl << "Setting wanted fps to " << fps;
    cap.set(CV_CAP_PROP_FPS, fps);   //set grabbing fps
    fps = cap.get(CV_CAP_PROP_FPS);  //read fps from cam
    clog << endl << "Measuring real fps...";
    for (cnt = 0,grabSum = 0; cnt < maxCnt; cnt++) {
        grabStart = getTickCount();
        cap >> frame;
        grabSum += (getTickCount() - grabStart);
    }
    procTestTimeMax = 1000.0 * grabSum / getTickFrequency() / cnt;
    clog << ": " << 1000. / procTestTimeMax << "fps" << endl
        << "\tMax processing time: " << procTestTimeMax << "ms" << endl;

    //--------------------------------------------
    //MEASURE LOOP DURATION VS PROCESSING DURATION
    clog << "\nStart processing time tests...\n";
    //same loop with increasing processing time
    for (procTestTime = 0; procTestTime < procTestTimeMax + 10; procTestTime +=10) {
        //N frames for average
        procSum  = 0;
        for (cnt ...