Ask Your Question
1

VideoCapture::set (CAP_PROP_POS_FRAMES, frameNumber) not exact in opencv 3.2 with ffmpeg?

asked 2017-06-29 05:19:58 -0600

nji gravatar image

updated 2017-06-30 11:29:20 -0600

For an application (Win7/ mingw) that analyses movie files I divide the reading and processing to the cores. When trying to jump to the beginning of the parts with

VideoCapture::set (CAP_PROP_POS_FRAMES, frameNumber);

and then

capt >> frame;

I do not get frame with number frameNumber but some frames before (the exact value depends on the movie format).

Although

VideoCapture::set (CAP_PROP_POS_FRAMES, frameNumber);

and then

ret = VideoCapture::get (CAP_PROP_POS_FRAMES);

lets frameNumber == ret

[Edit: This makes it even worse. As if you want to take into account that you don't get what you want always: For example you want to seek close to the end of the stream (say 10 elements before) and do want to take into account that you maybe get frame k-5 if you request frame k, and by this request the last 15 consecutive frames, then get() will return 0 for the last 5 as "it" thinks it is at the end already!]

There are many threads about this issue already. Some say that in newer versions of opencv that "bug" is gone.

Other say that it's about the keyframes in the video stream that would make ffmpeg seek to the first keyframe AFTER the desired position.

Has somebody up-to-date information what happens here and how to do (this quite common) task please?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2017-06-29 13:00:29 -0600

LBerger gravatar image

updated 2017-06-30 02:20:21 -0600

Windows 10 VS 2015 opencv-3.2.0-dev opencv_ffmpeg320_64.dll OpenH264 Video Codec provided by Cisco Systems, Inc.

With this program I haven't got any some problems (CAP_PROP_FOURCC: avc1)

vector<int> pos={1*50, 2 * 50,3 * 50,5 * 50,7 * 50,11 * 50,13 * 50,17 * 50,19 * 50,23 * 50,29 * 50,31 * 50,37 * 50,41 *50,43 * 50 };
vector<Mat> frameSelec;
VideoCapture video("b.mp4");
if (!video.isOpened())
{
    cout << "Video introuvable";
    return 0;
}
Mat frameVideo;
int fourcc=video.get(CAP_PROP_FOURCC);
string fourcc_str = format("%c%c%c%c", fourcc & 255, (fourcc >> 8) & 255, (fourcc >> 16) & 255, (fourcc >> 24) & 255);
cout << "CAP_PROP_FOURCC: " << fourcc_str << endl;
for (int j = 0, i = 0; j < pos.size(); i++)
{
    video >> frameVideo;
    if (i == pos[j])
    {
        frameSelec.push_back(frameVideo.clone());
        j++;
    }
}
video.set(CAP_PROP_POS_FRAMES, 0);
double minV[1],maxV[1];
for (int j = 0; j < pos.size(); j++)
{
    video.set(CAP_PROP_POS_FRAMES, pos[j]-3);
    for (int kk = 0; kk < 6; kk++)
    {
        video >> frameVideo;
        Mat d;
        imshow("flu", frameVideo);
        imshow("fpo", frameSelec[j]);
        subtract(frameVideo,frameSelec[j],d,noArray(),CV_16S);
        d = d.reshape(1);
        minMaxLoc(d,minV,maxV);
        if (abs(minV[0]) + abs(maxV[0]) == 0)
        {
            cout<<"Min for frame "<< pos[j] - 3+kk<<"expected "<<pos[j]<<" \t"<<minV[0]<<"\t"<<maxV[0]<<"\t";
            cout<<"\n";
        }
        if (kk == 3)
        {
            cout << "error for frame " << pos[j] - 3 + kk <<  " \t" << minV[0] << "\t" << maxV[0] << "\t";
               cout << "\n";

        }

        d=abs(d);
        d.convertTo(d,CV_8U);
        d = d.reshape(3, frameVideo.rows);
        imshow("diff", d);
    }
        waitKey(10);
}

Results for video : https://www.youtube.com/watch?v=tnWP2... are

CAP_PROP_FOURCC: avc1
Min for frame 50expected 50     0       0
error for frame 50      0       0
Min for frame 100expected 100   0       0
error for frame 100     0       0
Min for frame 149expected 150   0       0
error for frame 150     -166    198
Min for frame 249expected 250   0       0
error for frame 250     -162    144
Min for frame 349expected 350   0       0
error for frame 350     -225    251
Min for frame 550expected 550   0       0
error for frame 550     0       0
Min for frame 649expected 650   0       0
error for frame 650     -204    217
Min for frame 850expected 850   0       0
error for frame 850     0       0
Min for frame 950expected 950   0       0
error for frame 950     0       0
Min for frame 1149expected 1150         0       0
error for frame 1150    -213    204
Min for frame 1449expected 1450         0       0
error for frame 1450    -166    148
error for frame 1550    -217    194
Min for frame 1551expected 1550         0       0
Min for frame 1849expected 1850         0       0
error for frame 1850    -215    221
Min for frame 2050expected 2050         0       0
error for frame 2050    0       0
Min for frame 2150expected 2150         0       0
error for frame 2150    0       0
edit flag offensive delete link more

Comments

No problem with vector<int> pos={1*200, 2 * 200,3 * 200,5 * 200,7 * 200,11 * 200,13 * 200,17 * 200,19 * 200,23 * 200,29 * 200,31 * 200,37 * 200,41 * 200,43 * 200 };

LBerger gravatar imageLBerger ( 2017-06-29 15:07:47 -0600 )edit

Sorry, I did a (small) mistake. When I take your new init then I get

CAP_PROP_FOURCC: avc1
-160    114
-132    149
-207    202
-181    176
-178    175
-203    248
-156    164
-126    160
-241    221
-208    239
-202    208
-196    166
-182    117
-149    147
-128    138

You get all 0s? (That's what I get with the original init positions) Maybe it's because the different compilers and me using the 32bit ffmpeg? (BTW ffmpeg with codec included)

nji gravatar imagenji ( 2017-06-29 15:28:05 -0600 )edit

alls 0. Can you try to download http://ciscobinary.openh264.org/openh... and install dll where is your exe. Finally I think it is only to encode video

LBerger gravatar imageLBerger ( 2017-06-29 15:43:53 -0600 )edit

So you use the codecs from within ffmpeg, too? But as 64bit. Would you mind trying a second video file?

nji gravatar imagenji ( 2017-06-29 15:59:17 -0600 )edit

try with this video perso.univ-lemans.fr/~berger/Afsd56/Identification.avi but instead of 200 use 50 (and fourCC is h264)

LBerger gravatar imageLBerger ( 2017-06-29 16:08:59 -0600 )edit

All 0 for me too then! This means it depends also on the video. And then "Reading / writing properties involves many layers. Some unexpected result might happens along this chain. Effective behaviour depends from device hardware, driver and API Backend." Seems I've to insert some "intelligent" fuzzy logic.

nji gravatar imagenji ( 2017-06-29 16:20:58 -0600 )edit

can you upload a video with problem?

LBerger gravatar imageLBerger ( 2017-06-30 00:33:18 -0600 )edit

Take this (MP4 Medium 10,7 MB). With "pos-factor 50" it gives

CAP_PROP_FOURCC: avc1 0 0 0 0 -166 198 -162 144 -234 249 0 0 -204 217 0 0 -235 221 -213 204 -166 148 -217 194 -215 221 0 0 0 0

nji gravatar imagenji ( 2017-06-30 01:38:47 -0600 )edit

post an issue and give this post as reference

LBerger gravatar imageLBerger ( 2017-06-30 02:19:32 -0600 )edit

Yes all results are in my answer. I think you can close this post ( and post an issue of course)

LBerger gravatar imageLBerger ( 2017-06-30 02:38:03 -0600 )edit
1

Excuse for the confusion. I didn't notice your change in the answer. Still ... I'm a little in doubt if it makes any sense to post an issue, as I think it is quite likely that it is not a bug in opencv but an "effect" of the "layer chain" (see quote from the doc above). The maintainers might just point to that. What do you think? Maybe we (I) should just close this thread with an answer that one should NOT rely on seeking to be always exact?

nji gravatar imagenji ( 2017-06-30 02:59:19 -0600 )edit

I think you should post an issue. Everything is ready you have an example and everybody could reproduce results. Mainteners will give an answer (RFC, BUG...)

LBerger gravatar imageLBerger ( 2017-06-30 06:45:50 -0600 )edit

Done. Maybe I may draw your attention to the question where this problem arose from, and where I didn't get any satifying solution for)

nji gravatar imagenji ( 2017-06-30 07:57:32 -0600 )edit

@LBerger: The problem seems to be spreaded (as there are many posts in the forums), but nobody seems to care (see my posted issue). They say opencv is about image processing and reading in is utility/ peripheral. (Like providing a car but with no functional doors). So if (!) one detects that this bug exists one has to do some kind of work-around for correct positioning. For the problem of not getting the last frames (if mispositioned too early) I don't have a solution so far at all. Very sad all this.

nji gravatar imagenji ( 2017-07-19 03:50:33 -0600 )edit

Is it really an opencv or a ffmpeg problem? may be you can use this method to check it

LBerger gravatar imageLBerger ( 2017-07-19 04:10:23 -0600 )edit

I can't tell for sure. But as there are many posts in the different forums concerning getting wrong frames using opencv, for me seems to be a problem of collaboration (See my quote of the opencv docs 10 comments above). And nobody takes responsibilty. To me it looks as if opencv uses ffmpeg in a somekind wrong way. As ffmpeg is the base I think an error there quite unlikely. But I'm no expert at all.

nji gravatar imagenji ( 2017-07-19 04:55:55 -0600 )edit

I tried ffmpeg command line with the trim parameter but didn't get useful results...

nji gravatar imagenji ( 2017-07-19 05:38:39 -0600 )edit
1

(Sorry for the late addendum - but the server seems to be sleepy so I couldn' add to the comment above). But maybe the trim parameter (with start_frame/ end_frame for exact frame extraction) doesn't quick JUMP to the position (as opencv set () does), but walks from beginning?

nji gravatar imagenji ( 2017-07-19 05:54:17 -0600 )edit

Forum is too slow today. I have modified source code https://gist.github.com/LaurentBerger... and using this command ffmpeg -i video.webm thumb%04d.png -hide_banner

results are nothing constant. If you want to talk may be it's better on irc #opencv

LBerger gravatar imageLBerger ( 2017-07-19 07:49:47 -0600 )edit

Excuse me ... I've to ask: "nothing constant"? Do you mean that all compared frames are different? If that is the case then I think that the results of command ffmpeg are the correct ones, as they are "streamed", which seems to produce correct extractions. ... I think the question is which ffmpeg interface/ parameters opencv uses. Maybe the parameter is timestamp, not framenumber? Where are the opencv developers? ;-)

nji gravatar imagenji ( 2017-07-19 08:27:09 -0600 )edit

Can you try same program on linux ? can you disable ffmpeg when you build opencv ? ffmpeg developers https://github.com/opencv/opencv/blob...

LBerger gravatar imageLBerger ( 2017-07-19 08:38:08 -0600 )edit

I would like to point to. Solved. Or not solved? What is here the question?

nji gravatar imagenji ( 2017-07-22 02:39:14 -0600 )edit

I would say no.

LBerger gravatar imageLBerger ( 2017-07-22 03:43:10 -0600 )edit

Some more information. There is a (4 y/o) post at stackoverflow about it. And then ... if you have a look at opencv code (CvCapture_FFMPEG::seek(int64_t) ... you find some kind of "index acrobatics" in there. To me (being someone who just needs to USE it) it looks as if ffmpeg delivers wrong frames sometimes because of decoding issues (keyframe etc.). If that is the case it's the question who should take the responsibility: ffmpeg (to always deliver what is wanted) - or the customer (opencv) trying to correct the failure. Doing the correction on app layer (I did that) is a pretty index mess. Anybody out there?

nji gravatar imagenji ( 2017-07-23 03:00:07 -0600 )edit

And one more info: Using the directshow interface I never noticed (or heard about) getting wrong frames with IMediaSeeking::SetPositions. (For files directshow can't be used from opencv as this option is not implemented ...)

nji gravatar imagenji ( 2017-07-24 11:19:39 -0600 )edit

Question Tools

1 follower

Stats

Asked: 2017-06-29 05:19:58 -0600

Seen: 9,208 times

Last updated: Jun 30 '17