VideoCapture returns empty frames on Windows after a while

edit

I have a program which loads a video and does all sorts of stuff, irrelevant to this question. I use Ubuntu Linux, where it works without issue. Now I need it to run on Windows. When I compile and run it, at first it works fine. But at some point, VideoCapture starts returning empty frames. It differs from video to video exactly when it happens, usually around 13000-15000 frames into the video. The stopping point is the same every time I run the program on the same video, so at least it is deterministic.

I have created a very simple program in which the problem also happens:

#include <iostream>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

using namespace std;
using namespace cv;

int main()
{
    bool loopFlag = true;
    int frameCounter = 0;


    VideoCapture cap("E:\\workFolder\\ed_1024.avi");
    if(!cap.isOpened()) {
        cout << "Could not open video.\n";
        return -1;
    }

    namedWindow("Capture", 1);
    Mat frame;

    while(loopFlag)
    {
        cap >> frame;
        if(frame.empty()) {
            cout << "Frame " << frameCounter << " is empty!";
            loopFlag = false;
            break;
        }
        imshow("Capture", frame);
        cout << "Showed frame " << frameCounter << endl;
        frameCounter++;
    }
    waitKey(0);
    return 0;
}

That is as barebones as I can do. When I run it with some of the videos I have captured, as mentioned it stops after several thousand frames. To verify that it is not my videos that are the problem, I downloaded the Open Source animation movie Elephants Dream in DivX (my own videos are Xvid). When I run the program with this, it stops after 74 frames:

Showed frame 0
Showed frame 1
[...]
Showed frame 73
Frame 74 is empty!

I'm considering if it is somehow a codec problem, but it does read the first frames just fine. I have Xvid and FFDShow installed on the Windows machine.

Any ideas? I hardly know where to begin the debugging.