OpenCV move to begin frame position fail

edit

I am trying to use OpenCV VideoCapture to read some frames for training. After training, I would like to return to the beginning of the video and do processing. The problem is that OpenCV VideoCapture set(CV_CAP_PROP_POS_FRAMES, 0) cannot return to the beginning of the video.

VideoCapture cap("video1.mp4");
if( !cap.isOpened()){
    cout << "Cannot open the video file" << endl;
    return -1;
}

// read some frames here

int count = cap.get(CV_CAP_PROP_FRAME_COUNT); //get total frame count
cap.set(CV_CAP_PROP_POS_FRAMES, 0); //Set index to 0 (start frame)

int index = 1;
while(1)
{
   Mat frame;
   bool success = cap.read(frame);
   if (!success){
     cout << "Cannot read  frame " << endl;
     break;
   }
   cout << "the current frame: " << index << endl;
   index++;
}

In the program, the final index value would not be same as frame count. Say a sample running would be:

index = 3774 and count = 3786
index = 3764 and count = 3776

I also try to set frame index using CV_CAP_PROP_POS_MSEC (according to a post). But it didn't work.

My current solution is to reconstruct a VideoCapture and read from begin to end. Can anyone explain why set can even not return to the beginning of the video? I think it go nothing to do with the decompression algorithm.