why does cv::VideoCapture >> cv::Mat take so long time?

edit

On one single denver core of TX2 board, this function itself takes about 10ms. What's the process of VideoCapture>>Mat, why does it take that long time? I know with more cores, it run faster, but now for some reason, this line of code can only run on single core. (denver core is already the fastest core on TX2)

Is there an alternative that is faster than this line but same effect.

int main(int argc, char** argv) { cpu_set_t cpuset; int cpu = 1; CPU_ZERO(&cpuset);
CPU_SET( cpu , &cpuset); sched_setaffinity(0, sizeof(cpuset), &cpuset);

cv::VideoCapture cap;

cap.open("a video file");
cap.set(CV_CAP_PROP_CONVERT_RGB, true);
long long total_num = cap.get(CV_CAP_PROP_FRAME_COUNT);
float fps = cap.get(CV_CAP_PROP_FPS);

cv::Mat ori_frame;

for(int i=0; i<total_num; i++){ 

    float ms;
    auto t_begin = std::chrono::high_resolution_clock::now();

    cap >> ori_frame;

    auto t_end = std::chrono::high_resolution_clock::now();
    ms = std::chrono::duration<float, std::milli>(t_end - t_begin).count();
    printf("Main function: cap time************************************ %f ms\n", ms);

}
return 0;

}