HesNotTheStig's profile - activity

I'm just a tinkerer. I was simply playing with the algorithm and trying to see if I could use it for video. I can't attest to what performance improvement we would see from this. Before it makes it into the code, it should be independently confirmed that setting the border to zero makes no difference for key points.

In my tinkering process, I was also trying to see the cv::resize function could be refactored to reused offset buffers between frames of identical dimensions. Not seeing a performance improvement. No idea why.

On the other hand, my use a a streamlined background subtraction model means I can focus on moving elements in the video and improve descriptor quality without sacrificing much in performance.

I've been working on this. I don't know if the reflected border gets used at all in the descriptor's algorithm, but if you're only looking to produce key points (perhaps to feed into another algorithm's compute method), then it is entirely unnecessary to reflect this border.

I confirmed this by tweaking the code to reduce the border to zero and then comparing the keypoints generated. No difference. Perhaps an optimization could be put into place for this.

So I'm working on an implementation of OpenCV's ORB keypoint algorithm (not the descriptors, just keypoints). My implmentation produces key points VERY close to the original, but something's a bit off and I'm trying to find the problem. One thing that always puzzled me about the OpenCV implementation of the ORB algorithm is that as it is pre-computing the image pyramid, it feels to need to use the "copyMakeBorder" function to relect a mirror image of the border of the image around each level of the pyramid.

// Pre-compute the scale pyramids
for (level = 0; level < nLevels; ++level)
{
    Rect linfo = layerInfo[level];
    Size sz(linfo.width, linfo.height);
    Size wholeSize(sz.width + border*2, sz.height + border*2);
    Rect wholeLinfo = Rect(linfo.x - border, linfo.y - border, wholeSize.width, wholeSize.height);
    Mat extImg = imagePyramid(wholeLinfo), extMask;
    Mat currImg = extImg(Rect(border, border, sz.width, sz.height)), currMask;

    if( !mask.empty() )
    {
        extMask = maskPyramid(wholeLinfo);
        currMask = extMask(Rect(border, border, sz.width, sz.height));
    }

    // Compute the resized image
    if( level != firstLevel )
    {
        resize(prevImg, currImg, sz, 0, 0, INTER_LINEAR);
        if( !mask.empty() )
        {
            resize(prevMask, currMask, sz, 0, 0, INTER_LINEAR);
            if( level > firstLevel )
                threshold(currMask, currMask, 254, 0, THRESH_TOZERO);
        }

        copyMakeBorder(currImg, extImg, border, border, border, border,
                       BORDER_REFLECT_101+BORDER_ISOLATED);
        if (!mask.empty())
            copyMakeBorder(currMask, extMask, border, border, border, border,
                           BORDER_CONSTANT+BORDER_ISOLATED);
    }
    else
    {
        copyMakeBorder(image, extImg, border, border, border, border,
                       BORDER_REFLECT_101);
        if( !mask.empty() )
            copyMakeBorder(mask, extMask, border, border, border, border,
                           BORDER_CONSTANT+BORDER_ISOLATED);
    }
    prevImg = currImg;
    prevMask = currMask;
}

Does this reflected border serve any purpose in terms of generating key points? I've studied the key point section carefully and as far as I can tell, it never gets used. Can I skip that part and save some CPU cycles?

(P.S.: If you're wondering, my implementation is going to be optimized for use on video key frames. In other words, I can calculate most of the buffers once per video since the resolution between frames remains the same. I can also play with background subtraction to restrict the mask to focus only on moving objects.)

So I'm using ORB keypoints and feature descriptors in a video comparison algorithm. It works pretty well. However, I wonder if it is necessary to scale each key-frame to 1920x1080 before I feed it into the keypoint and feature descriptor methods. Sometimes tells me that accounts for a considerable chunk of the signature cration time. Am I helping the algorithm or wasting time and hurting the accuracy be scaling all frames to be the same resolution?

I understand that I can't expect cv::VideoCapture to always return pristine frames if the video is corrupted, but I have a video comparison algorithm that DEMANDS that videos track time from beginning to end.

When I'm calculating a signature for a video, Determine who many frames are between each key frame by taking the total number of frames listed for the video and dividing by the FPS to derive who many frames separate one second. Unfortunately, if cv::VideoCapture finds a broken frame, it seems to decide to skip to the next frame rather than just returning that broken frame. I'm find with the frame being broken. My algorithm can handle that. So long as most of the frames are intact, I'm fine.

Unfortunately, if the video contains a patch of broken frames early on, then cv::VideoCapture skips ahead in time and introduces an undeseriable time shift in the video signature. When it comes time to compare this signature to another signature, the frames don't line up and I get a false negative.

Is there some way to make cv::VideoCapture.readFrame(Mat) return Mat::zeros when it encounters a damaged frame rather than skip ahead? I know it isn't doing this because I can have two videos with identical content and an identical listed run time turn up vastly different numbers of key frames.

When I open up a Video Capture of a few of my .wmv videos, the fps will come out as "-nan" (not a number). I'm presuming this means that this video uses a variable frame rate.

I'm trying to calculate video signatures for these videos and to do that, I need to know how many frames elapse per second. When the FPS is variable, how do I keep track of passage of time as I continue to call VideoCapture.read(Mat)?