Tackle low FPS for correlation code to compute shift in Image

asked 2018-10-11 21:40:59 -0600

Anurag gravatar image

I am trying to track an object using correlation. I am finding a smaller patch in a larger image, frame by frame. For this, I am finding the shift in the patch, and where correlation is maximum, update the patch with a new patch.

My code is:

cv::Mat im_float_2,imagePart_out;
cv::Mat im_floatBig;
cv::Scalar im1_Mean, im1_Std, im2_Mean, im2_Std;

double covar, correl;
int n_pixels;

void computeShift()
    int maxRow=0, maxCol=0, TX, TY;
    double GMAX=0;
    Mat image_window = Mat::zeros(imagePart.rows, imagePart.cols, CV_32F);

    imagePart.convertTo(im_float_2, CV_32F);

    for(maxRow=0; maxRow<=imageBig.rows-image_window.rows;maxRow++)
        for(maxCol=0; maxCol<imageBig.cols-image_window.cols;maxCol++)

            image_window = im_floatBig( cv::Rect( maxCol, maxRow, 
            image_window.cols, image_window.rows ) );

            n_pixels = image_window.rows * image_window.cols;

            // Compute mean and standard deviation of both images

            meanStdDev(image_window, im1_Mean, im1_Std);
            meanStdDev(im_float_2, im2_Mean, im2_Std);

            // Compute covariance and correlation coefficient
            covar = (image_window - im1_Mean).dot(im_float_2 - im2_Mean) / n_pixels;

            correl = covar / (im1_Std[0] * im2_Std[0]);
            if (correl > GMAX)
            GMAX = correl; TX = maxRow; TY=maxCol;
            image_window.convertTo(imagePart, CV_8UC1);

            cvtColor(imagePart, imagePart_out, CV_GRAY2BGR);
            printf("\nComputed shift: [%d, %d] MAX: %f\n", TX, TY,GMAX);                


But when executing this I am getting very low FPS(1-2) even for small video size (Frame size-262x240, Patch size- 25x25).

Is there any way to achieve higher FPS. I am also looking in the direction of phase correlation, but not sure how to go about it from here. Can converting it to frequency domain will help?

For now, I want to optimize the above code for speed.

edit retag flag offensive close merge delete


why not use a ready-made tracker, like MOSSE ?

what you're doing there seems more like a full-scale template matching, not like tracking (where you would exploit a previously known position)

berak gravatar imageberak ( 2018-10-12 00:29:49 -0600 )edit

hmm, also: if (correl > GMAX) , -- shouldn't it be if (correl < GMAX) ?

(imho, minimum dot product is the closest)

berak gravatar imageberak ( 2018-10-13 01:54:25 -0600 )edit

I have to import this to Vivado HLS so I am trying to use as less as inbuilt functions. And if (correl > GMAX) is working just fine because in correlation when images are similar we get maximum value.

Anurag gravatar imageAnurag ( 2018-10-15 22:59:56 -0600 )edit