OpenCV4android template matching

asked 2016-11-19 10:16:38 -0500

Ftahir192 gravatar image


I'm using OpenCV4Android to try out template matching on an android phone. Essentially what I'm doing is taking a picture of some object that I wish to detect, cropping it and saving it as a template.

I then use my android camera and a surface view to constantly get images from the camera. I am then applying template matching on each image, converting the image to a Mat first. However, when applying template matching I only get around 3-4 fps.

What I am essentially doing is this:

** mCameraMat = inputFrame.rgba();

            int matchMethod = Imgproc.TM_CCOEFF_NORMED;

            // mTemplateMat resized in terms of video size in prepareMediaRecorder.
            // Very hacky solution so need to fix it!
            int result_cols = mCameraMat.cols() - mTemplateMat.cols() + 1;
            int result_rows = mCameraMat.rows() - mTemplateMat.rows() + 1;

            mResult = new Mat(result_rows, result_cols, CvType.CV_32F);

            // Move this to a new thread.
            Imgproc.matchTemplate(mCameraMat, mTemplateMat, mResult, matchMethod);
            Core.normalize(mResult, mResult, 0, 1, Core.NORM_MINMAX, -1, new Mat());

            // Localizing the best match with minMaxLoc
            MinMaxLocResult mmr = Core.minMaxLoc(mResult);

            Point matchLoc;
            if (matchMethod == Imgproc.TM_SQDIFF || matchMethod == Imgproc.TM_SQDIFF_NORMED) {
                matchLoc = mmr.minLoc;
            } else {
                matchLoc = mmr.maxLoc;

            // Draw a boundary around the detected object.
            Imgproc.rectangle(mCameraMat, matchLoc, new Point(matchLoc.x + mTemplateMat.cols(),
                matchLoc.y + mTemplateMat.rows()), new Scalar(TrackingActivity.r, TrackingActivity.g,
                TrackingActivity.b, TrackingActivity.a), 2);


Where mTemplateMat is the template bitmap image converted into a Mat object.

The bottleneck is on the line

Imgproc.matchTemplate(mCameraMat, mTemplateMat, mResult, matchMethod);

If I remove that line, I get around 25 fps, which is much more acceptable. I'd be fine with anything above 13-14. I understand that template matching is a very expensive process and doing it every frame can be costly. I have tried to do it every 20 frames, but it still slows down the processing considerably, and the end video looks worse as there is a constant transition from a smooth fps display to a low fps display.

What are my options in optimising matchTemplate? Any tips are much appreciated.

edit retag flag offensive close merge delete



Have you try to reduce template size (and image)?

LBerger gravatar imageLBerger ( 2016-11-19 10:58:32 -0500 )edit

Wouldn't this affect the end result if I decrease the source image size? Furthermore, I would have to resize it for each image that comes in, which could be 20+ resizes a second. Would that not be costly also? The size of the source images are 1280x720 currently.

I am using the template size as an indicator to what size the rectangle bounding box would be, and if I decrease the template size further, I'm not sure how I could indicate what size the bounding box (to indicate detected object) should be drawn at.

Ftahir192 gravatar imageFtahir192 ( 2016-11-19 11:07:19 -0500 )edit

Of course it would affect result but you can try. You can try lucas kanade tracking with you object

LBerger gravatar imageLBerger ( 2016-11-19 11:24:05 -0500 )edit

If I go with downscaling, what image and template size is recommended? As stated currently, the image size is 1280x720, and template is normally no more than half of that resolution. Thanks for the help.

Ftahir192 gravatar imageFtahir192 ( 2016-11-19 12:16:10 -0500 )edit

try rescaling to 1280/2 X 720/2

LBerger gravatar imageLBerger ( 2016-11-19 12:32:23 -0500 )edit

Sounds good, if I also scale the template matching, I'd be doing only 1/4 of the evaluations as opposed to non-scaling, which is a massive difference in and of itself.

Regarding downscaling, would using the resize() function built into OpenCV be considered a fast-scaling method, or can I do better?

Ftahir192 gravatar imageFtahir192 ( 2016-11-19 12:35:30 -0500 )edit

I don(t know. resize with nearest neighbour should be faster. But you should try copy to usng parallelloopbody.

About matchtemplate may be it is a good method but in your case may be you should do it yourself :

using matchtemplate template fft is calculate each frame. It is not usefull you already done it...

LBerger gravatar imageLBerger ( 2016-11-19 12:47:34 -0500 )edit

Hmm, I've been advised on SO to stick to matchTemplate as it is already well optimised.

Ftahir192 gravatar imageFtahir192 ( 2016-11-19 12:48:59 -0500 )edit

It is well optimised for one call not for two may be!

LBerger gravatar imageLBerger ( 2016-11-19 12:53:29 -0500 )edit

Ah I see, I'm calling it for every frame! I'll look into this, I'll see firstly if downscaling helps. Thanks a lot for your help!

Ftahir192 gravatar imageFtahir192 ( 2016-11-19 12:55:13 -0500 )edit