Ask Your Question
0

OpenCV multi-threading frames on Android

asked 2016-08-06 19:01:03 -0600

carlos123alberto gravatar image

I am doing some work with OpenCV on Android. My code was all on the Java interface and its performance was very bad, so I implemented almost all the image processing with JNI. So on Java I have:

@Override
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame)
{
    _imgBgrTemp = inputFrame.rgba();
    jniFrame(_imgBgrTemp.nativeObj);
    return _imgBgrTemp;
}

So jniFrame() function takes care of all the image processing in C++, and with this there was a good improvement in performance. But it is still around 8fps and my camera can do 30fps without any processing on the image.

Looking more close I saw that even while processing the code, it uses only 25% CPU of my Android, witch is a Zenfone 2 with a quad-core processor.

I am thinking about having it running in 4 threads so I would have o FIFO pool to receive frames, process, and display it in the received order.

I am thinking in use this: Creating a Manager for Multiple Threads

So my questions are:

I am going the right way ?

Do you have any tips ?

What should I consider (As I am working with JNI) ?

I didn't post the jniFrame() here because I don't think it is very relevant as it is a Work in progress code, very big. But it is about recognizing a rubik cube and getting its colors. if you also can give me any tips on that... but I may create another question only about this part later.

An update: I as just searching about using OpenCL, but it seeams even more complicated then multi-threading and I am not sure if the improvement would be good. would it be better then multi-threading ?

edit retag flag offensive close merge delete

Comments

You can also subsample your input image if it doesn't affect your vision task.

Eduardo gravatar imageEduardo ( 2016-08-09 11:39:44 -0600 )edit

Hi. Sorry but I didn't really understand what you meant by it.

carlos123alberto gravatar imagecarlos123alberto ( 2016-08-10 05:01:46 -0600 )edit

Instead of working on full resolution image (e.g. 1920x1080), you could also subsample the image to 640x480 and do your processing on this reduced image if this doesn't affect your vision task.

For example, if you want to detect faces, you can do it on 640x480 instead of 1920x1080.

Eduardo gravatar imageEduardo ( 2016-08-10 10:39:59 -0600 )edit

Yes. Reducing the resolution does help a lot, but I am already doing it, I am working with 640x360. But it is some heavy processing and I only have like 6fps and I should have at least 20fps for it to work great

carlos123alberto gravatar imagecarlos123alberto ( 2016-08-10 11:12:52 -0600 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2016-08-09 08:33:13 -0600

mshabunin gravatar image

You can use cv::parallel_for_ from OpenCV for processing images if your algorithm allows it. It uses pthreads or TBB on Android, depending on your OpenCV library build environment and options.

Some algorithms in OpenCV are implemented using mentioned parallelization routine, you can change global behavior by calling cv::setNumThreads.

If your algorithm can not be implemented to process image in several threads, probably you can use pthreads functions to process each frame in separate thread, but this question is out of scope.

edit flag offensive delete link more

Comments

Is parallel for compatible with all architecture? I had the impression that it was compatible only with Intel processors, but I think I might be wrong. I need compatibility in the application. But anyway, my frame processing wouldn't be compatible with multiple threads in the same frame as it stores points of the frame and compares them in the loop.

I am currently think about editing Opencv functions to be compatible with multi threading, I am studying DeliverAndDrawFrame, it receives the frame from a thread, calls OnCameraFrame that is our natural callback to edit the frame and draws the frame, I am thinking of segmenting it in just deliver frame in a queue for multiple threads to catch it when ready and then a draw frame for when there is a processed frame do be draw

carlos123alberto gravatar imagecarlos123alberto ( 2016-08-10 04:56:32 -0600 )edit

parallel_for uses different backends depending on platform, fallback pthreads backend should be automatically activated on Android.

mshabunin gravatar imagemshabunin ( 2016-08-10 17:21:47 -0600 )edit

Question Tools

1 follower

Stats

Asked: 2016-08-06 19:01:03 -0600

Seen: 2,409 times

Last updated: Aug 09 '16