OpenCV4Android 3.1 Mat to Allocation Renderscript

asked 2016-08-03 09:50:07 -0600

I am making an Android app to simulate colourblindness. It takes a camera feed and manipulates at runtime.

My question:

I have an OpenCV Mat object rgba, containing all Pixels of a given frame with red, green, blue and alpha (rgba) values. To speed up processing I'd like to use Renderscript, but when I convert that Mat rgba to Bitmap I lose about 5 fps, making any speedgain by Renderscript pointless. So I would like to know a way of placing the contents of Mat rgba into an Allocation that I can give to Renderscript, where I then just do Matrix multiplication for it all and return it.

What I tried:
- Converting Mat to Bitmap and then give to Renderscript, which works but is way too expensive (minus ~5 fps).
- Allocation aName2 = Allocation.createFromString(mRS, rgba.dump(), 3); // this just shows i am desperate ;(
In case you're interested what I am doing at the moment:

I based my app on the imagemanipulations example provided by OpenCV, specifically the Sepia filter option.

In onCameraViewStarted{} I set a Kernel such as for example the Sepia Kernel:

        // Fill sepia kernel
        mSepiaKernel = new Mat(4, 4, CvType.CV_32F);
        mSepiaKernel.put(0, 0, /* R */0.189f, 0.769f, 0.393f, 0f);
        mSepiaKernel.put(1, 0, /* G */0.168f, 0.686f, 0.349f, 0f);
        mSepiaKernel.put(2, 0, /* B */0.131f, 0.534f, 0.272f, 0f);
        mSepiaKernel.put(3, 0, /* A */0.000f, 0.000f, 0.000f, 1f);

Then in onCameraFrame OpenCV places the frame into a Mat and the filter is run over it, causing it to have this yellowish tint.

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {  
Mat rgba = inputFrame.rgba(); 
Core.transform(rgba, rgba, mSepiaKernel); 
return rgba;  
}

I have this running for Protan, Deutan and Tritan and on the S7 edge I get 10 fps. This is okayish, but as soon as we put the app on an older phone it becomes incredibly slow. Hence I would like to do what Core.transform() does in Renderscript instead.

edit retag flag offensive close merge delete

add a comment

answered 2016-08-04 09:19:27 -0600

cmaster11
26 ●2

updated 2016-08-04 12:25:56 -0600

Hi!

This is a crazy one, but possible to achieve.

Note: a feature like this one has to be treated as experimental. It is a proof of concept, based upon non-official ways of "doing things with RenderScript".

I created this example just for this case: https://bitbucket.org/cmaster11/rsbookexamples/src/tip/OpenCVInteropExample/

Main file, for reference: MainActivity.java

The concept of this example is based upon the RenderScript (RS) ability to instantiate an Allocation using a user-provided data pointer.

Short explanation: inside RenderScript source code, you can see one method declaration:

long nAllocationCreateTyped(long type, int mip, int usage, long pointer)

The last argument, pointer, can be used to tell RenderScript that it should create a new Allocation, pointing at a certain memory address. Binding this behavior with OpenCV, where you can invoke the Mat.dataAddr() method to get a Mat data pointer, lets you create a RenderScript allocation that completely overlaps the OpenCV mat.

What the sample project does is:

Loads a custom drawable file, bundled together with the app, inside a Bitmap.
Creates an OpenCV mat from the Bitmap.
Instantiates a RenderScript allocation, which points to the OpenCV mat data pointer.
Instantiates another RS allocation, which points to another OpenCV mat, to be used as output.
Executes a kernel over the input allocation.
Converts the OpenCV output allocation to a Bitmap. Note: the output allocation is DIRECTLY bound to the output OpenCV allocation because they share the same memory address.

Note: please, refer to the README.md, which comes together with the example, to have it work correctly. It requires you to download the OpenCV SDK.

The ability to support a user-provided data pointer is dependent upon the RS device-specific driver. When tested on a Galaxy Note 3 (Android 5.1), it didn't work. There is a high chance that later versions of Android provide good support for this functionality.

This functionality works by enabling the RS support library at the cost of using a non-performant RS driver.

The process of binding a RS allocation to a user-provided pointer can be achieved by directly calling the native RS function RenderScript.nAllocationCreateTyped and passing it, as last argument, the pointer address to the user data (in this case, the OpenCV mat data pointer).

This process, however, requires the usage of Java reflection:

Retrieve the RenderScript.nAllocationCreateTyped hidden method.
Invoke it to obtain an Allocation pointer.
Instantiate a new Allocation class, using the hidden Allocation() constructor.

Reference: RenderScript: parallel computing on Android, the easy way

edit flag offensive delete link

Comments

Hi and thank you so much! I think this is doing exactly what I need. I ran into problems with applying the Sepia filter (it just turns the image into shades of green). I had this problem before, as I ran the intrinsic over Bitmap, so I don't think there's a problem in your code. I've marked your answer, but maybe you're a genius for this too:

ScriptIntrinsicColorMatrix colorMatrix = ScriptIntrinsicColorMatrix.create(mRS, element);
final Matrix4f mSepia = new Matrix4f(new float[]{
                0.189f, 0.769f, 0.393f, 0f,
                0.168f, 0.686f, 0.349f, 0f,
                0.131f, 0.534f, 0.272f, 0f,
                0.000f, 0.000f, 0.000f, 1f
        });
colorMatrix.setColorMatrix(mSepia); 
colorMatrix.forEach(inputAllocation, outputAllocation);

AllBecomesGood ( 2016-08-04 16:04:21 -0600 )edit

Isn't the matrix the following one?

0.393f, 0.349f, 0.272f, 0f, 
0.769f, 0.686f, 0.534f, 0f, 
0.189f, 0.168f, 0.131f, 0f, 
0.000f, 0.000f, 0.000f, 1f

cmaster11 ( 2016-08-04 16:30:37 -0600 )edit

If you are on Stack Overflow and want to get points here is the Thread I made there: http://stackoverflow.com/questions/38...

Didn't see your reply until it refreshed after my answer, will check.

AllBecomesGood ( 2016-08-04 16:48:56 -0600 )edit

This looks promising for Sepia, but absolutely not for my Protan Matrix. Why is yours left-rotated? Or do I completely misunderstand how the OpenCV Matrix is structured? I'm gonna be off soon, so might be looking at this again tomorrow.

AllBecomesGood ( 2016-08-04 17:11:27 -0600 )edit

Honestly, I'm not fond of this kind of topics, but I found this page: https://docs.rainmeter.net/tips/colormatrix-guide/

cmaster11 ( 2016-08-04 17:18:59 -0600 )edit

No worries, you've helped me plenty! It is odd as to why their Sepia Matrix is rotated, but I will be investigating tomorrow :)

AllBecomesGood ( 2016-08-04 18:27:16 -0600 )edit

add a comment

OpenCV4Android 3.1 Mat to Allocation Renderscript

1 answer

Comments

Links

Question Tools

Stats

Related questions

OpenCV4Android 3.1 Mat to Allocation Renderscript edit

1 answer

Comments

Links

Question Tools

Stats

Related questions

OpenCV4Android 3.1 Mat to Allocation Renderscript