Ask Your Question
0

DNN.net.forward() in Android

asked 2018-11-19 05:09:52 -0500

Mishal077 gravatar image

updated 2018-11-23 02:20:05 -0500

  String model = getPath("model.net", getApplicationContext());
  net = Dnn.readNet(model);    
 Mat frame = inputFrame.rgba();
 //   Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
    // Forward image through network.
    Imgproc.resize(frame,frame,new Size(500,441));
    Mat blob = Dnn.blobFromImage(frame, IN_SCALE_FACTOR,
            new Size(IN_WIDTH, IN_HEIGHT),
            new Scalar(MEAN_VAL, MEAN_VAL, MEAN_VAL), /*swapRB*/true);
    net.setInput(blob);
    Mat detections = net.forward()

following is the issue I get at net.forward()

Assertion failed (inputs[0] == inputs[i]) in virtual bool cv::dnn::EltwiseLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const, file /build/3_4_pack-android/opencv/modules/dnn/src/layers/eltwise_layer.cpp,

edit retag flag offensive close merge delete

Comments

imho, you NEED the cvtColor call.

which network model are you using ? are IN_WIDTH, IN_HEIGHT correct for that ?

berak gravatar imageberak ( 2018-11-19 05:24:20 -0500 )edit

I am using the Caffe Model Enet https://modeldepot.io/timosaemann/enet

Yup Height & Width are perfect for them

Mishal077 gravatar imageMishal077 ( 2018-11-19 05:37:33 -0500 )edit

I uncommented the cvtColor & now i am getting this

Assertion failed (blobs.size() != 0) in virtual bool cv::dnn::ConvolutionLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const, file /build/3_4_pack-android/opencv/modules/dnn/src/layers/convolution_layer.cpp, line 233
Mishal077 gravatar imageMishal077 ( 2018-11-19 05:41:53 -0500 )edit

what about the prototxt ? don't you think, you need it ?

also, check the dims, again , please

there's also https://github.com/opencv/opencv_extr...

(guaranteed to work, at least)

berak gravatar imageberak ( 2018-11-19 05:43:56 -0500 )edit
1

Yup i had prototxt file for the same still getting the issue.

I am trying the other way around here switch to enet-model.net model checking it now.

Mishal077 gravatar imageMishal077 ( 2018-11-19 06:15:15 -0500 )edit

@Mishal077, please check blob.size(0), blob.size(1), blob.size(2) and blob.size(3).

dkurt gravatar imagedkurt ( 2018-11-23 02:57:16 -0500 )edit

@dkurt Android the size function is not there.

Mishal077 gravatar imageMishal077 ( 2018-11-27 04:37:05 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-11-23 05:25:26 -0500

berak gravatar image

updated 2018-11-23 06:08:48 -0500

odd image sizes are not allowed with enet, try like Size(512,256) instead (also needs to be a multiple of 4)

since you cannot access a 4d tensor from java directly, you'll have to reshape() the network output to a 2d strip of probability maps per class, like:

---
|A| 
---
|B|
---
|C|
---

now you have to find the highest probability for each pixel, and knowing which class it came from, assign a pixel color to your segmentation image:

import org.opencv.core.*;
import org.opencv.dnn.*;
import org.opencv.imgcodecs.*;
import org.opencv.imgproc.*;

public class SimpleSample {
    public static void main(String[] args) {
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        Scalar[] colors = new Scalar[]{ // obvious TODO: find better colours ;)
            new Scalar(1,22,123), new Scalar(28,53,4), new Scalar(81,2,243),
            new Scalar(2,223,244), new Scalar(12,2,3), new Scalar(2,3,4), 
            new Scalar(161,162,3), new Scalar(124,13,144), new Scalar(112,2,223),
            new Scalar(2,3,4), new Scalar(81,82,3), new Scalar(222,223,24),
            new Scalar(12,222,32), new Scalar(82,3,4), new Scalar(221,12,3), 
            new Scalar(182,83,84), new Scalar(221,12,3), new Scalar(52,3,4), 
            new Scalar(91,162,23), new Scalar(211,3,49)
        };
        Mat img = Imgcodecs.imread("c:/data/img/warp.png"); // your data here !
        Net net = Dnn.readNetFromTorch("c:/data/mdl/enet-model-best.net");
        Mat inputBlob = Dnn.blobFromImage(img, 
                    1.0/255, new Size(512,256), new Scalar(0, 0, 0), true, false);
        net.setInput(inputBlob);
        Mat res = net.forward();
        // the net's output is a list of "probability maps", one per class 
        //   (size(1) is, how many classes there are, 2 and 3 are H,W)
        System.out.println(res.size(0)+" "+res.size(1)+" "+res.size(2)+" "+res.size(3));
        // make a long, vertical "image strip" of it 
        //   (since we can't access 4d tensors from java)
        Mat strip = res.reshape(1, res.size(1) * res.size(2));
        // will keep "highest probability" per pixel
        Mat probs = new Mat(res.size(2), res.size(3), res.type(), Scalar.all(0));
        // color overlay
        Mat segm  = new Mat(probs.size(), CvType.CV_8UC3);
        // check each map, keep the pixel with highest probability 
        //   and (re)assign a color to it
        for (int i=0; i<res.size(1); i++) {
            // probs for class i
            Mat sub = strip.submat(i*res.size(2), (i+1)*res.size(2), 0, res.size(3));
            // find out, which pixels int the current map had a higher probability,
            //   than all of the prev. ones
            Mat gt = new Mat(); // 'greater than all' mask
            Core.compare(sub, probs, gt, Core.CMP_GT);
            // update probs and segmentation
            segm.setTo(colors[i], gt);
            sub.copyTo(probs, gt);
        }
        Mat result = new Mat();
        Imgproc.resize(segm, segm, img.size());
        Core.addWeighted(img, 0.8, segm, 0.2, 0, result); // alpa + beta == 1
        Imgcodecs.imwrite("segm.png", result);
    }
}
edit flag offensive delete link more

Comments

System.out.println(res.size(0)+" "+res.size(1)+" "+res.size(2)+" "+res.size(3));

Here the code is in native OpenCV in native Java but when it comes to Android we do not have the size function for the mat.What alternative i can use

Mishal077 gravatar imageMishal077 ( 2018-11-27 04:36:23 -0500 )edit

you need a more recent opencv ;(

or just "guess" the values: size(1) == number of classes (20), size(2) == input height, size(3) == input width

(at least for enet)

berak gravatar imageberak ( 2018-11-27 04:48:32 -0500 )edit

I am using the OpenCV SDK 3.4.4 for Android, I think it is the latest one? Ok I am trying the guess as mentioned

Mishal077 gravatar imageMishal077 ( 2018-11-27 04:53:48 -0500 )edit
1

Necessary changes have been merged about one month ago so it should be in 3.4.4 release: https://github.com/opencv/opencv/pull.... As an alternative solution, try this: m.size(i) == (i > 0 ? m.step1(i - 1) : m.total()) / m.step1(i);

dkurt gravatar imagedkurt ( 2018-11-27 06:59:16 -0500 )edit

Got it. @berak@dkurt I am trying to get the live feed from Mobile Camera using JavaCameraPreview. I am developing on Nexus 6P. The Frame I got from the JavaCameraPreview is of size 1980x1080 and the segm from the above code is 512x256. I got the crash at the

Imgproc.resize(segm, segm, img.size());


2018-11-28 11:05:21.078 15045-15123/? E/cv::error(): OpenCV(3.4.3) Error: Assertion failed (func != 0) in void cv::hal::resize(int, const uchar*, size_t, int, int, uchar*, size_t, int, int, double, double, int), file /build/3_4_pack-android/opencv/modules/imgproc/src/resize.cpp, line 3922

2018-11-28 11:05:21.110 15045-15123/? E/org.opencv.imgproc: imgproc::resize_13() caught cv::Exception: OpenCV(3.4.3) /build/3_4_pack-android/opencv/modules/imgproc/src/

Mishal077 gravatar imageMishal077 ( 2018-11-27 23:54:40 -0500 )edit

I found that OpenCV resize is missing support for certain combinations of input and output depths (numeric types), and interpolation types

Mishal077 gravatar imageMishal077 ( 2018-11-27 23:55:57 -0500 )edit

I am facing the issues with the

 Mat result = new Mat();
    Imgproc.resize(segm, segm,
            frame.size());
    Core.addWeighted(frame, 0.8, segm, 0.2, 0, result); // alpa + beta == 1

I got the following issues

 CvException [org.opencv.core.CvException: cv::Exception: OpenCV(3.4.3) /build/3_4_pack-android/opencv/modules/core/src/arithm.cpp:659: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'void cv::arithm_op(cv::InputArray, cv::InputArray, cv::OutputArray, cv::InputArray, int, void (**)(const uchar*, size_t, const uchar*, size_t, uchar*, size_t, int, int, void*), bool, void*, int)'
Mishal077 gravatar imageMishal077 ( 2018-11-28 04:08:22 -0500 )edit

i can't reproduce any of your current problems, but you might

System.out.println(frame);
System.out.println(segm);

to see, where or why they differ.

hmmm, you probably have to convert the image from the android cam from rgba to bgr first (or , the other way round, make the segm image CV_8UC4)

berak gravatar imageberak ( 2018-11-28 04:48:58 -0500 )edit

@berak i think i am doing something wrong here can you share the complete java file.

Mishal077 gravatar imageMishal077 ( 2018-11-28 05:02:45 -0500 )edit

the one in the answer IS the complete file.

berak gravatar imageberak ( 2018-11-28 05:22:39 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-19 05:09:52 -0500

Seen: 40 times

Last updated: Nov 23