Ask Your Question
1

Dnn forward result has rows = -1, colums = -1

asked 2018-12-23 06:46:31 -0500

francy91 gravatar image

updated 2018-12-23 08:50:34 -0500

berak gravatar image

Hi there,

I'm trying to translate this WORKING Python code for Neural Transfer Style on Java (Android).(the original work is from https://www.pyimagesearch.com/2018/08...).

In python the test image shape is uint8 (479, 700, 3) and the output of the DNN is of shape float32 (1,3,480,700).

When I try converting the code to Java (see below), the resulting matrix I get is Mat [ -1*-1*CV_32FC1, isCont=true, isSubmat=true, nativeObj=0x75cf3fd9e0, dataAddr=0x75cea5e000 ], with rows = -1 and columns = -1. It also seems from CV_32FC1 that the output in Java has 1 float32 channel, while on python there are 3 I think. Maybe there's something I'm missing, but I don't know how to solve it.

I post below my working code in python and my initial tentative in Java.

I've searched a lot these days to find a solution, but I cannot find it, please help me!!! Thank you

PYTHON CODE

      import cv2
      import base64
      import numpy as np

    // read image directly from file, shape is uint8 (479, 700, 3)
    img = cv2.imread("./testImages/bike.jpg")
    // load dnn model
    net = cv2.dnn.readNetFromTorch("./models/instance_norm/mosaic.t7")
    // resize images if width > 700
    h, w = img.shape[:2]
    width = 700
    if w <= width:
        dim = (w, h)
    else:
        ratio = width / float(w)
        dim = (width, int(h * ratio))
        img = cv2.resize(img, dim, interpolation=cv2.INTER_AREA)
    // Feed the neural net
    blob = cv2.dnn.blobFromImage(img, 1.0, dim,
        (103.939, 116.779, 123.680), swapRB=False, crop=False)
    net.setInput(blob)
    // Run neural network, out shape is float32 (1,3,480,700)
    out = net.forward()
    // Add back in the mean subtraction
    // output shape is now float32 (3, 480, 700)
    out = out.reshape((3, out.shape[2], out.shape[3]))
    out[0] += 103.939
    out[1] += 116.779
    out[2] += 123.680
   // output shape is now float32 (480, 700, 3)
    out = out.transpose(1,2,0)
    // convert image from BGR to RGB and avoid artifacts. output shape is now uint8 (480, 700, 3)
    out = cv2.convertScaleAbs(cv2.cvtColor(out, cv2.COLOR_BGR2RGB))

JAVA CODE

// read image directly from file 
Mat image = Imgcodecs.imread(filePath);
// load dnn model 
Net net = Dnn.readNetFromTorch(modelPath);
// resize images if width > 700
int h = image.height();
int w = image.width();
int width = 700;
int height;
Mat resized;
if(w <= width) {
    width = w;
    height = h;
    resized = image;
} else {
    float ratio = width / (float) w;
    height = (int)(h * ratio);
    resized = new Mat();
    Imgproc.resize(image, resized, new Size(width, height), 0, 0, Imgproc.INTER_AREA);
}

// Feed the neural net
net.setInput(Dnn.blobFromImage(resized,
                               1.0,
                               new Size(width, height),
                               new Scalar(103.939, 116.779, 123.680),
                               false,
                               false));

// I have problems here with the resulting matrix
Mat stylizedImage = net.forward();
Core.add(stylizedImage, new Scalar(103.939, 116.779, 123.680), stylizedImage);

// ....
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
3

answered 2018-12-23 07:14:47 -0500

berak gravatar image

updated 2018-12-23 07:44:03 -0500

there is no bug here, you're only looking at the wrong variables

(rows and cols can only hold 2 dimensions, so they are not used here, and set to -1).

the dimensions for the dnn blob Mat's are in:

mat.dims();  // blob's dimensions
mat.size(0); // how many images in the blob
mat.size(1); // how many "channels" or "planes"
mat.size(2); // height
mat.size(3); // width

i don't have your data, so i cannot try, but if the network output from the python version is (1,3,480,700) , it means, there are 3 seperate color channels in it. so,

Mat result = net.forward();

int H = result.size(2);
int W = result.size(3);

// step 1: reshape it to a long vertical strip:
Mat strip = result.reshape(1, H * 3);

// step 2: collect the color planes into a list:
List<Mat> lis = new ArrayList<>();
lis.add(strip.submat(0,H, 0,W));
lis.add(strip.submat(H,2*H, 0,W));
lis.add(strip.submat(2*H,3*H, 0,W));

// step 3: merge planes into final bgr image
Mat bgr = new Mat();
Core.merge(lis, bgr);

// last: add the mean value
Core.add(bgr, new Scalar(103.939, 116.779, 123.680), bgr);
edit flag offensive delete link more

Comments

data can be obtained from https://github.com/jcjohnson/fast-neu..., and a complete example is here: https://github.com/berak/opencv_small...

berak gravatar imageberak ( 2018-12-23 08:46:21 -0500 )edit
1

Thanks for the reply. The image passed to the blob has in Java these characteristics resizedMat [ 479*700*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0x75dd839ea0, dataAddr=0x75cbe0d000 ], which is same as python code (3 channels, 8 bit integers). After passing the data to the blob, the blob has these characteristics Mat [ -1*-1*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x75dd839c60, dataAddr=0x75cae00000 ]. I tried your approach but the java bindings of Android OpenCv don't allow me to take the size using indexes, like int H = result.size(2);

francy91 gravatar imagefrancy91 ( 2018-12-23 08:50:05 -0500 )edit
1

hmm indeed this would require opencv 4.0.

but luckily here, the image planes are the same size as your input, so you can take W and H from there ;)

berak gravatar imageberak ( 2018-12-23 08:59:31 -0500 )edit
1

I try the example using OpenCv 4.0 and report to you soon. In the meanwhile, thks again for the support!

francy91 gravatar imagefrancy91 ( 2018-12-23 09:01:46 -0500 )edit
2

Just tried with OpenCv Android SDK 4.0.1, I confirm everything works as expected with your sample. Thanks again for your help!

francy91 gravatar imagefrancy91 ( 2018-12-23 09:50:49 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-23 06:46:31 -0500

Seen: 233 times

Last updated: Dec 23 '18