Ask Your Question
0

Opencv dnn import dropout layer error after finetuning Keras vgg16

asked 2018-02-01 04:20:20 -0500

MennoK gravatar image

updated 2018-02-01 06:43:43 -0500

Hi,

A few days ago I asked a question about importing a pretrained keras vgg16 model into Opencv dnn [1].

Now I finetuned the vgg16 for my own application by excluding the existed imagenet head and adding a new head to the model. Below shows the "pseudocode" how it's done:

    baseModel = VGG16(input_shape=(224, 224, 3), weights='imagenet', include_top=False)

    headModel = baseModel.output

    headModel = Flatten(name="flatten")(headModel)
    headModel = Dense(256, activation="relu")(headModel)
    headModel = Dropout(0.5)(headModel)
    headModel = Dense(5, activation="softmax")(headModel)

    model = Model(inputs=baseModel.input, outputs=headModel)

Subsequently, I train the new model with my own data and export it similar to the answer of my previous question. However when I try to read the net into opencv, it returns a ImportError:

 cv2.error: C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_importer.cpp:1487: error: (-2) Unknown layer type PlaceholderWithDefault in op dropout_1/keras_learning_phase in function cv::dnn::experimental_dnn_v3::`anonymous-namespace'::TFImporter::populateNet

I've read on github, there is a solution to include dropout layers (https://github.com/opencv/opencv/pull...). Do you have any suggestions on how to implement this with keras? Or am I just making it myself difficult using Keras on top of tensorflow.

I have one more additional question: Do you ever plan to implement a readNetFromKeras(...) where a config.json and weights.h5 is given?

Edit:

Pbtxt file before (so flatten and dropout layers are included)

 ... some stuff before ...    
 node {
  name: "flatten/Reshape"
  op: "Reshape"
  input: "block5_pool/MaxPool"
  input: "flatten/stack"
}
node {
  name: "dense_1/MatMul"
  op: "MatMul"
  input: "flatten/Reshape"
  input: "dense_1/kernel"
  attr {
    key: "transpose_a"
    value {
      b: false
    }
  }
  attr {
    key: "transpose_b"
    value {
      b: false
    }
  }
}
node {
  name: "dense_1/BiasAdd"
  op: "BiasAdd"
  input: "dense_1/MatMul"
  input: "dense_1/bias"
}
node {
  name: "dense_1/Relu"
  op: "Relu"
  input: "dense_1/BiasAdd"
}
node {
  name: "dropout_1/keras_learning_phase"
  op: "PlaceholderWithDefault"
  input: "dropout_1/keras_learning_phase/input"
  attr {
    key: "dtype"
    value {
      type: DT_BOOL
    }
  }
  attr {
    key: "shape"
    value {
      shape {
      }
    }
  }
}
node {
  name: "dropout_1/cond/Switch"
  op: "Switch"
  input: "dropout_1/keras_learning_phase"
  input: "dropout_1/keras_learning_phase"
}
node {
  name: "dropout_1/cond/mul/Switch"
  op: "Switch"
  input: "dense_1/Relu"
  input: "dropout_1/keras_learning_phase"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@dense_1/Relu"
      }
    }
  }
}
node {
  name: "dropout_1/cond/mul"
  op: "Mul"
  input: "dropout_1/cond/mul/Switch:1"
  input: "dropout_1/cond/mul/y"
}
node {
  name: "dropout_1/cond/dropout/Shape"
  op: "Shape"
  input: "dropout_1/cond/mul"
  attr {
    key: "out_type"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "dropout_1/cond/dropout/random_uniform/RandomUniform"
  op: "RandomUniform"
  input: "dropout_1/cond/dropout/Shape"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "seed"
    value {
      i: 87654321
    }
  }
  attr {
    key: "seed2"
    value {
      i: 7788661
    }
  }
}
node {
  name: "dropout_1/cond/dropout/random_uniform/sub"
  op: "Sub"
  input: "dropout_1/cond/dropout/random_uniform/max"
  input: "dropout_1/cond/dropout/random_uniform/min"
}
node {
  name: "dropout_1/cond/dropout/random_uniform/mul"
  op: "Mul"
  input: "dropout_1/cond/dropout/random_uniform/RandomUniform"
  input: "dropout_1/cond/dropout/random_uniform/sub"
}
node {
  name: "dropout_1/cond/dropout/random_uniform"
  op: "Add"
  input: "dropout_1/cond/dropout/random_uniform/mul"
  input: "dropout_1/cond/dropout/random_uniform/min"
}
node {
  name: "dropout_1/cond/dropout/add"
  op: "Add"
  input: "dropout_1/cond/dropout/keep_prob"
  input: "dropout_1 ...
(more)
edit retag flag offensive close merge delete

Comments

Here is link to the used .pb and .pbtxt file: https://drive.google.com/file/d/120G3...

I edited already the flatten layers in the said pbtxt file

MennoK gravatar imageMennoK ( 2018-02-01 04:23:35 -0500 )edit

@MennoK, dropout layer actually does nothing during testing phase. You can just exclude all the unused nodes the same way as we did for Flatten node. Create a text graph for this model and keep only used nodes.

dkurt gravatar imagedkurt ( 2018-02-01 05:14:53 -0500 )edit

Hey, I tried this already (if I remember correctly, you mentioned it in another github issue). So I removed every node with name: "dropout_1/..." and changed the lines where input = dropout_1 to input = Node name above in the pbtxt file.

After this, I'm able to read the model into opencv. However the test results are completely different in opencv compared to the results in Keras itself.

MennoK gravatar imageMennoK ( 2018-02-01 05:32:53 -0500 )edit

@dkurt I added a snippet of the .pbtxt file before and after the changes.

MennoK gravatar imageMennoK ( 2018-02-01 06:44:34 -0500 )edit

@MennoK, Have you checked that TensorFlow produces similar results for similar inputs? I mean dropout branch is disabled. Please show how you run a model and pass input tensor and testing phase flag.

dkurt gravatar imagedkurt ( 2018-02-01 07:06:50 -0500 )edit

I use Keras on top of Tensorflow, so I test the results using Keras which is simply model.predict(image) for each image in the test set. This also means I have to do an extra step, namely converting the Keras model (json and h5) to tensorflow's pb.

I do this as follows:

MennoK gravatar imageMennoK ( 2018-02-01 07:28:55 -0500 )edit

Code to convert Keras to Tensorflow:

jsonFile = open('vgg16.json', 'r')
loadModelJson = jsonFile.read()
model = model_from_json(loadModelJson)
model.load_weights("modelweights_vgg16.h5")

K.set_learning_phase(0)

pred_node_names = [None]
pred = [None]
for i in range(1):
    pred_node_names[i] = "output_node"+str(i)
    pred[i] = tf.identity(model.outputs[i], name=pred_node_names[i])

sess = K.get_session()
constant_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), ["output_node0"])
graph_io.write_graph(constant_graph, ".", "model.pb", as_text=False)
MennoK gravatar imageMennoK ( 2018-02-01 07:29:41 -0500 )edit

After this step I followed your instructions

MennoK gravatar imageMennoK ( 2018-02-01 07:33:18 -0500 )edit

@MennoK, I mean you said that results are different between OpenCV and Keras but are you sure that you receive expected results in Keras? It's about dropout. Can you share a way how you compare results from Keras and OpenCV.

dkurt gravatar imagedkurt ( 2018-02-01 08:18:11 -0500 )edit

@dkurt I'm not sure I understand what you mean. I can't send all the code, but I'm sure I receive the expected results in keras. However a confusion matrix from opencv shows:

[[   8.    0.    0. 1862.  122.]
 [   0.    0.    0.  272.   32.]
 [   0.    0.    7.  170.   22.]
 [   0.    0.    0.  137.   42.]
 [   0.    0.    0.  401. 1012.]]
MennoK gravatar imageMennoK ( 2018-02-01 08:56:09 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
2

answered 2018-02-01 09:23:23 -0500

dkurt gravatar image

Optimize

python ~/tensorflow/tensorflow/python/tools/optimize_for_inference.py \
  --input model.pb \
  --output opt_model.pb \
  --input_names input_1 \
  --output_names output_node0

Text graph

See http://answers.opencv.org/question/18...

Flatten

Remove nodes flatten/Shape, flatten/strided_slice, flatten/Prod, flatten/stack Replace

node {
  name: "flatten/Reshape"
  op: "Reshape"
  input: "block5_pool/MaxPool"
  input: "flatten/stack"
}

to

node {
  name: "flatten/Reshape"
  op: "Flatten"
  input: "block5_pool/MaxPool"
}

Dropout

Remove nodes dropout_1/keras_learning_phase, dropout_1/cond/Switch, dropout_1/cond/mul/Switch, dropout_1/cond/mul, dropout_1/cond/dropout/Shape, dropout_1/cond/dropout/random_uniform/RandomUniform, dropout_1/cond/dropout/random_uniform/sub, dropout_1/cond/dropout/random_uniform/mul, dropout_1/cond/dropout/random_uniform, dropout_1/cond/dropout/add, dropout_1/cond/dropout/Floor, dropout_1/cond/dropout/div, dropout_1/cond/dropout/mul, dropout_1/cond/Switch_1, dropout_1/cond/Merge.

Replace input: "dropout_1/cond/Merge" onto input: "dense_1/Relu".

Test

import tensorflow as tf
import cv2 as cv
import numpy as np

# Read the graph.
with tf.gfile.FastGFile('model.pb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())

np.random.seed(223)
inp = np.random.standard_normal([1, 224, 224, 3]).astype(np.float32)

with tf.Session() as sess:
    # Restore session
    sess.graph.as_default()
    tf.import_graph_def(graph_def, name='')

    out = sess.run(sess.graph.get_tensor_by_name('output_node0:0'),
                   feed_dict={'input_1:0': inp})

cvNet = cv.dnn.readNetFromTensorflow('model.pb', 'graph.pbtxt')
cvNet.setInput(inp.transpose(0, 3, 1, 2))
cvOut = cvNet.forward()
print np.max(np.abs(cvOut - out))

Here it's a graph:

node {
  name: "input_1"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "block1_conv1/convolution"
  op: "Conv2D"
  input: "input_1"
  input: "block1_conv1/kernel"
  attr {
    key: "padding"
    value {
      s: "SAME"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 1
        i: 1
        i: 1
      }
    }
  }
}
node {
  name: "block1_conv1/BiasAdd"
  op: "BiasAdd"
  input: "block1_conv1/convolution"
  input: "block1_conv1/bias"
}
node {
  name: "block1_conv1/Relu"
  op: "Relu"
  input: "block1_conv1/BiasAdd"
}
node {
  name: "block1_conv2/convolution"
  op: "Conv2D"
  input: "block1_conv1/Relu"
  input: "block1_conv2/kernel"
  attr {
    key: "padding"
    value {
      s: "SAME"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 1
        i: 1
        i: 1
      }
    }
  }
}
node {
  name: "block1_conv2/BiasAdd"
  op: "BiasAdd"
  input: "block1_conv2/convolution"
  input: "block1_conv2/bias"
}
node {
  name: "block1_conv2/Relu"
  op: "Relu"
  input: "block1_conv2/BiasAdd"
}
node {
  name: "block1_pool/MaxPool"
  op: "MaxPool"
  input: "block1_conv2/Relu"
  attr {
    key: "ksize"
    value {
      list {
        i: 1
        i: 2
        i: 2
        i: 1
      }
    }
  }
  attr {
    key: "padding"
    value {
      s: "VALID"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 2
        i: 2
        i: 1
      }
    }
  }
}
node {
  name: "block2_conv1/convolution"
  op: "Conv2D"
  input: "block1_pool/MaxPool"
  input: "block2_conv1/kernel"
  attr {
    key: "padding"
    value {
      s: "SAME"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 1
        i: 1
        i: 1
      }
    }
  }
}
node {
  name: "block2_conv1/BiasAdd"
  op: "BiasAdd"
  input: "block2_conv1/convolution"
  input: "block2_conv1/bias"
}
node {
  name: "block2_conv1/Relu"
  op: "Relu"
  input: "block2_conv1/BiasAdd"
}
node {
  name: "block2_conv2/convolution"
  op: "Conv2D"
  input: "block2_conv1/Relu"
  input: "block2_conv2/kernel"
  attr {
    key: "padding"
    value {
      s: "SAME"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 1
        i: 1
        i: 1
      }
    }
  }
}
node {
  name: "block2_conv2/BiasAdd"
  op: "BiasAdd"
  input: "block2_conv2/convolution"
  input: "block2_conv2/bias"
}
node {
  name: "block2_conv2/Relu"
  op: "Relu"
  input: "block2_conv2/BiasAdd"
}
node {
  name: "block2_pool/MaxPool"
  op: "MaxPool"
  input: "block2_conv2 ...
(more)
edit flag offensive delete link more

Comments

1

@dkurt thank you for your responses. I tested with tensorflow, and the results were as expected (same as in Keras). However Opencv still gives unexpected results. Here is a snippet of the code when I test with opencv

imagePaths = list(im.paths.list_images("./testset/"))
labels = getLabels(imagePaths)
resultsCv = np.zeros((5, 5), dtype=np.int64)

cvNet = cv.dnn.readNetFromTensorflow("model.pb", "graph.pbtxt")

for i, imagePath in enumerate(imagePaths):
    img = cv.imread(imagePath)
    blob = cv.dnn.blobFromImage(img, 1.0, (224, 224))
    cvNet.setInput(blob)
    result = cvNet.forward()
    bestPrediction = np.argmax(result, axis=1)[0]
    truth = np.argmax(labels[i], axis=0)
    resultsCv[truth][bestPrediction] += 1
MennoK gravatar imageMennoK ( 2018-02-02 02:46:26 -0500 )edit

GetLabels is own function to retrieve label indices from the imagepaths

MennoK gravatar imageMennoK ( 2018-02-02 02:48:50 -0500 )edit
1

Results Tensorflow:

[[1846  143    0    0    3]
 [   4  281   18    0    1]
 [   0   14  183    0    2]
 [   0    0    0  179    0]
 [   2    2    2    5 1402]]

Results Opencv:

[[  20    1    0 1911   60]
 [   3    2    0  276   23]
 [   0    0   13  166   20]
 [   0    0    0  150   29]
 [   0    0    0  591  822]]
MennoK gravatar imageMennoK ( 2018-02-02 02:52:53 -0500 )edit

@MennoK, Am I right that maximal absolute difference between OpenCV and Keras is about 1e-7 using code from answer? If you have different results in the application - let's find a mistake in the application's code. cv.imread reads an image and returns BGR interleaved Mat. cv.dnn.blobFromImage makes a planar 4D blob from an image. Take a look onto the flags of blobFromImage. There are crop to enable/disable keeping ratios and swapRB to make a blob with RGB or BGR channels. I think you need to use the following arguments: cv.dnn.blobFromImage(img, 1.0, (224, 224), (0, 0, 0), swapRB=True, crop=False). BTW you didn't show how you receive a target confusion matrix.

dkurt gravatar imagedkurt ( 2018-02-02 03:40:43 -0500 )edit

Using the code from the Test in your answer. The max absolute error is 1.937151e-07. Also the outputs are the same

Output cv:

[[0.18312776 0.06833366 0.5722188  0.09505653 0.08126326]]

Output Tensorflow:

[[0.18312757 0.06833372 0.5722189  0.09505657 0.08126317]]

I tried the suggested configuration settings for blobFromImage. Still no success. With target confusion matrix, you mean the confusion matrix I receive with Tensorflow? It's same process as the opencv part. Only I do the preprocessing myself. e.g. resizing to (224, 224), then I convert the img to an array using the img_to_array function of Keras, finally I extend dimensions: (img = np.expand_dims(img, axis=0)). The resulting image I give to the sess.run(..)

MennoK gravatar imageMennoK ( 2018-02-02 04:21:21 -0500 )edit

Tensorflow code:

# Read the graph.
with tf.gfile.FastGFile('model.pb', 'rb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())
with tf.Session() as sess:
    # Restore session
    sess.graph.as_default()
    tf.import_graph_def(graph_def, name='')

     imagePaths = list(im.paths.list_images("./testset/"))
    imgs, labels = preprocess(imagePaths) # resize to 224,224 and convert to array and get labels
    for i, img in enumerate(imgs):
        img = np.expand_dims(img, axis=0)
        out = sess.run(sess.graph.get_tensor_by_name('output_node0:0'),
                       feed_dict={'input_1:0': img})

        bestPrediction = np.argmax(out, axis=1)[0]

        truth = np.argmax(labels[i], axis=0)
MennoK gravatar imageMennoK ( 2018-02-02 04:26:15 -0500 )edit

@dkurt you're indicating that an input image for tensorflow and opencv are different after the preprocessing is done?

MennoK gravatar imageMennoK ( 2018-02-02 06:28:18 -0500 )edit

@MennoK, it's just a math. If for all xf(x) == g(x) then f(x) != g(y) => x != y. Try to use swapRB=False if preprocess doesn't swap channels. Or at least preprocess image for OpenCV using the same function: blob = expandedImg.transpose(0, 3, 1, 2).

dkurt gravatar imagedkurt ( 2018-02-02 07:31:52 -0500 )edit
1

True, Ill try to figure this out. Thanks for the responses and good luck with the further development of dnn

MennoK gravatar imageMennoK ( 2018-02-02 09:40:01 -0500 )edit

@dkurt I tried your suggestions:

1) use the result image of blobFromImage both for tensorflow (transposing to (0,2,3,1) and opencv. This results in the same output, but unexpected.

2) preprocessing the image for Opencv using the same function as I preprocess the images for training and testing in Tensorflow. This results an expected prediction for Tensorflow, but opencv results a NaN prediction for each class sometimes or an unexpected result.

MennoK gravatar imageMennoK ( 2018-02-05 03:48:01 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-02-01 04:20:20 -0500

Seen: 761 times

Last updated: Feb 01 '18