Ask Your Question
0

Keras DenseNet121 breaks on opencv dnn

asked 2018-11-26 22:28:46 -0500

updated 2018-11-27 03:51:12 -0500

following http://answers.opencv.org/question/18... I'm trying to get densenet to work on openCV DNN but getting:

"error: OpenCV(3.4.2) /io/opencv/modules/dnn/src/tensorflow/tf_graph_simplifier.cpp:712: error: (-2:Unspecified error) Tensor's data type is not supported in function 'getTensorContent'"

import numpy as np
from keras import applications
from keras import backend as K
import cv2 as cv
import tensorflow as tf

model = applications.densenet.DenseNet121(input_shape=(224, 224, 3), weights='imagenet', include_top=True)
K.set_learning_phase(0)
sess = K.get_session()

print(model.input, model.outputs)

## Tensor("input_1:0", shape=(?, 224, 224, 3), dtype=float32) [<tf.Tensor 'fc1000/Softmax:0' shape=(?, 1000) dtype=float32>]

from tensorflow.python.tools import freeze_graph
from tensorflow.python.tools import optimize_for_inference_lib

MODEL_PATH = 'out'
MODEL_NAME = 'test'
input_node_name = 'input_1'
output_node_name = 'fc1000/Softmax'
!rm -rf {MODEL_PATH}/

tf.train.write_graph(sess.graph_def, MODEL_PATH, f'{MODEL_NAME}_graph.pb', as_text=False)
tf.train.write_graph(sess.graph_def, MODEL_PATH, f'{MODEL_NAME}_graph.pbtxt')
tf.train.Saver().save(sess, f'{MODEL_PATH}/{MODEL_NAME}.chkp')

freeze_graph.freeze_graph(f'{MODEL_PATH}/{MODEL_NAME}_graph.pbtxt',
                          None, False,
                          f'{MODEL_PATH}/{MODEL_NAME}.chkp',
                          output_node_name,
                          "save/restore_all",
                          "save/Const:0",
                          f'{MODEL_PATH}/frozen_{MODEL_NAME}.pb',
                          True, "")

graph_def = tf.GraphDef()
with tf.gfile.Open(f'{MODEL_PATH}/frozen_{MODEL_NAME}.pb', "rb") as f:
    graph_def.ParseFromString(f.read())

output_graph_def = optimize_for_inference_lib.optimize_for_inference(
    graph_def, [input_node_name], [output_node_name], tf.float32.as_datatype_enum)

with tf.gfile.GFile(f'{MODEL_PATH}/opt_{MODEL_NAME}.pb', "wb") as f:
    f.write(output_graph_def.SerializeToString())

# Strip Const nodes.
for i in reversed(range(len(graph_def.node))):
    if graph_def.node[i].op == 'Const':
        del graph_def.node[i]
#     for attr in ['T', 'data_format', 'Tshape', 'N', 'Tidx', 'Tdim',
#                  'use_cudnn_on_gpu', 'Index', 'Tperm', 'is_training',
#                  'Tpaddings']:
#         if attr in graph_def.node[i].attr:
#             del graph_def.node[i].attr[attr]

# Save stripped model.
tf.train.write_graph(graph_def, "", f'{MODEL_PATH}/stripped_{MODEL_NAME}.pbtxt', as_text=True)

net = cv.dnn.readNetFromTensorflow(f'{MODEL_PATH}/opt_{MODEL_NAME}.pb', f'{MODEL_PATH}/stripped_{MODEL_NAME}.pbtxt')

error: OpenCV(3.4.2) /io/opencv/modules/dnn/src/tensorflow/tf_graph_simplifier.cpp:712: error: (-2:Unspecified error) Tensor's data type is not supported in function 'getTensorContent'

It's hard to tell which node is causing that from the error. any idea please? cheers

edit retag flag offensive close merge delete

Comments

I also get the same error when tried:

# net = cv.dnn.readNetFromTensorflow(f'{MODEL_PATH}/opt_{MODEL_NAME}.pb')
# net = cv.dnn.readNetFromTensorflow(f'{MODEL_PATH}/frozen_{MODEL_NAME}.pb')
Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-26 22:29:50 -0500 )edit

error: OpenCV(3.4.2) /io/opencv/modules/dnn/src/tensorflow/tf_graph_simplifier.cpp:659: error: (-215:Assertion failed) !field.empty() in function 'getTensorContent'

if I strip the attributes commented out on the list above

Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-26 22:35:08 -0500 )edit

optimize_for_inference also gives lots of warnings:

"WARNING:tensorflow:Incorrect shape for mean, found (0,), expected (64,), for node conv1/bn/FusedBatchNorm
WARNING:tensorflow:Didn't find expected Conv2D input to 'conv1/bn/cond/FusedBatchNorm'
WARNING:tensorflow:Didn't find expected Conv2D input to 'conv2_block1_0_bn/FusedBatchNorm'
WARNING:tensorflow:Didn't find expected Conv2D input to 'conv2_block1_0_bn/cond/FusedBatchNorm'
WARNING:tensorflow:Incorrect shape for mean, found (0,), expected (128,), for node conv2_block1_1_bn/FusedBatchNorm
WARNING:tensorflow:Didn't find expected Conv2D input to 'conv2_block1_1_bn/cond/FusedBatchNorm'"
Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-26 22:39:37 -0500 )edit

using tf 1.12.0 and cv 3.4.3

Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-27 00:02:34 -0500 )edit

The problem is in Test/Train phase switches at an every batch normalization node. We need to disable all of them somehow differently from modifying text graph. Is there a way in Keras to turn all the keras_learning_phase nodes to false?

dkurt gravatar imagedkurt ( 2018-11-27 03:06:51 -0500 )edit

Thanks for the quick reply. I've tried K.set_learning_phase(0) (see above) but it doesn't seem to change anything. Is that what do you mean?

Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-27 03:50:45 -0500 )edit
1

I got it, you have to set the learning phase before loading the model! will post the answer tomorrow. However, I wasn't able to get optimize_for_inference to work due to the FusedBatchNorm: opencv-4.0.0/modules/dnn/src/tensorflow/tf_importer.cpp:497: error: (-2:Unspecified error) Input layer not found: conv2_block1_1_bn/FusedBatchNorm_1 in function 'connect' Any idea?

Henrique Mendonça gravatar imageHenrique Mendonça ( 2018-11-27 06:08:31 -0500 )edit

1 answer

Sort by » oldest newest most voted
1

answered 2018-11-29 18:52:49 -0500

Although, I still wasn't able to get optimize_for_inference to work due to the FusedBatchNorm, thanks for the feedback from @dkurt and https://github.com/keras-team/keras/i... which explains the keras learning_phase. You have to set the learning phase before loading the model!

load the model with set_learning_phase(0):

import numpy as np
from keras import applications
from keras import backend as K
import tensorflow as tf

K.set_learning_phase(0)  ##
model = applications.densenet.DenseNet121(input_shape=(224, 224, 3), weights='imagenet', include_top=True)
sess = K.get_session()

print(model.input, model.outputs)
## Tensor("input_1:0", shape=(?, 224, 224, 3), dtype=float32) [<tf.Tensor 'fc1000/Softmax:0' shape=(?, 1000) dtype=float32>]

freeze it:

from tensorflow.python.tools import freeze_graph
from tensorflow.python.tools import optimize_for_inference_lib

MODEL_PATH = 'out'
MODEL_NAME = 'test'
input_node_name = 'input_1'
output_node_name = 'fc1000/Softmax'
!rm -rf {MODEL_PATH}/

tf.train.write_graph(sess.graph_def, MODEL_PATH, f'{MODEL_NAME}_graph.pb', as_text=False)
tf.train.write_graph(sess.graph_def, MODEL_PATH, f'{MODEL_NAME}_graph.pbtxt')
tf.train.Saver().save(sess, f'{MODEL_PATH}/{MODEL_NAME}.chkp')

freeze_graph.freeze_graph(f'{MODEL_PATH}/{MODEL_NAME}_graph.pbtxt',
                          None, False,
                          f'{MODEL_PATH}/{MODEL_NAME}.chkp',
                          output_node_name,
                          "save/restore_all",
                          "save/Const:0",
                          f'{MODEL_PATH}/frozen_{MODEL_NAME}.pb',
                          True, "")

then load it with dnn:

import cv2 as cv
net = cv.dnn.readNetFromTensorflow(f'{MODEL_PATH}/frozen_{MODEL_NAME}.pb')

# Smoke test
inp = np.ones([1, 3, 224, 224]).astype(np.float32)
net.setInput(inp)
dnn_out = net.forward()
print(dnn_out.shape, dnn_out[0,:5])
## (1, 1000) [2.0760612e-04 2.6876197e-04 5.9680151e-05 5.5908626e-05 1.4762023e-04]

As said, I wasn't able to get optimize_for_inference to work due to the FusedBatchNorm: WARNING:tensorflow:Didn't find expected Conv2D input to 'conv2_block1_0_bn/FusedBatchNorm_1' opencv-4.0.0/modules/dnn/src/tensorflow/tf_importer.cpp:497: error: (-2:Unspecified error) Input layer not found: conv2_block1_1_bn/FusedBatchNorm_1 in function 'connect' So please let me know if you know a solution for that. Thanks

edit flag offensive delete link more

Comments

Just use the native dnn?

holger gravatar imageholger ( 2018-11-30 08:11:05 -0500 )edit
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2018-11-26 22:28:46 -0500

Seen: 589 times

Last updated: Nov 29 '18