Ask Your Question
0

dnn::readNetFromTensorflow() fail on loading pre-trained network on age-gender detection

asked 2018-03-04 21:51:43 -0500

Paul Kuo gravatar image

Dear opencv dnn developers,

[Environment] TensorFlow 1.5 python 3.5 Win7 opencv3.4

I am very new to the Tensor flow. Recently I found a pretrained dnn for age gender detection https://github.com/BoyuanJiang/Age-Gender-Estimate-TF. The model is defined in "inception_resnet_v1.py" and the latest model checkpoints can be found https://mega.nz/#!kaZkWDjb!xQvWi9B--FgyIPtIYfjzLDoJeh2PUBEZPotmzO9N6_M

I loaded model with the latest check point into Tensor Flow and running detection ok, and managed to freeze the graph using freeze_graph.py but when I load this frozen graph by cv::dnn::readNetFromTensorflow("XXX.pb"), it gets error:..."Unknown layer type shape in op map/Shape)".... or I tried cv::dnn::readNetFromTensorflow("XXX.pb", "XXX.pbtxt"), it also gets error:...unknown enumeration values of "DT_RESOURCE" for field "type..."

I have searched over the internet trying to find a solution... someone suggested using "optimize_for_inference" or "graph_transform" might help. As I have limited Tensor Flow knowledge, I do not understand how these two processes can solve the problem. and Also, cv::dnn::readNetFromTensorflow() method has the 2nd argument, which I also do not know in what situation I should/should not provide a .pbtxt for it?

Please help. If there is any further information needed to clarify the question, just let me know and I will supply.

Thanks in advance

edit retag flag offensive close merge delete

Comments

@Paul Kuo, please attach a reference to a frozen graph.

dkurt gravatar imagedkurt ( 2018-03-05 02:24:47 -0500 )edit

Hi, dkurt, Thank you for your reply. The detail of getting the frozen graph is described below... .....

after loading the model, we perform some age-gendger detection and then save it again and is going to convet to a frozen graph

saver.save(sess, "./pbs/model_XXX.ckpt") #save to a check point

tf.train.write_graph(sess.graph_def, ".\pbs", "graphDef.pb", as_text=False) # saved in binary form

tf.train.write_graph(sess.graph_def, ".\pbs", "graphDef.pbtxt", as_text=True) # saved in text form .....

And then run freeze_graph.py (provided from tensorflow/python/tools) with the script below...

python freeze_graph.py --input_graph=./pbs/graphDef.pb --input_checkpoint=./pbs/model_XXX.ckpt --output_graph =./pbs/frozenGraph.pb --output_node_names=genderOut,ageOut --input_binary=true

Paul Kuo gravatar imagePaul Kuo ( 2018-03-05 23:28:40 -0500 )edit

then a frozenGraph.pb is generated.

Here you can download my generated "graphDef.pb" "graphDef.pbtxt" and "frozenGraph.pb" from https://www.dropbox.com/sh/l48teumzewpece8/AAA8T2brnQBnanh_gGMkY4Fra?dl=0 and see if anyone can figure out any error from it...

Thanks

Paul Kuo gravatar imagePaul Kuo ( 2018-03-05 23:30:07 -0500 )edit

Hi @dkurt, is there any luck to fix this problem...?

Everyone, any suggestions will be appreciated...

Thank you

Paul Kuo gravatar imagePaul Kuo ( 2018-03-12 20:37:08 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2018-03-13 03:33:36 -0500

dkurt gravatar image

updated 2018-03-30 03:26:11 -0500

@Paul Kuo, the problem is that graph has been saved in training mode. I think there is some placeholder similar to isTraining that should be turned into False before graph saving (note that it's about graph but not about checkpoint).

image description

Moreover you may see some unusual transformations over input image.

image description

May I ask you to try to find that isTraining flag, set it to false and save the graph again by tf.train.write_graph. Then freeze the same checkpoint files with new version of graph. Our first goal is emit train/test switches from the graph. Thank you!


UPDATE

@Paul Kuo, the following are steps to create a graph without training-testing switches. An extra steps are required to import it into OpenCV.

Step 1: Create a graph definition

Make a script with the following code at the root folder of Age-Gender-Estimate-TF.

import tensorflow as tf
import inception_resnet_v1

inp = tf.placeholder(tf.float32, shape=[None, 160, 160, 3], name='input_image')
age_logits, gender_logits, _ = inception_resnet_v1.inference(inp, keep_probability=0.8,
                                                             phase_train=False,
                                                             weight_decay=1e-5)
print age_logits  # logits/age/BiasAdd
print gender_logits # logits/gender/BiasAdd

with tf.Session() as sess:
    graph_def = sess.graph.as_graph_def()
    tf.train.write_graph(graph_def, "", 'inception_resnet_v1.pb', as_text=False)

Here we create a graph definition for testing mode only (phase_train=False).

Step 2: Freeze resulting graph with checkpoint.

python ~/tensorflow/tensorflow/python/tools/freeze_graph.py \
    --input_graph=inception_resnet_v1.pb \
    --input_checkpoint=savedmodel.ckpt \
    --output_graph=frozen_inception_resnet_v1.pb \
    --output_node_names="logits/age/BiasAdd,logits/gender/BiasAdd" \
    --input_binary

Step 3: Checking

Using TensorBoard, check that our graph has no training subgraphs (compare with images above). C:\fakepath\Screenshot from 2018-03-30 10-41-03.png

Step 4: Help OpenCV to import graph.

Unfortunately, current version of OpenCV cannot interpret this graph correctly because of single Reshape layer that takes dynamically estimated target shape: C:\fakepath\Screenshot from 2018-03-30 10-30-43.png

Actually, it's just a flattening that means reshaping from 4-dimensional blob to 2-dimensional keeping the same batch size. We replace manage it during graph definition because it's out of user's code:

# inception_resnet_v1.py, line 262:
net = slim.fully_connected(net, bottleneck_layer_size, activation_fn=None,
                           scope='Bottleneck', reuse=False)

But we can help OpenCV to manage it by modifying a text graph.

Step 5: Create a text graph

import tensorflow as tf

# Read the graph.
with tf.gfile.FastGFile('frozen_inception_resnet_v1.pb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())

# Remove Const nodes.
for i in reversed(range(len(graph_def.node))):
    if graph_def.node[i].op == 'Const':
        del graph_def.node[i]
    for attr in ['T', 'data_format', 'Tshape', 'N', 'Tidx', 'Tdim',
                 'use_cudnn_on_gpu', 'Index', 'Tperm', 'is_training',
                 'Tpaddings']:
        if attr in graph_def.node[i].attr:
        del graph_def.node[i].attr[attr]

# Save as text.
tf.train.write_graph(graph_def, "", "frozen_inception_resnet_v1.pbtxt", as_text=True)

Step 6: Modify a text graph

Remove the Shape node

node {
  name: "InceptionResnetV1/Bottleneck/BatchNorm/Shape"
  op: "Shape"
  input: "InceptionResnetV1/Bottleneck/MatMul"
  attr {
    key: "out_type"
    value {
      type: DT_INT32
    }
  }
}

Replace Reshape to Identity:

from

node {
  name: "InceptionResnetV1/Bottleneck/BatchNorm/Reshape_1"
  op: "Reshape"
  input: "InceptionResnetV1/Bottleneck/BatchNorm/FusedBatchNorm"
  input: "InceptionResnetV1/Bottleneck ...
(more)
edit flag offensive delete link more

Comments

Hi @dkurt,

Thank you for your suggestion. I was switched to another project now, but will come back to this issue and try your suggestion probably next week. Will let you know what is going on here. Thank you~~

Paul

Paul Kuo gravatar imagePaul Kuo ( 2018-03-15 20:22:51 -0500 )edit

Hi @dkurt,

I have checked the cod and, to the best of my knowledge, there should not be any "isTraining==True" existed. Here are the code for generating graphDef.pb (before freezing it) https://www.dropbox.com/s/z7rzx4pmjl2vuux/convetGraphToPb.py?dl=0 and definition of inception_resent_v1.https://www.dropbox.com/s/ln71cijw2jo9yte/inception_resnet_v1.py?dl=0

Could you please look through it and help me to find out anything might result in this conversion error.

Thank you again~

Paul

Paul Kuo gravatar imagePaul Kuo ( 2018-03-18 22:50:09 -0500 )edit

@Paul Kuo, There are phase_train, is_training. They should be turned to False before the tf.train.write_graph.

dkurt gravatar imagedkurt ( 2018-03-19 03:13:24 -0500 )edit

yes, It has been done at line 76 convertGraphToPb.py

ages, genders = sess.run([age, gender], feed_dict={images_pl: aligned_imageX, train_mode: False})

Paul Kuo gravatar imagePaul Kuo ( 2018-03-28 20:29:37 -0500 )edit

@Paul Kuo, This flag is used to run the model but not define it. You have to pass is_training=False into inception_resnet_v1.

dkurt gravatar imagedkurt ( 2018-03-29 04:05:31 -0500 )edit

@dkurt,

after setting all "phase_train" and "is_training" flags equal to false, I ran tf.train.write_graph(...) and called "freeze_graph.py" to obtain the frozen graph (named "frozenGraph_ageGender.pb"). Then in VS2015, worte: cv::dnn::Net ageGenderNet; cv::String model = "D:\PaulProject\Deep Learning\Python_faceAgeGender\pbs\frozenGraph_ageGender.pb"; cv::String config = "D:\PaulProject\Deep Learning\Python_faceAgeGender\pbs\graphDef.pbtxt"; ageGenderNet = cv::dnn::readNetFromTensorflow(model/, config/);

and executed. Errors are still there.....

Paul Kuo gravatar imagePaul Kuo ( 2018-03-29 22:59:09 -0500 )edit

@dkurt

Upload the latest frozenGraph_agedGender.pbhttps://www.dropbox.com/s/h4vqsj4eu1ksmbm/frozenGraph_ageGender.pb?dl=0 and graphDef.pbtxt https://www.dropbox.com/s/9z890s854ngihyq/graphDef.pbtxt?dl=0

Please take a look at them to see if there are still any mistakes in them..

Thank you

Paul Kuo gravatar imagePaul Kuo ( 2018-03-29 23:01:41 -0500 )edit

@Paul Kuo, please read an updated answer. You may skip all the steps and just download the file by the following location: https://gist.github.com/dkurt/eca7ef0... . Use it as a second argument at readNetFromTensorFlow with frozenGraph_ageGender.pb.

dkurt gravatar imagedkurt ( 2018-03-30 03:32:26 -0500 )edit

@dkurt Thank you for your help. Just downloaded the "frozen_inception_resnet_v1.pbtxt" you provided and tried in my VS2015 environment. cv::dnn::Net ageGenderNet; cv::String model = "D:\PaulProject\Deep Learning\Python_faceAgeGender\pbs\frozenGraph_ageGender.pb"; cv::String config = "D:\PaulProject\Deep Learning\Python_faceAgeGender\pbs\dkurt\frozen_inception_resnet_v1.pbtxt"; ageGenderNet = cv::dnn::readNetFromTensorflow(model, config);

Unfortunately, I am still getting error for readNetFromTensorflow(...) >_<, but the error type changed and I show it below: OpenCV Error: Assertion failed(size==(int)dstBlob.total()) in cv::dnn::experimental_dnn_v3::'anonymous-namespace'::parseTensor, file c:\build\master_winpack-build-win64-vc14...\tf_importer.cpp, line179

Paul Kuo gravatar imagePaul Kuo ( 2018-03-30 22:38:31 -0500 )edit

@Paul Kuo, please update local OpenCV source code.

dkurt gravatar imagedkurt ( 2018-03-31 02:08:58 -0500 )edit
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2018-03-04 21:51:43 -0500

Seen: 470 times

Last updated: Mar 30