Ask Your Question
0

keras to tensorflow to opencv dnn

asked 2019-01-06 05:51:06 -0600

Shay Weissman gravatar image

updated 2019-01-06 05:57:59 -0600

berak gravatar image
System information (version)
  • OpenCV => :grey_question:
  • Operating System / Platform => :grey_question:
  • Compiler => :grey_question:
Detailed description

I have a lenet keras model that was trained for car color recognition. I converted it to tensorflow pb file using keras_to_tensorflow from https://github.com/amir-abdi/keras_to... I created pbtxt with the keras_to_tensorflow. I tried loading it (cvInference.py)

cvNet = cv.dnn.readNetFromTensorflow('d:\\tfs\\LPR\\IP\\MAIN\\SRC\\PythonProjects\\Keras\\lenetcarColor.pb',
                                     'd:\\tfs\\LPR\\IP\\MAIN\\SRC\\PythonProjects\\Keras\\lenetcarColor.pbtxt')

I got the error:

cv2.error: OpenCV(3.4.3) C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_importer.cpp:614: error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function 'cv::dnn::experimental_dnn_34_v7::`anonymous-namespace'::addConstNodes'
Steps to reproduce

All my files are at https://www.dropbox.com/sh/yi4b529v01... including the train images in a zip file

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2019-01-06 10:03:04 -0600

berak gravatar image

updated 2019-01-06 11:02:49 -0600

your .pbtxt file isn't useful "as is", you have to edit it a bit:

# Read the graph.
with tf.gfile.FastGFile('lenet.pb', 'rb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())

with tf.Session() as sess:
    # Restore session
    sess.graph.as_default()
    tf.import_graph_def(graph_def, name='')

# Strip Const nodes.
for i in reversed(range(len(graph_def.node))):
    if graph_def.node[i].op == 'Const':
        del graph_def.node[i]
    for attr in ['T', 'data_format', 'Tshape', 'N', 'Tidx', 'Tdim',
                  'use_cudnn_on_gpu', 'Index', 'Tperm', 'is_training',
                  'Tpaddings']:
        if attr in graph_def.node[i].attr:
             del graph_def.node[i].attr[attr]

# Save stripped model.
tf.train.write_graph(graph_def, "", 'lenet2.pbtxt', as_text=True)

then, you have to manually replace all flatten nodes with a single Flatten one,and rewire the inputs. the end of your .pbtxt should look like this:

node {
  name: "max_pooling2d_1/MaxPool"
  op: "MaxPool"
  input: "activation_1/Relu"
  attr {
    key: "ksize"
    value {
      list {
        i: 1
        i: 2
        i: 2
        i: 1
      }
    }
  }
  attr {
    key: "padding"
    value {
      s: "VALID"
    }
  }
  attr {
    key: "strides"
    value {
      list {
        i: 1
        i: 2
        i: 2
        i: 1
      }
    }
  }
}
node {
  name: "flatten/Shape"
  op: "Flatten"
  input: "max_pooling2d_1/MaxPool"
  attr {
    key: "out_type"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "dense/kernel/read"
  op: "Identity"
  input: "dense/kernel"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@dense/kernel"
      }
    }
  }
}
node {
  name: "dense/bias/read"
  op: "Identity"
  input: "dense/bias"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@dense/bias"
      }
    }
  }
}
node {
  name: "dense/MatMul"
  op: "MatMul"
  input: "flatten/Shape"
  input: "dense/kernel/read"
  attr {
    key: "transpose_a"
    value {
      b: false
    }
  }
  attr {
    key: "transpose_b"
    value {
      b: false
    }
  }
}
node {
  name: "dense/BiasAdd"
  op: "BiasAdd"
  input: "dense/MatMul"
  input: "dense/bias/read"
}
node {
  name: "activation_2/Relu"
  op: "Relu"
  input: "dense/BiasAdd"
}
node {
  name: "dense_1/kernel/read"
  op: "Identity"
  input: "dense_1/kernel"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@dense_1/kernel"
      }
    }
  }
}
node {
  name: "dense_1/bias/read"
  op: "Identity"
  input: "dense_1/bias"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@dense_1/bias"
      }
    }
  }
}
node {
  name: "dense_1/MatMul"
  op: "MatMul"
  input: "activation_2/Relu"
  input: "dense_1/kernel/read"
  attr {
    key: "transpose_a"
    value {
      b: false
    }
  }
  attr {
    key: "transpose_b"
    value {
      b: false
    }
  }
}
node {
  name: "dense_1/BiasAdd"
  op: "BiasAdd"
  input: "dense_1/MatMul"
  input: "dense_1/bias/read"
}
node {
  name: "activation_3/Softmax"
  op: "Softmax"
  input: "dense_1/BiasAdd"
}

(the whole file can be found on a gist)

then we can run it from opencv:

import cv2 as cv

cvNet = cv.dnn.readNetFromTensorflow('lenet.pb', 'lenet2.pbtxt')

img = cv.imread('blue_car.jpg')
cvNet.setInput(cv.dnn.blobFromImage(img, size=(40, 40), swapRB=False, crop=False))
cvOut = cvNet.forward()    

cvOut = cvOut.flatten()
classId = np.argmax(cvOut)
confidence = cvOut[classId]
labels = ["black", "blue", "gray", "green", "red", "white", "yellow"]
label = labels[classId]

cv.putText(img, label, (100, 130), cv.FONT_HERSHEY_SIMPLEX,  3.7, (0, 0, 0), 5)

import matplotlib.pyplot as plt
orig = cv.cvtColor(img, cv.COLOR_BGR2RGB)
plt.imshow(orig)

hey, it got it wrong ;)

please also have a look at the answer here , where most of this came from.

edit flag offensive delete link more

Comments

1

Many thanks. It runs. The images I use are not full car images but roi around the front license plate. They exist at the the dropbox link as zip file. The results I get from the model running with keras are different from running with opencv. I am trying to figure out why now.

Shay Weissman gravatar imageShay Weissman ( 2019-01-07 03:16:05 -0600 )edit

I fixed it but still get some difference between tensorflow and cv inference. With opencv the confidence I get is always 1.0 (100%) I attached to the dropbox link: cvInference.py is the file I use for inference. (I used your pbtxt) test_car_color.py is the tensorflow/keras inference. resultOpencv and resultTensorflow are directories of images with classification result on the green cars. Thanks.

Shay Weissman gravatar imageShay Weissman ( 2019-01-07 07:14:40 -0600 )edit

mayve take a closer look at your processing. e.g. your test_car_color.py scales the images by 1/255, the cv one doesn't. also BGR vs RGB (i have no idea, what you used from keras). stuff like that.

berak gravatar imageberak ( 2019-01-07 07:37:18 -0600 )edit

thanks again for the quick response. There really was RGB BGR issue all greens became yello. This was fixed before my last comment. Do I have to normalize my image before inference? I will try a tip on how would be great. thanks.

Shay Weissman gravatar imageShay Weissman ( 2019-01-07 07:50:10 -0600 )edit

if you use 1/255 in keras, you will need scale=1.0/255 in blobFromImage(), too, i guess.

berak gravatar imageberak ( 2019-01-07 07:51:44 -0600 )edit

I scaled the image to 1.0/255 but still results are different from the keras inference and still confidence i always 100.

Shay Weissman gravatar imageShay Weissman ( 2019-01-21 01:17:57 -0600 )edit

Question Tools

1 follower

Stats

Asked: 2019-01-06 05:51:06 -0600

Seen: 3,821 times

Last updated: Jan 06 '19