Hello everyone! I am trying to load pretrained CNN from .pb file and run it using OpenCV dnn module. The problem is, when I load it in Jupyter Notebook using Tensorflow, I get correct results. But, when I load it using OpenCV dnn module, specifically using readNetFromTensorflow function, I get terrible results. I have printed out the weights and I am guessing that for some reason OpenCV did not read the weights along the graph structure.
Here is my code for reading .pb file with Tensorflow (this works correctly):
def load_graph(frozen_graph_filename):
with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
with tf.Graph().as_default() as graph:
tf.import_graph_def(graph_def, name="prefix")
return graph
frozen_graph_filename = "C:/Users/stefan_cepa995/Desktop/Posao/Character_classify/training/freezed/model.pb"
graph = load_graph(frozen_graph_filename)
x = graph.get_tensor_by_name('prefix/conv_layer1_input:0')
y = graph.get_tensor_by_name('prefix/activation_11/Softmax:0')
def get_class_index(classes, cls):
for i in range(0, len(classes)):
if classes[i] == cls:
return i
def get_class_name(classes, pred):
for i in range(0, len(classes)):
if i == pred:
return classes[i]
check_for_time = []
for i in range(0, 17):
check_for_time.append(concacanate_channels(validX[i]*1./255, second_channel, third_channel))
check_for_time = np.array(check_for_time)
print(check_for_time.max())
print(check_for_time.min())
with tf.Session(graph=graph) as sess:
#%timeit y_out = sess.run(y, feed_dict={ x: check_for_time}) #2.62ms GPU / 14.5ms CPU
#%timeit y_out = sess.run(y, feed_dict={x : check_for_time[0:1]}) #1.45 GPU / 1.31ms CPU
y_out = sess.run(y, feed_dict={ x: check_for_time})
for y in y_out:
print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))
Now here is my code for reading it with OpenCV (produces incorrect predictions):
net = cv2.dnn.readNetFromTensorflow(frozen_graph_filename)
print(check_for_time.shape)
if check_for_time.shape == (17,70,20,3):
check_for_time= check_for_time.reshape(check_for_time.shape[0], 3, check_for_time.shape[1], check_for_time.shape[2])
net.setInput(check_for_time)
cvOut = net.forward('activation_11/Softmax')
#print(check_for_time.shape)
#print(check_for_time.reshape(check_for_time.shape[0], check_for_time.shape[2], check_for_time.shape[3], 3).shape)
prd = model.predict(check_for_time.reshape(check_for_time.shape[0], check_for_time.shape[2], check_for_time.shape[3], 3))
for y in prd:
print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))
print('-'*20)
for y in cvOut:
print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))
I have printed out the weights of the first filter of the first convolution layer from both models (one loaded with Tensorflow and other loaded with OpenCV) and they seem different.
EDIT: Some screenshots of results produced by both models
Here are predicted results of the model I loaded using Tensorflow and model I loaded using OpenCV dnn module respectively:
Predicted Class: S
Predicted Class: 7
Predicted Class: D
Predicted Class: 8
Predicted Class: H
Predicted Class: 7
Predicted Class: 4
Predicted Class: Y
Predicted Class: Z
Predicted Class: D
Predicted Class: V
Predicted Class: W
Predicted Class: 4
Predicted Class: Z
Predicted Class: H
Predicted Class: 9
Predicted Class: U
--------------------
Predicted Class: M
Predicted Class: 1
Predicted Class: 1
Predicted Class: 1
Predicted Class: M
Predicted Class: 1
Predicted Class: M
Predicted Class: M
Predicted Class: 1
Predicted Class: 1
Predicted Class: 1
Predicted Class: 1
Predicted Class: M
Predicted Class: 1
Predicted Class: M
Predicted Class: 1
Predicted Class: M
The first set of results (from model loaded using load_graph function I have presented above) are CORRECT results. Second set of results are from model loaded using readNetFromTesnroflow from OpenCV dnn modules, and they are INCORRECT.
Here I have executed following code to print out 1st filter of 1st Conv layer from model loaded with Tensorflow and from model loaded with OpenCV dnn module. Note that I had to reshape weights of model loaded with Tensorflow since I saw in source code of dnn module that they are reshaping everything to CHANNEL FIRST.
l = net.getLayer(net.getLayerId('conv_layer1/convolution'))
print(l.blobs[0].shape)
print(l.blobs[0])
print('-'*20)
print(model.get_weights()[0].reshape(20,3,5,5).shape)
print(model.get_weights()[0].reshape(20,3,5,5))
Here are the weights I get:
From model loaded with OpenCV dnn module function
[[[-1.91858143e-01 5.37612801e-03 1.16247185e-01 2.11091250e-01
5.12935668e-02]
[-1.02800028e-02 1.90526575e-01 2.78309911e-01 2.69920677e-01
1.62888378e-01]
[ 2.46958132e-03 1.86067119e-01 2.89623946e-01 3.86989266e-01
1.14388265e-01]
[ 3.42311971e-02 2.37777263e-01 3.01134527e-01 3.13065022e-01
7.70063773e-02]
[-5.86964786e-02 5.33587627e-05 7.53456801e-02 1.43435672e-01
-1.82987303e-02]]
[[-3.54021527e-02 -3.98019655e-03 -6.25648499e-02 -3.95921804e-02
-3.54411826e-02]
[-2.52517611e-02 -1.00688534e-02 -1.06774963e-01 -3.97736989e-02
-2.27216780e-02]
[-1.17921144e-01 -9.06459391e-02 -1.03303224e-01 1.06597319e-02
-3.78685035e-02]
[ 3.19255255e-02 -6.02772981e-02 -1.45394012e-01 -1.48354396e-01
-6.23231754e-02]
[-2.63450723e-02 1.22204535e-02 -1.41175300e-01 -1.38809040e-01
-1.01256676e-01]]
[[ 7.23090693e-02 6.18805513e-02 -3.11869550e-02 9.14030969e-02
-5.62425032e-02]
[ 6.03295714e-02 3.88031416e-02 3.13969776e-02 3.22816917e-03
8.39628875e-02]
[ 6.83484226e-02 -9.79482308e-02 -8.48144814e-02 -2.75416933e-02
-2.13653427e-02]
[-5.39059192e-02 -5.67030981e-02 -1.71107017e-02 3.95301804e-02
-2.77140923e-02]
[-3.17484997e-02 -1.62207633e-02 -8.32035467e-02 2.62847869e-03
-4.42562327e-02]]]
From model loaded with Tensorflow
[[[-0.19185814 -0.11371256 0.02440604 -0.03983486 0.18133903]
[-0.06918904 -0.02724461 -0.03240528 -0.05901188 -0.1054622 ]
[-0.07114249 0.07545941 0.03924959 -0.0195534 -0.09568904]
[ 0.19387466 0.00156229 -0.1630032 0.02165342 0.0633769 ]
[-0.03540215 -0.02772055 0.03354672 -0.08050121 0.03763024]]
[[-0.0671616 0.09309757 0.01437161 0.00547361 0.04276887]
[-0.08936539 -0.10751003 -0.06520302 0.05693933 0.01986157]
[ 0.12102143 0.06037022 -0.0565586 -0.02597986 -0.11647336]
[ 0.07230907 0.08715077 -0.05727058 -0.03701202 0.07409618]
[ 0.00787397 0.0675556 -0.02530471 0.10071266 0.039113 ]]
[[ 0.00779035 0.04990828 -0.01289141 -0.0176944 0.02423616]
[-0.04980068 0.11551064 0.08677906 -0.10342686 -0.00582745]
[ 0.00537613 0.11585362 -0.01451444 0.0757063 0.02271813]
[ 0.04121283 -0.1136414 0.08113346 0.12850036 0.21825306]
[-0.04748515 0.1537471 0.04467716 -0.01889283 -0.04135783]]]
Does anyone have any idea what the issue might be?
