Running forward() method from opencv dnn module gives wrong predictions

asked 2018-11-22 03:15:26 -0500

StefanCepa995 gravatar image

updated 2018-11-22 03:53:51 -0500

Hello everyone! I am trying to load pretrained CNN from .pb file and run it using OpenCV dnn module. The problem is, when I load it in Jupyter Notebook using Tensorflow, I get correct results. But, when I load it using OpenCV dnn module, specifically using readNetFromTensorflow function, I get terrible results. I have printed out the weights and I am guessing that for some reason OpenCV did not read the weights along the graph structure.

Here is my code for reading .pb file with Tensorflow (this works correctly):

def load_graph(frozen_graph_filename):

     with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
         graph_def = tf.GraphDef()
         graph_def.ParseFromString(f.read())

     with tf.Graph().as_default() as graph:
         tf.import_graph_def(graph_def, name="prefix")

     return graph

frozen_graph_filename = "C:/Users/stefan_cepa995/Desktop/Posao/Character_classify/training/freezed/model.pb"
graph = load_graph(frozen_graph_filename)

x = graph.get_tensor_by_name('prefix/conv_layer1_input:0')
y = graph.get_tensor_by_name('prefix/activation_11/Softmax:0')

def get_class_index(classes, cls):
    for i in range(0, len(classes)):
        if classes[i] == cls:
            return i

def get_class_name(classes, pred):
    for i in range(0, len(classes)):
        if i == pred:
            return classes[i]

check_for_time = []
for i in range(0, 17):
    check_for_time.append(concacanate_channels(validX[i]*1./255, second_channel, third_channel))
check_for_time = np.array(check_for_time)

print(check_for_time.max())
print(check_for_time.min())

with tf.Session(graph=graph) as sess:
    #%timeit y_out = sess.run(y, feed_dict={ x:  check_for_time}) #2.62ms GPU / 14.5ms CPU
    #%timeit y_out = sess.run(y, feed_dict={x : check_for_time[0:1]}) #1.45 GPU / 1.31ms CPU
    y_out = sess.run(y, feed_dict={ x:  check_for_time})

    for y in y_out:
        print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))

Now here is my code for reading it with OpenCV (produces incorrect predictions):

net = cv2.dnn.readNetFromTensorflow(frozen_graph_filename)

print(check_for_time.shape)

if check_for_time.shape == (17,70,20,3): 
    check_for_time= check_for_time.reshape(check_for_time.shape[0], 3, check_for_time.shape[1], check_for_time.shape[2])

net.setInput(check_for_time)
cvOut = net.forward('activation_11/Softmax')

#print(check_for_time.shape)
#print(check_for_time.reshape(check_for_time.shape[0], check_for_time.shape[2], check_for_time.shape[3], 3).shape)

prd = model.predict(check_for_time.reshape(check_for_time.shape[0], check_for_time.shape[2], check_for_time.shape[3], 3))
for y in prd:
        print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))
print('-'*20)
for y in cvOut:
        print('Predicted Class: ',get_class_name(le.classes_, np.argmax(y)))

I have printed out the weights of the first filter of the first convolution layer from both models (one loaded with Tensorflow and other loaded with OpenCV) and they seem different.

EDIT: Some screenshots of results produced by both models

Here are predicted results of the model I loaded using Tensorflow and model I loaded using OpenCV dnn module respectively:

Predicted Class:  S
Predicted Class:  7
Predicted Class:  D
Predicted Class:  8
Predicted Class:  H
Predicted Class:  7
Predicted Class:  4
Predicted Class:  Y
Predicted Class:  Z
Predicted Class:  D
Predicted Class:  V
Predicted Class:  W
Predicted Class:  4
Predicted Class:  Z
Predicted Class:  H
Predicted Class:  9
Predicted Class:  U
--------------------
Predicted Class:  M
Predicted Class:  1
Predicted Class:  1
Predicted Class:  1
Predicted Class:  M ...
(more)
edit retag flag offensive close merge delete

Comments

@StefanCepa995 , please be more certain. Put references to used models. Specify what means "correct" and what are "terrible results". Attach an image at least and expected prediction.

dkurt gravatar imagedkurt ( 2018-11-22 03:26:36 -0500 )edit

Thank you for quick reply. I will do that right away!

StefanCepa995 gravatar imageStefanCepa995 ( 2018-11-22 03:44:56 -0500 )edit

@StefanCepa995, How we can reproduce your issue without the model? Please upload it somewhere and put a downloadable link.

dkurt gravatar imagedkurt ( 2018-11-23 01:55:23 -0500 )edit
1

i see, that your preprocessing differs much, like you scale the 1st channel in the tf code, but not in the cv2 one. also BGR vs. RGB.

berak gravatar imageberak ( 2018-11-23 05:37:10 -0500 )edit

@dkurt Sadly I am not allowed to upload any of the data since this is a project from work. @berak both inputs to tf and to cv2 are scaled ( I double checked ). What about BGR vs. RGB? Thing that bugs me is that the weights loaded with OPENCV are not the same as the weights loaded with Tensorlfow.

StefanCepa995 gravatar imageStefanCepa995 ( 2018-11-23 06:12:31 -0500 )edit

what are model or load_graph or check_for_time ? how do you obtain the weights ?

berak gravatar imageberak ( 2018-11-23 06:50:09 -0500 )edit

@StefanCepa995, You model is a classification model. Try to replace your network to a public one and compare outputs from OpenCV and TensorFlow.

dkurt gravatar imagedkurt ( 2018-11-23 07:52:24 -0500 )edit

I found a script online which freezes model into .pb file. I created load_graph in order to read that .pb file and I tested it by making predictions and it predicted everything correctly. And I apologize, I forgot to mention that model is actually the model I have also loaded using keras load_model (from .h5 file). So I can compare results.

StefanCepa995 gravatar imageStefanCepa995 ( 2018-11-23 07:54:39 -0500 )edit