Ask Your Question
2

Tensorflow Custom Model in OpenCV

asked 2017-12-10 02:23:17 -0500

chamath gravatar image

updated 2018-01-24 06:06:17 -0500

I have trained new model on top of ssd_mobilenet_v1_coco for a custom data set. This model works fine in tensorflow. But now I want to use this in OpenCV.

net = cv2.dnn.readNetFromTensorflow("model/frozen_inference_graph.pb", "model/protobuf.pbtxt")
detections = net.forward()

So for the config file I convert frozen_graph to pbtxt and add it. But then I got the following error [libprotobuf ERROR /home/chamath/Projects/opencv/opencv/3rdparty/protobuf/src/google/protobuf/text_format.cc:298] Error parsing text-format tensorflow.GraphDef: 731:5: Unknown enumeration value of "DT_RESOURCE" for field "type".

As suggested here I try to use this config file mentioned in the thread but when I use it object detection is not working properly. Incorrect number of squares detected and they are misplaced.

Is there any method to create a pbtxt config file that works with OpenCV? Or any suggestions how to make my model work in OpenCV?

Result in OpenCV :

image description

Result in Tensorflow

image description

edit retag flag offensive close merge delete

Comments

1

@chamath, what the difference between TensorFlow's MobileNet-SSD and yours? Number of classes is the same? Have you tried to run a sample https://github.com/opencv/opencv/blob... using your model? What input sizes did you use? Please share with us as much information as possible. At least an image with wrong predictions and code that draws them (if it isn't an OpenCV's sample code).

dkurt gravatar imagedkurt ( 2017-12-10 03:24:07 -0500 )edit

@dkurt, My opencv code is https://github.com/chamathabeysinghe/... . Number of classes are 1. I update the prototxt file's number of classes. I tested with the sample also. In video stream there is many squares as in the photo.

chamath gravatar imagechamath ( 2017-12-10 05:19:53 -0500 )edit

@chamath, thanks for the details! So you've changed attr { key: "num_classes" value { i: 91 } } to attr { key: "num_classes" value { i: 2 } }, right? Please try to replace cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5) to cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), (127.5, 127.5, 127.5)) because now it subtracts mean value only from blue channel.

dkurt gravatar imagedkurt ( 2017-12-10 08:30:59 -0500 )edit

@dkurt, Yes I have changed num_classes. I made your change to image blob but still the issue is there.

chamath gravatar imagechamath ( 2017-12-10 08:37:21 -0500 )edit

If no more changes were made for model training (TensorFlow's config file), we need to know how you received a target image. Please share a code with TensorFlow model invocation.

dkurt gravatar imagedkurt ( 2017-12-10 09:12:44 -0500 )edit

Tensorflow code used for my model, https://gist.github.com/chamathabeysi.... This is exact copy from google object detection api https://github.com/tensorflow/models/... without model download part

chamath gravatar imagechamath ( 2017-12-10 09:30:02 -0500 )edit

@chamath, For training you've taken a config file, https://github.com/tensorflow/models/... and replaced only num_classes: 1? Can you share a model?

dkurt gravatar imagedkurt ( 2017-12-10 09:50:46 -0500 )edit

Yes for training replaced num_classes: 1 and PATH_TO_BE_CONFIGURED in the mentioned config file. Model is available in https://drive.google.com/file/d/1u_Hi... . My test images are available on https://github.com/chamathabeysinghe/...

chamath gravatar imagechamath ( 2017-12-10 10:05:29 -0500 )edit

@dkurt, any suggestions to solve this?

chamath gravatar imagechamath ( 2017-12-11 07:09:00 -0500 )edit

Any updates on this? Did you manage to make it work @chamath ?

I have the same problem, and from the issue-posts on github and here, I think the problem might be because of a mismatch between the config file used for retraining in the Object Detection API and the pbtxt-file used by Opencv.

XenonHawk gravatar imageXenonHawk ( 2018-01-12 08:59:14 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
4

answered 2018-01-24 05:14:34 -0500

dkurt gravatar image

@chamath, thank you for pointing an issue! The problem was in prior boxes generation (anchors). Let's consider MultipleGridAnchorGenerator/concat_5 layer of MobileNet-SSD named as ssd_mobilenet_v1_coco_11_06_2017 (some old version) output is

[0.02500001 0.02500001 0.97500002 0.97500002]  // min size
[0.01266029 0.01266029 0.98733974 0.98733974]  // max size
[0.16412428 -0.17175144 0.83587575 1.1717515 ] // aspect ratio 2.0
[-0.17175144 0.16412428 1.1717515  0.83587575] // aspect ratio 0.5
[0.22575861 -0.3227241  0.77424139 1.3227241 ] // aspect ratio 3.0
[-0.32276523 0.22577232 1.32276523 0.77422768] // aspect ratio 0.333

OpenCV produces the following proposals at the corresponding layer:

[0.025      0.025      0.97500002 0.97500002]  // min size
[0.01266027 0.01266027 0.98733968 0.98733968]  // max size
[-0.17175145 0.16412427 1.1717515  0.83587575] // aspect ratio 2.0
[0.16412427 -0.17175145 0.83587575 1.1717515 ] // aspect ratio 0.5
[-0.3227241  0.22575861 1.3227241  0.77424139] // aspect ratio 3.0
[0.22575861 -0.32272416 0.77424139 1.32272422] // aspect ratio 0.333

Note that TensorFlow produces it in [ymin, xmin, ymax, xmax] order but OpenCV in [xmin, ymin, xmax, ymax]. OpenCV can manage it so it's OK.

OpenCV's PriorBox layer is based on origin Caffe-SSD framework (https://github.com/weiliu89/caffe/blo...). It produces anchors for min_size specified. Then for max_size and all the aspect ratios. It looks like TensorFlow followed this rule before some point.

Let's consider the same layers for your model. Perhaps, it was received from newer TensorFlow, right?

[0.02500001 0.02500001 0.97500002 0.97500002]  // min size
[0.16412428 -0.17175144 0.83587575 1.1717515 ] // aspect ratio 2.0
[-0.17175144 0.16412428 1.1717515  0.83587575] // aspect ratio 0.5
[0.22575861 -0.3227241  0.77424139 1.3227241 ] // aspect ratio 3.0
[-0.32276523 0.22577232 1.32276523 0.77422768] // aspect ratio 0.333
[0.01266029 0.01266029 0.98733974 0.98733974]  // max size

As you may see, anchor related to the max_size hyper-parameter moved to the bottom of the list. In short words, OpenCV for the same order of anchors (min-max-ratios) predicts deltas for an updated order (min-ratios-max).

To make it works properly please apply changes from PR https://github.com/opencv/opencv/pull... and use the following text graph: https://gist.github.com/dkurt/fb324b4...

Sample:

import numpy as np
import cv2 as cv

cvNet = cv.dnn.readNetFromTensorflow('hat_model/frozen_inference_graph.pb', 'ssd_mobilenet_v1_coco_hat.pbtxt')

img = cv.imread('/home/dkurtaev/Pictures/image5.jpg')
cvNet.setInput(cv.dnn.blobFromImage(img, 1.0/127.5, (300, 300), (127.5, 127.5, 127.5), swapRB=True, crop=False))
cvOut = cvNet.forward()

for detection in cvOut[0,0,:,:]:
    score = float(detection[2])
    if score > 0.5:
        left = detection[3] * img.shape[1]
        top = detection[4] * img.shape[0]
        right = detection[5] * img.shape[1]
        bottom = detection[6] * img.shape[0]
        cv.rectangle(img, (int(left), int(top)), (int(right), int(bottom)), (0, 255, 0))

cv.imshow('img', img)
cv.waitKey ...
(more)
edit flag offensive delete link more

Comments

1

Wow! Thank you! Thats really tricky

fastdeath gravatar imagefastdeath ( 2018-01-25 00:07:22 -0500 )edit

Hey @dkurt, how did you get this 'ssd_mobilenet_v1_coco_hat.pbtxt'? I have trained a mdel for a single class and I have something in very different format. Like the one that we have in object_detection catalogue..

item { id: 1 name: 'ball' }

What should I do so that I can use it for direct OpenCV use the way you did in the code above?

Thanks..

Prasanna Routray gravatar imagePrasanna Routray ( 2018-09-04 06:58:42 -0500 )edit

@Prasanna Routray, please follow a guide.

dkurt gravatar imagedkurt ( 2018-09-04 07:00:51 -0500 )edit

Hello @dkurt, I followed the guide and now able to run the python code.

Is there any repo or example for the same thing that can be done with c++?

That would help in production.

Thanks...

Prasanna Routray gravatar imagePrasanna Routray ( 2018-09-05 00:05:06 -0500 )edit

@Prasanna Routray, do you mean .pbtxt generation or a sample application?

dkurt gravatar imagedkurt ( 2018-09-05 04:22:55 -0500 )edit

@dkurt I want a sample application so that I can use the model .pb file and configuration .pbtxt file to test.

I hope you understood my concern.

Thanks...

Prasanna Routray gravatar imagePrasanna Routray ( 2018-09-05 04:28:40 -0500 )edit
dkurt gravatar imagedkurt ( 2018-09-05 07:18:59 -0500 )edit

@dkurt, actually I have tried to implement. I'm facing some unknown error about which I have no idea.

modified object_detection.cpp. https://drive.google.com/open?id=1eBz... the .pb model. https://drive.google.com/open?id=1QKx... the .pbtxt config file. https://drive.google.com/open?id=1sUd... test image is here https://drive.google.com/open?id=1lw3... python file that is working is here https://drive.google.com/open?id=1CT7...

The following is the error that I'm getting while doing 'make' " https://drive.google.com/open?id=1Oks... " here is the CMakeLists.txt that https://drive.google.com/open?id=1T4z...

Prasanna Routray gravatar imagePrasanna Routray ( 2018-09-05 07:38:47 -0500 )edit

Hello @dkurt and all others , do you have any answer to my problem?

Thanks..

Prasanna Routray gravatar imagePrasanna Routray ( 2018-09-08 05:57:57 -0500 )edit

@Prasanna Routray how did you manage to solve this problem?

ajinkyabobade93@gmail.com gravatar imageajinkyabobade93@gmail.com ( 2019-05-20 04:49:12 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-12-10 02:23:17 -0500

Seen: 2,467 times

Last updated: Jan 24 '18