Ask Your Question
4

application stopped working with caffe network dnn module, forward()

asked 2018-04-03 10:02:40 -0500

aguila gravatar image

Hello,

i am implementing a facedetector in opencv, Single Stage Headless detector SSH using the pretrained model provided, modifying the .prototxt file to be compatible without python layers accordingly to the example described in faster_rcnn_opencv which would be something like the following for each of the python proposal layers:

layer {
name: 'm2@ssh_proposal'
type: 'Proposal'
bottom: 'm2@ssh_cls_prob_reshape_output'
bottom: 'm2@ssh_bbox_pred_output'
bottom: 'im_info'
top: 'm2@ssh_boxes'
top: 'm2@ssh_cls_prob'
proposal_param {
feat_stride: 16
scale: 4
scale: 8
ratio: 1
}
}

and my code looks like:

String model = "SSH.caffemodel";
String proto = "SSH.prototxt";
Net SSH_net = readNetFromCaffe(proto, model);

Mat img = imread("test.jpg");
resize(img, img, Size(224, 224));
img = img - Scalar(102.9801, 115.9465, 122.7717);

Mat inputBlob = blobFromImage(img, 1.0f, Size(), Scalar(), true, false);
Mat imInfo = (Mat_<double>(1, 3) << 224, 224, 1);

SSH_net.setInput(inputBlob, "data");
SSH_net.setInput(imInfo, "im_info");

SSH_net.forward();

The network is being created succesfully, but at the moment of forwarding the network it just crashes.. I havent found any workaround for this, does anyone has any suggestions or ideas? or have been facing this issue? Any help is appreciated! thank you!

edit retag flag offensive close merge delete

Comments

@aguila, Looks like OpenCV's Proposal layer produces just rois but an origin layer optionally produces scores too.

dkurt gravatar imagedkurt ( 2018-04-04 00:44:48 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
4

answered 2018-04-04 07:11:39 -0500

dkurt gravatar image

updated 2018-04-04 07:13:33 -0500

@aguila, Thank you! Some bugs in proposal layer have been fixed. To make this model work correctly you need to apply changes from a PR https://github.com/opencv/opencv/pull...

As you mentioned, we need to modify .prototxt in the following way:

layer {
  name: 'm3@ssh_proposal'
  # type: 'Python'
  type: 'Proposal'
  bottom: 'm3@ssh_cls_prob_reshape_output'
  bottom: 'm3@ssh_bbox_pred_output'
  bottom: 'im_info'
  top: 'm3@ssh_boxes'
  top: 'm3@ssh_cls_prob'
  # python_param {
  #   module: 'SSH.layers.proposal_layer'
  #   layer: 'ProposalLayer'
  #   param_str: "{'feat_stride': 32,'scales': [16,32], 'ratios':[1,]}"
  proposal_param {
    feat_stride: 32
    scale: 16
    scale: 32
    ratio: 1.0
    nms_thresh: 1.0
  }
}

...

layer {
  name: 'm2@ssh_proposal'
  # type: 'Python'
  type: 'Proposal'
  bottom: 'm2@ssh_cls_prob_reshape_output'
  bottom: 'm2@ssh_bbox_pred_output'
  bottom: 'im_info'
  top: 'm2@ssh_boxes'
  top: 'm2@ssh_cls_prob'
  # python_param {
  #   module: 'SSH.layers.proposal_layer'
  #   layer: 'ProposalLayer'
  #   param_str: "{'feat_stride': 16,'scales': [4,8], 'ratios':[1,]}"
  proposal_param {
    feat_stride: 16
    scale: 4
    scale: 8
    ratio: 1.0
    nms_thresh: 1.0
 }
}

...

layer {
  name: 'm1@ssh_proposal'
  # type: 'Python'
  type: 'Proposal'
  bottom: 'm1@ssh_cls_prob_reshape_output'
  bottom: 'm1@ssh_bbox_pred_output'
  bottom: 'im_info'
  top: 'm1@ssh_boxes'
  top: 'm1@ssh_cls_prob'
  # python_param {
  #   module: 'SSH.layers.proposal_layer'
  #   layer: 'ProposalLayer'
  #   param_str: "{'feat_stride': 8,'scales': [1,2], 'ratios':[1,]}"
  proposal_param {
    feat_stride: 8
    scale: 1
    scale: 2
    ratio: 1.0
    nms_thresh: 1.0
  }
}

The most of arguments are mapped correspondingly but nms_thresh: 1.0is used to disable NMS postprocessing as model's authors did (see https://github.com/mahyarnajibi/SSH/b...)

NOTE: without non-maximum suppression these layers just produce all the predicted bounding boxes (no more than 300 by default). They are sorted by confidence and almost all of them have a quite small confidence close to zero. However if you know that your image has more than 300 faces (that's hardly not possible), add an extra parameter post_nms_topn which equals 300 by default. In example, post_nms_topn: 1000.

To get bounding boxes and scores prediction, run OpenCV:

import cv2 as cv
import numpy as np

imInfo = np.array([224, 224, 1.6], dtype=np.float32)

CONF_THRESH = 0.5
NMS_THRESH = 0.45

net = cv.dnn.readNet('test_ssh.prototxt', 'SSH.caffemodel')

img = cv.imread('/path/to/image')

imgHeight = img.shape[0]
imgWidth = img.shape[1]

inp = cv.dnn.blobFromImage(img, 1.0, (224, 224), (102.9801, 115.9465, 122.7717), False, False)

net.setInput(inp)
net.setInput(imInfo, 'im_info')

outs = net.forward(['ssh_cls_prob', 'ssh_boxes'])

Then you need to apply NMS to filter predicted faces:

scores = outs[0].flatten().tolist()
boxes = []
for box in outs[1]:
    left = int(box[1] / 224 * imgWidth)
    top = int(box[2] / 224 * imgHeight)
    right = int(box[3] / 224 * imgWidth)
    bottom = int(box[4] / 224 * imgHeight)
    boxes.append([left, top, right - left + 1, bottom - top + 1])

ids = cv.dnn.NMSBoxes(boxes, scores, CONF_THRESH, NMS_THRESH)

for idx in ids:
    idx = idx[0]
    score = scores[idx]
    box = boxes[idx]

    left = box[0]
    top = box[1]
    right = left + box[2] - 1
    bottom = top + box[3] - 1
    cv.rectangle(img, (left, top), (right, bottom), (0, 255, 0))

cv.imshow('ssh', img)
cv.waitKey()
edit flag offensive delete link more

Comments

@dkurt woow thank you for the very fast answer and solution, i just compiled with your PR and it works in both C++ and python interface.. Thanks again! cheers!

aguila gravatar imageaguila ( 2018-04-04 11:02:20 -0500 )edit

@aguila, you're welcome!

dkurt gravatar imagedkurt ( 2018-04-04 12:07:30 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-04-03 10:02:40 -0500

Seen: 487 times

Last updated: Apr 04 '18