Assertion Error with Custom Trained Caffe

edit

Way to reproduce

Get https://github.com/NVIDIA/caffe/blob/... and comment all the layers which have phase: TRAIN or stage: "val". Try to import using OpenCV (at least https://github.com/opencv/opencv/comm...).

Error message

Can't create layer "cluster" of type "Python" in function 'getLayerInstance'

Solution

Duplicate a .prototxt file to use it in OpenCV only. Replace the following layer:

layer {
    type: 'Python'
    name: 'cluster'
    bottom: 'coverage'
    bottom: 'bboxes'
    top: 'bbox-list'
    python_param {
        module: 'caffe.layers.detectnet.clustering'
        layer: 'ClusterDetections'
        param_str : '1248, 352, 16, 0.6, 3, 0.02, 22, 1'
    }
    include: { phase: TEST }
}

to

layer {
  name: 'cluster'
  type: 'ClusterDetections'
  bottom: 'coverage'
  bottom: 'bboxes'
  top: 'bbox-list'
  python_param {
    param_str: '1248, 352, 16, 0.6, 3, 0.02, 22, 1'
  }
}

Copy some usage from https://github.com/NVIDIA/caffe/blob/... to your script:

gridbox_to_boxes, vote_boxes, cluster methods without any changes

setup -> __init__ (modified)

reshape -> getMemoryShapes (modified)

forward -> forward (modified)

class ClusterDetections(object):
    def __init__(self, params, blobs):
        self.is_groundtruth = False
        try:
            plist = params['param_str'].split(',')
            self.image_size_x = int(plist[0])
            self.image_size_y = int(plist[1])
            self.stride = int(plist[2])
            self.gridbox_cvg_threshold = float(plist[3])
            self.gridbox_rect_thresh = int(plist[4])
            self.gridbox_rect_eps = float(plist[5])
            self.min_height = int(plist[6])
            self.num_classes = int(plist[7]) if len(plist) > 7 else 1
        except ValueError:
            raise ValueError("Parameter string missing or data type is wrong!")

    def getMemoryShapes(self, bottom):
        n_images = bottom[0][0]
        num_classes = bottom[0][1]
        if num_classes != self.num_classes:
            raise ValueError("Unexpected number of classes: %d != %d, bottom[0] shape=%s" % (num_classes, self.num_classes, repr(bottom[0].data.shape)))
        top = []
        for i in xrange(num_classes):
            top.append([1, n_images, MAX_BOXES, 5])  # Make it 4-dimensional
        return top

    def forward(self, bottom):
        top = []
        for i in xrange(self.num_classes):
            data0 = bottom[0][:,i:i+1,:,:]
            bbox = cluster(self, data0, bottom[1])
            top.append(np.expand_dims(bbox.astype(np.float32), 0))
        return top

cv.dnn_registerLayer('ClusterDetections', ClusterDetections)

TODO: I've tested that both OpenCV and Caffe returns similar values from coverage/sig and bbox/regressor layers using randomly initialized weights. I can test that detection post-processing also works good if you can share reference to trained .caffemodel because my returns just zeros.

P.S. Thanks to unsupported layer is the last you can just obtain both outputs and post-process them without a custom layer insertion.