Multiple concurrent detections using a single net

edit

Hi, is it possible to load a single network via cv2.dnn.readNet(weights_file_abs_path, config_file_abs_path) and run multiple detections at one time?

I have a situation where I am implementing an ML API gateway that runs, YoloV3 for object detection. When single requests arrive, its fine, but the moment concurrent requests arrive, the detection gets corrupted.

The approach I have implemented is this: a) On gateway start, I create a global object that loads the weights. I do that for performance reasons, as loading is costly. b) When I get a new request, I launch a new thread for detection and pass it the global object

I had initially assumed that the network would be read only but later realized that during detection, we do things like:

self.net.setInput(blob)
outs = self.net.forward(self.get_output_layers())

Which likely changes the state of self.net

So is it at all possible to do concurrent processing with a single network? I'm trying to avoid multiple copies of the network as each one takes up a lot of memory.