onnx model shows slower inference when setting cuda as backend
I have a net loaded from onnx:
net = cv2.dnn.readNetFromONNX(xxx.onnx)
when i directly do net.forward(), the inference time is around 0.2s
If I set cuda as backend after loading model:
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA) net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
The inference time will be around 0.4s
(But this backend setting is working for YOLOv3 readNetFromDarknet, I can see an inference time reduce after setting cuda as backend)
Is it because the model is not supporting cuda backend?
it's for sure not about onnx, probably more like: some layers in your network have better or worse cuda support than others (or none at all)
yolo3 is a well tested network, while we don't know anything about yours.