onnx model shows slower inference when setting cuda as backend

asked 2020-10-21 10:26:37 -0500

xz1990 gravatar image

I have a net loaded from onnx:

net = cv2.dnn.readNetFromONNX(xxx.onnx)

when i directly do net.forward(), the inference time is around 0.2s

If I set cuda as backend after loading model:

net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA) net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

The inference time will be around 0.4s

(But this backend setting is working for YOLOv3 readNetFromDarknet, I can see an inference time reduce after setting cuda as backend)

Is it because the model is not supporting cuda backend?

edit retag flag offensive close merge delete

Comments

the model is not supporting cuda backend?

it's for sure not about onnx, probably more like: some layers in your network have better or worse cuda support than others (or none at all)

yolo3 is a well tested network, while we don't know anything about yours.

berak gravatar imageberak ( 2020-10-21 10:46:24 -0500 )edit