Revision history [back]

OpenCV webcam stream slows down alongside caffe prediction

I'm attempting use caffe and python to do real-time image classification. I'm using OpenCV to stream from my webcam in one process, and in a separate process, using caffe to perform image classification on the frames pulled from the webcam. Then I'm passing the result of the classification back to the main thread to caption the webcam stream.

The problem is that even though I have an NVIDIA GPU and am performing the caffe predictions on the GPU, the main thread gets slown down. Normally without doing any predictions, my webcam stream runs at 30 fps; however, with the predictions, my webcam stream gets at best 15 fps.

Even when I run the two components are separate python programs (i.e. pull frames from the webcam in one script and run a separate script doing caffe predictions in an infinite loop) I still get a slowdown in OpenCV's ability to grab webcam frames. I've run the code in C++ with multithreading and experienced the exact same result.

I've verified that caffe is indeed using the GPU when performing the predictions, and that my GPU or GPU memory is not maxing out. I've also verified that my CPU cores are not getting maxed out at any point during the program. I'm wondering if I am doing something wrong or if there is no way to keep these 2 processes truly separate. Any advice is appreciated. Here is my code for reference

class Consumer(multiprocessing.Process):

def __init__(self, task_queue, result_queue):
multiprocessing.Process.__init__(self)
self.result_queue = result_queue
#other initialization stuff

def run(self):
caffe.set_mode_gpu()
caffe.set_device(0)
#Load caffe net -- code omitted
while True:
#crop image -- code omitted
text = net.predict(image)
self.result_queue.put(text)

return

import cv2
import caffe
import multiprocessing
import Queue

results = multiprocessing.Queue()
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
rval, frame = vc.read()
else:
rval = False
frame_copy[:] = frame
while rval:
if not results.empty():
text = results.get()
#Add text to frame
cv2.putText(frame,text)

#Showing the frame with all the applied modifications
cv2.imshow("preview", frame)

#Getting next frame from camera
rval, frame = vc.read()
frame_copy[:] = frame
#Getting keyboard input
key = cv2.waitKey(1)
#exit on ESC
if key == 27:
break


OpenCV webcam stream slows down alongside caffe prediction

I'm attempting use caffe and python to do real-time image classification. I'm using OpenCV to stream from my webcam in one process, and in a separate process, using caffe to perform image classification on the frames pulled from the webcam. Then I'm passing the result of the classification back to the main thread to caption the webcam stream.

The problem is that even though I have an NVIDIA GPU and am performing the caffe predictions on the GPU, the main thread gets slown down. Normally without doing any predictions, my webcam stream runs at 30 fps; however, with the predictions, my webcam stream gets at best 15 fps.

Even when I run the two components are separate python programs (i.e. pull frames from the webcam in one script and run a separate script doing caffe predictions in an infinite loop) I still get a slowdown in OpenCV's ability to grab webcam frames. I've run the code in C++ with multithreading and experienced the exact same result.

I've verified that caffe is indeed using the GPU when performing the predictions, and that my GPU or GPU memory is not maxing out. I've also verified that my CPU cores are not getting maxed out at any point during the program. I'm wondering if I am doing something wrong or if there is no way to keep these 2 processes truly separate. Any advice is appreciated. Here is my code for reference

class Consumer(multiprocessing.Process):

def __init__(self, task_queue, result_queue):
multiprocessing.Process.__init__(self)
self.result_queue = result_queue
#other initialization stuff

def run(self):
caffe.set_mode_gpu()
caffe.set_device(0)
#Load caffe net -- code omitted
while True:
#crop image -- code omitted
text = net.predict(image)
self.result_queue.put(text)

return

import cv2
import caffe
import multiprocessing
import Queue

results = multiprocessing.Queue()
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
rval, frame = vc.read()
else:
rval = False
frame_copy[:] = frame
while rval:
if not results.empty():
text = results.get()
#Add text to frame
cv2.putText(frame,text)

#Showing the frame with all the applied modifications
cv2.imshow("preview", frame)

#Getting next frame from camera
rval, frame = vc.read()
frame_copy[:] = frame
#Getting keyboard input
key = cv2.waitKey(1)
#exit on ESC
if key == 27:
break


I've tried testing the code skeleton by passing dummy text from the consumer process to the main one, and get no slowdown at all. I'm not sure why running the prediction itself makes OpenCV slow to get webcam frames. Here's that code below:

class Consumer(multiprocessing.Process):

def __init__(self, task_queue, result_queue):
multiprocessing.Process.__init__(self)
self.result_queue = result_queue
#other initialization stuff

def run(self):
caffe.set_mode_gpu()
caffe.set_device(0)
#Load caffe net -- code omitted
while True:
#crop image -- code omitted
#text = net.predict(image)
text = "dummy text"
self.result_queue.put(text)

return

import cv2
import caffe
import multiprocessing
import Queue

results = multiprocessing.Queue()
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
rval, frame = vc.read()
else:
rval = False
frame_copy[:] = frame
while rval:
if not results.empty():
text = results.get()
#Add text to frame
cv2.putText(frame,text)

#Showing the frame with all the applied modifications
cv2.imshow("preview", frame)

#Getting next frame from camera
rval, frame = vc.read()
frame_copy[:] = frame
#Getting keyboard input
key = cv2.waitKey(1)
#exit on ESC
if key == 27:
break