Code coming to a halt after 45 seconds

edit

I'm using opencv to do some object tracking and after running it for more than ~45 it slows down and quickly becomes totally unresponsive and I have to manually kill the process. CPU and RAM usage stay constant at 35% and 2.5% respectively for the first ~35 seconds and then in the following 10 seconds CPU usage blows up and quickly goes to 99% and locks up the machine.

The basic algorithm goes like this:

1. Import the video frame
2. Cut size in half to reduce calculation time
3. Convert to HSV color space
4. Define upper and lower bounds to define color to track
5. Create mask based on that color
6. Apply mask to isolate the color
7. Dilate the resulting image and erode the edges to reduce noise
8. Use numpy.where to locate white pixels in resulting binary array
9. Find the average value of the x-axis position
10. Compare to left/middle/right boundaries of the frame
11. Print out relative x-axis position, and send character to arduino
12. Rinse and repeat

The arduino is running a short code to look for a hex character and has some if cases to turn on LEDs to give visual feedback of the object's location. I have it sleep for 1/100th of a second after sending commands to the arduino to ensure that my computer and the arduino are synced up, and I'm communiating with it via serial at 9600 baud. I don't see any reason why CPU usage would go crazy after such a short time when it has no problem handling the initial 35 seconds.