Ask Your Question

newbe question about OpenCV live video capabities

asked 2017-10-27 14:20:28 -0600

miner_tom gravatar image

updated 2019-12-09 07:45:24 -0600

ojesus gravatar image

I am an electronics engineer in my 60's and I remember a time when image processing was done with expensive special purpose hardware. I have worked on some of these processing engines. These days, OpenCV can even run on small project boards like the Pi and others.

I am interested in processing real time video and before I get too involved I would like to know if OpenCV can do what I require. For the moment, lets not consider video bandwidth, although I am sure that is an issue.

This should sound a lot like gaming. What I would like to do is take real time video from a camera, and with a joystick, move a visible cursor around in what I believe would be considered a type of overlay (that may have meant something different 30 years ago). If the cursor was in the form of a crosshair (like a target) then I would like to be able to put the crosshair over objects in the camera view.

That does not sound like it would be too hard for OpenCV because there is not any image processing to be done at this stage.

Image processing would come into play if i wanted to tag (shoot, click on, etc.) an object in the camera view. At that point the processor would have to know the position of or follow the object in question.

So, the analogy to a video game would be if I saw a person or moving object in the video field, and I put the cursor onto the object, the processor would have to "know" that the cursor is on the object in question, if i "trained" the processor to follow that kind of object. That would sound to me like a first person shooter game where the object was in live video.

To put this another way, the target would have to "know" that it was tagged.

I hope that makes sense. Thank You Tom

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2017-10-27 17:12:12 -0600

Howdy, from the other end of the spectrum; early 20s. It is always exciting to have an elder still excited in CV!

OpenCV can do what you are asking for in its entirety actually. The skeletal view of your problem can be broken into four sections.

1. Image Preprocessing

You would want to perform some sort of thresholding, edge detection, morphological operations, contour extraction, etc in order to remove any form of noise and also help you isolate the targets in your game. C++ tutorial Python tutorial

2. Event Detection

OpenCV offers mouse callbacks which would give you the respective (x, y). C++ tutorial Python tutorial. I recommend the Python tutorial link here for reference then convert the logic to your desired language.

3. Object Detection

After stage 1, you hopefully have a feed consisting of a majority of your targets and unnecessary noise removed. But since you can't guarantee having a noiseless image, its time to detect your object in this less noisy image.

If your object is going to be similar throughout, you can employ template matching keypoint matching. Here's a full list with some info.

If your objects have certain polygonal features, you can tweak attributes such as circularity, area, perimeter etc from the blob extractor in an attempt to detect your object. Or you can use one of the following features detectors to achieve the task as well.

Depending on your object, you can go search online for suggestions on how to detect it.

4. Object Tracking

Hopefully at this point we have achieved a good threshold of detection, it is time to start tracking the object. This is where the tracking API comes into play. This blog does a good job of explaining them.


I added the Object Detection and Tracking sections because I am not sure of your game design.

Scenario I

If your game is event driven i.e. depending on where the user points the crosshair do something, then you do not really need any of them. Depending on the bounding (x, y) you can just remove whatever object is around that area. No detection or tracking required.

Scenario II

If the user's input (x, y) would require you to track the object, then no need for detection. Simply extract features around (x, y) and track them.

Other scenarios

Maybe you need detection and tracking, or just detection, etc. Point is, you do not necessarily need to utilize both detection and tracking or even either.

Two Cents

I would recommend you mess around with the skeletal design in an attempt to design a good foundational system before jumping to the masking and creating other layers; you probably already know this :)

OpenCV offers sample code both in C++ and Python illustrating how to do a majority of the stuff I stated above. Enjoy this video showing an application that covers a huge chunk of your game's core.


edit flag offensive delete link more


Wow! Thank you so much for the reply. I don't have the time to read the details at the moment but I can assure you that I will give your reply the attention that it deserves. Also, thank you for welcoming me into this group. Yes, I used to work for an image processing company called Vicom Systems, back in the early 80's. The system did 2D and other high pass low pass filtering in real time by doing multiple and repeated 3x3 convolutions (FFT's were far too costly in terms of processor performance at the time).

Like I said, I will really look over your post and may have more questions as I go on.


miner_tom gravatar imageminer_tom ( 2017-10-27 17:28:46 -0600 )edit

Question Tools



Asked: 2017-10-27 14:20:28 -0600

Seen: 508 times

Last updated: Oct 27 '17