Detecting movement of the camera itself

asked 2019-09-02 04:27:16 -0500

The goal is to detect whether or not the computer camera is moving. Reaction needs to be fast. If people are moving in front of the camera, it shouldn't confuse the camera.

I tried using mobilenet SSD on the images, masking away people detected in the image, then using background subtraction to determine how much movement there is. It works okay, but it is slow. Problems ; mobilenet SSD is slow, and background subtraction isn't always that accurate (if I go stand in front of the window the change in lighting makes it look like there was a lot of movement).

I tried considering only corners but what's in the corners is usually considered background by the algorithm.

I tried doing feature matching to detect if keypoints all moved in a similar direction (in the corners of the image, for example).

More ideas I've yet to try ; optical flow (no experience with this), camera calibration with RANSAC (no experience either) and training my own machine learning model (based on 1000fps Human Segmentation with Deep Convolutional Neural Networks by C. Song).

Does anyone here have any other ideas that might increase the accuracy of the algorithm? Accuracy and speed of detection are equally as important.

edit retag flag offensive close merge delete


The goal is to detect whether or not the computer camera is moving

why so ? can you try to explain ? what is your input ?

berak gravatar imageberak ( 2019-09-02 13:16:47 -0500 )edit

It's just the end goal to detect camera movement on any laptop, it's the point of the application. The input is the camera frames, with maybe 15FPS or so (maximum of 30). The initial approach (doing background subtraction on the input frames after masking out all persons detected with machine learning) worked okay, but the machine learning model is rather slow (max 6 frames per second) leading to frequent false positives.

Peire C gravatar imagePeire C ( 2019-09-02 13:53:09 -0500 )edit

@Peire C. You can't. How does the camera is moving? By hand, motor or pods(dolly or stick)=> that is the answer, Not camera itself.

supra56 gravatar imagesupra56 ( 2019-09-03 06:54:32 -0500 )edit

What do you mean? It's an integrated camera, like the iSight of a Macbook. There's no motion sensor. I was wondering if you could analyse images to decide whether or not the camera is moving. It shouldn't be accurate in all cases. The initial approach worked okay to be honest, but the machine learning model makes it slow.

Peire C gravatar imagePeire C ( 2019-09-03 07:24:57 -0500 )edit

How do you mount on camera? I am not talking about sensor.

supra56 gravatar imagesupra56 ( 2019-09-03 08:23:52 -0500 )edit

Not sure what you're asking ... I have a camera. It captures frames. I analyse those frames. The camera moves if the computer moves.

Peire C gravatar imagePeire C ( 2019-09-03 08:48:11 -0500 )edit

In my hope. I was consider about motion detection.

supra56 gravatar imagesupra56 ( 2019-09-03 12:51:28 -0500 )edit

Any ideas how to tackle it? Dense optical flow is slow, optical flow is not robust, absdiff isn't either, background subtraction captures any sort of movement and it is hard to discriminate between camera movements and movement of objects in the environment, ... the best approach so far is single-shot-detection of people -> masking them away -> doing background subtraction on the rest. It only fails in one case ; when someone moves very close to the camera. I'm gonna try segmentation at higher FPS now, but it may just become one more wasted effort.

Peire C gravatar imagePeire C ( 2019-09-03 13:54:14 -0500 )edit

Unable help you. Because, I'm getting 28" monitor 4k UHD by tomorrow. I need more time.

supra56 gravatar imagesupra56 ( 2019-09-04 21:37:07 -0500 )edit

I managed to get satisfactory results with segmentation (something like body-pix), taking the areas where it's least probable that there are humans there, and looking at the absolute difference between frames in those areas.

Peire C gravatar imagePeire C ( 2019-09-15 10:10:33 -0500 )edit