Ask Your Question

Tracking a large number of dots on a cloth

asked 2020-11-25 18:04:41 -0500

Rocketmagnet gravatar image

updated 2020-11-25 18:10:56 -0500

I'm reasonably new to OpenCV, but have used it a little for capturing the webcam and writing video.

I would like to know if my problem is fairly easily solvable using OpenCV, and would be grateful for some pointers in the general direction of the solution. I.E. the name of the type of tracking I would like to do.

I have a flexible surface covered in a non-repeating pattern of dots. The pattern is known and each dot has a number associated with it.

image description

There will be about 2000 dots. The dots can be various shades of grey. The pattern of dots is known beforehand, but the shade of each dot is not. The surface can move and distort, but will never self occlude. The surface will start in its known, neutral state, then begin to distort.

I would like to track the position of every dot as accurately as possible, even for those dots who's shade of grey makes them almost indistinguishable from the background. This should be possible because their positions can be inferred from the positions of nearby dots. I also need to know which dot on the screen corresponds to which numbered point in the reference pattern.

The lighting will be controlled, diffuse, and consistent.

What functions does openCV offer that could help me solve this problem? Which functions should I begin to research? Any help appreciated.

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2020-11-25 19:42:54 -0500

crackwitz gravatar image

updated 2020-11-25 20:14:10 -0500

I'm unaware of anything in OpenCV that is a complete solution to this. I hope you'll get answers that come closer to a solution.

you should first find all/most points, calculate a centroid for each. you'll need that as a starting base for pretty much anything. lots of ways to do that. thresholding and contours/connected components, blob detection, convolution...

if you're lucky you can track them individually... if you have sufficiently small movement frame-to-frame for "nearest point in next frame" to be unambiguous. match a point in one frame to the nearest point in the other frame. see if they're a match in the other direction too for stronger evidence.

you can even try optical flow. that may or may not see the coarser structure of your pattern (which is important for movement farther than half a mesh edge). try opencv's DIS optical flow. it's faster and better than the older methods in OpenCV. you can map points from the previous frame through the flow field and match them to the nearest point in the current frame (or the opposite).

if none of that works... you'll have to come up smarter ways to give a point an identity. that can be done with "feature descriptors", which use local structure to describe any point in an image. you can match those descriptors to establish correspondence. you'd start from good/strong matches and refine the mesh into the less identifiable points.

throw an existing feature descriptor on the problem, such as SIFT or AKAZE. they come with their own feature detector (source of interesting points) but you can also feed them your points.

if none of those do too well, you can engineer your own descriptor. things you can turn into description/identity: number of direct neighbors (see next paragraph), number of points within radius r, distances to nearest n non-black points, radius of nearest n points, count of each color (histogram) for closest n/radius, some measure of gradients or orientations (look at "histogram of oriented gradients" for inspiration), ...

you'll want to come up with some logic that establishes a mesh relationship from a set of points, so you can walk it, and also detect if you're trying to match points that don't have matching neighborhoods (a feature for the previous paragraph). you'll need a mechanism to establish "direct neighbors" for a given point. from your picture that can be 5-7 neighbors. I'd do a... 7-nearest query, figure the median distance of those, and keep those that are close to the median (discarding the ones further away, if there were any).

you'll need a data structure that gives you k-nearest neighbor (KNN) points for any point efficiently, so you can visit the mesh and propagate correspondence throughout it. OpenCV comes with FLANN, which can do that. it's mostly used to compare feature descriptors (vectors of numbers/bits) but you can use it on ... (more)

edit flag offensive delete link more

Question Tools

1 follower


Asked: 2020-11-25 18:04:41 -0500

Seen: 42 times

Last updated: Nov 25 '20