Revision history [back]

This is an area with a rather large space for trading off complexity, speed, and accuracy. If you want a very quick, simple, possibly inaccurate version, what you have described is fine. It's likely to be inaccurate only if you have pretty large errors, such that your coordinates for one track match more closely with a nearby track in the other camera.

The google search term to use is Track Association. You can find many many papers on ways to do this, although you should look pretty far back to find ones that deal with situations as simple as yours. Many current ones are meant to deal with many more uknowns.

If you have reasonable confidence is some of your associations you can even use them to update your calibrations, which will reduce the errors in the future.