3D pose from multiple cameras and known key points

asked 2020-08-13 09:54:51 -0500

andyfilms gravatar image

Hi all, I have a question that I'm sure is simple, but being new to MV I just don't know the "magic words" that describe what I'm trying to do.

Lets' say I have a number of cameras in a room, and I have an object with IR LEDs of a known configuration. For the sake of argument, let's say the LEDs are bright enough that the cameras exposure can be low, so that they're effectively the only thing the cameras see. I can calibrate the cameras in any way needed.

What I'd like to be able to do to is get the position of the object in 3D space, for example in a way that I could feed into Unity to attach a 3D object to in a scene. This is similar to some commercial motion capture systems.

What would be the best place for me to get started with this? I feel like it's simpler than most projects I see here--but again, assume I know nothing :)

edit retag flag offensive close merge delete


an example image would be helpful ;)

are you free to position the led's on your object ? (e.g. in a grid)

how large is it ? (maybe it's posssible to attach IR backlit aruco markers ?)

berak gravatar imageberak ( 2020-08-13 10:00:31 -0500 )edit

Absolutely, here's an image of the marker from the commercial solution I'm trying to emulate: https://tracklab.com.au/wp-content/up... And the kind of setup it would be used in: https://optitrack.com/public/images/f...

andyfilms gravatar imageandyfilms ( 2020-08-13 10:06:32 -0500 )edit

solvePnP() would be possible technique, but it would require having 3d model points and corresponding 2d image points

berak gravatar imageberak ( 2020-08-13 10:13:26 -0500 )edit

pardon my ignorance, but where are the ir led's in the image ?

and how would your "object" be used in the mocap setup ?

what does this thing do ?

berak gravatar imageberak ( 2020-08-13 10:16:04 -0500 )edit

No worries, so the balls at the end of that device are IR emitters—thats basically all it is. That device is simply placed on objects that you want to motion track. In this case, I’m not doing body skeletal tracking or anything—just objects.

andyfilms gravatar imageandyfilms ( 2020-08-13 10:33:14 -0500 )edit

oooh, i see. but that's only the hardware, (is there any sw coming with it ?) and now you want to do the 6dof pose tracking using opencv ?

499$, -- hefty price tag for a plastic battery container and some led's, btw ;)

berak gravatar imageberak ( 2020-08-13 10:57:16 -0500 )edit

You’re telling me! The whole setup including cameras costs over $250k. Now you see why I want a DIY solution;) Lets say I’m making my own emitter based on that one, so I can integrate it anyway I want—so if it requires a custom driving of the LEDs, that can be done. But yes, I’d like to use OpenCV to get the pose and position in space.

andyfilms gravatar imageandyfilms ( 2020-08-13 11:03:24 -0500 )edit

i see, ;)

i'm wondering, why they made it quadratic, there's a 90° rotation (around Z) ambiguity, no ?

i think you could use solvePnP(), if you could arrange led's into some "asymmetric circles grid" pattern like:

*   *   *   *
  *   *   *
*   *   *   *

(similar to what's used in calibration, and you could reuse functionality to retrieve those)

see, the problem with solvePnP is that you need corresponding 2d and 3d points, meaning -- in exactly the same order, which is hard to achieve otherwise (works perfect with facial landmarks or anything else "ordered")

berak gravatar imageberak ( 2020-08-13 11:59:58 -0500 )edit

heretic question: why ir leds, and not just use daylight and aruco markers or similar ? (i mean - it's all ready-made there, -- you throw in an image, and receive rot / trans vecs wrt. the camera ... )

berak gravatar imageberak ( 2020-08-13 12:02:24 -0500 )edit

I see what you mean, I’m unsure why they’re symmetrical tracker—I understand why that’s a problem. Honestly I do like your idea of backlit markers, I’m thinking I could maybe build a cube with a different marker on each face. Really I just want whatever’s most accurate. Thanks!

andyfilms gravatar imageandyfilms ( 2020-08-13 13:31:31 -0500 )edit