Real world coordinates from a pixel

edit

I can't use a reference point because the stereo are mounted in a robot arm. I read about other solution that use the opencv function reprojectImageTo3D() but I need the disparity array and Q matrix, but I'm not sure if I'm on the right direction.

Can you tell me more about the application? Do you want to know the height of an object in order to give the right commands to pick it up with a claw?

I have a work piece with holes and I want to determine where are the holes to move the robot near it.

I once did a project where I had to detect coloured cubes and stack them accordingly with a robotic arm. What I was doing was moving the arm in order to make a 90º angle between the camera and the working top. Then I'd detect the object and move the arm until the centre of the object was in the centre of the image, at which point I would compensate the offset between the camera and the centre of the tool. I didn't use any absolute coordinates, just incremented movement to the claw until It was aligned with the object. If you can detect those holes you can also use this method. Check it out https://www.youtube.com/watch?v=yGOz2...

Nice work!! But how do you determine the depth?

The height of the working top and the cubes was known.

OK. In my application the work piece is not always the same. So I need XYZ coord of the pixel.

You told me that you had a working stereo set. With that, you can calculate the distance of your tool to the surface, and then calculate the height of your work piece. For this you'll need to have a plane model computation at the initialization phase. Define the working top plane at the beginning, and then calculate the height of your work pieces relatively to the plane