Direct conversion of pixels to millimeters?

edit

use intrinsic calibration parameters to turn the pixel coordinate (in 2d screen space) into a viewing ray (in 3d camera space). that is done in two parts, undistortion and "un"-projection (using the camera projection matrix). when you write down the linear algebra, you'll see how that results in a ray: projection removes one dimension, so this operation must add one dimension.

use extrinsic calibration parameters (transformation matrix between world space and camera space) to transform the viewing ray from camera space to world space.

intersect the ray with whatever surface you think made that pixel.