Translating from camera space, into see through display space

edit

The short answer is, you can't do this properly, but you can fake it.

Long answer is, because your camera isn't aligned with your display, you need to know how far away to draw the object. Any single pixel in the camera, when drawn at different ranges in the display, makes a line across it. See the tutorial HERE.

You can fake it by picking a range to draw at. You take your camera, and the point you wish to draw. "Push" the point into 3d space, then project it back onto the display. If you had the real range from some source, you would use that.

To do this, you need to calibrate the camera, and the hypothetical camera that is the eye. You have to know the relationship (IE: rvec, tvec) between them.

You can fake that by scaling the FOV and translating, but unless what you're looking at is really far away, the camera is perfectly aligned with "front" and neither is very distorted, it probably won't look right.