Track with monocular camera-See with stereo cameras

edit

Hi guys! I want to build an application, so that the user can wear Oculus+Stereo cameras in front of oculus+RGB-D camera mounted also on top of oculus. The goal will be to use RGB-D camera to track a marker in a scene and then I want the user to see the virtual content through the stereo cameras instead of the RGB-D Camera.

So RGB-D camera will do all the tracking stuff,but user will see virtual objects through stereo cameras.

What exactly should I do to achieve this?Do I need to transform the virtual objects from RGB-D camera to the stereo cameras?How can I do it?Any resources/tips or similar projects?