I have a camera which points in a direct, I have a unit vector $\vec{c}$ which describes the orientation of the camera in world coordinates.
There is a point of interest in the image taken by the camera. Given the field of view of the camera and image size, I can compute two vectors in pixel space:
$\vec{a}$, the principal point (center point of the image), and
$\vec{b}$ the point of interest in pixel space.
I want to rotate the camera $\vec{c}$ (in world coordinates) such that it now points at the object represented on screen by $\vec{b}$
It's unclear to me how to transition between the on-screen pixel-space orientation of vectors $\vec{a}$ and $\vec{b}$ and the world space vector $\vec{c}$.