solvePnP for equirectangular image

edit

I have an equirectangular image of a room (360), and an accurate model of the room.

I have several matches between image-points and object-points (coordinates in the room) and I want to find the position and orientation of the camera where the image was taken.

SolvePnp (and solvePnPRansac) expect to receive the intrinsic parameters of the camera or to evaluate it, but since the image is equirectangular I can directly know the vector describing each pixel in the image (pitch and yaw).

I managed to solve it manually using some iterative algorithm for least-squares of the offsets between the vectors (something like Gauss-Newton). But it's not very fast and seems like the wrong way to go...