Using triangulatePoints() and projectPoints() with stereo cameras

edit

Thanks for the reply. Just to confirm: In my application, there are two cameras but camera2 does not stay in one place: hence the issue of estimating its position relative to camera1 (and subsequently trying to stabilize it at a certain baseline: part 2). As the scale keeps changing for every pair of frame I analyze with the essential matrix, I thought bundle adjustment would not work. Do you still think BA would work for my system? Perhaps I used the word extrinsics wrongly: I meant the baseline would not be constant either.

The camera extrinsics are the pose of the camera w.r.t a (arbitrary user-defined) 3D coordinate system, so for a 2-camera setup with the 3D coordinate system defined at the center of the first camera, the translation part of the second camera's extrinsics corresponds to the baseline when speaking in terms of a stereo setup. You're right about the change of scale, 2D-to-2D via essential matrix normalizes the translation vector to unit for each pair of views. So what you could do is using 2D-to-2D to get an initial estimate of pose, then you use this estimate to triangulate the points. Once you have established this inital set of pose and 3D points, you can use PnP for further pose estimation (with correct scale). Then you can use BA to minimize the reprojection error.

I see. That makes things clearer. A follow-up question: You said "you can use PnP for further pose estimation (with correct scale)." I don't have any objects of known size etc. in the scene to recover the scale information: at the end of the day, I will be using another sensor (GPS/IMU) to get metric translations (those sensors not being precise enough by themselves to do this job). So throughout the whole essential matrix->triangulation->PnP->BA pipeline, I would still be working with an arbitrary scale. Would that cause any issues for my end goal?

You'll get a correct estimation/ scene reconstruction up-to-scale. So the relative translation between the cameras and the positions of the triangulated 3D points will be correct with respect to the translation from the first to second camera, which is assumed to have length 1. You will obtain an up-to-scale metric reconstruction, which unfortunately does not have any absolute information. BA will work without that information, as it refines your pose and 3D points up-to-scale as well. To get the absolute scale, you either need to have previous knowledge about the size of an object in the scene or as you plan, you could use the information from GPS/IMU. This can be included after the BA by multiplying all your metric quantities with the GPS/IMU estimate of the first translation vector.

Got it. Thanks a lot, that was very helpful!