Revision history [back]

You are trying to mix two different approaches to pose estimation: 2D-to-2D ( via essential matrix) and 2D-to-3D using triangulated 3D points. solvePnP is an algorithm for the latter approach. Bundle adjustment basically minimizes the reprojection error over the set of poses and 3D points, so changing extrinsics are actually essential for bundle adjustment. The tutorial here and here gives a very good overview on the topic of visual odometry.

Usually, the 3D coordinate frame is defined such that its origin is at the center of the first camera so in this sense, yes, K1[I|0] for the first camera and K2[R|t] for the second camera is correct. The first time you triangulate the points, the obtained 3D points are expressed in the 3D coordinate frame defined by the first camera. Therefore, when you want to reproject the 3D points in e.g. view 3 in order to calculate the reprojection error, you need to know the pose of the 3rd camera w.r.t the first view.

Hope this helps.

You are trying to mix two different approaches to for pose estimation: 2D-to-2D ( via essential matrix) and 2D-to-3D using triangulated 3D points. solvePnP is an algorithm for the latter approach. Bundle adjustment basically minimizes the reprojection error over the set of poses and 3D points, so changing extrinsics are actually essential for bundle adjustment. The tutorial here and here gives a very good overview on the topic of visual odometry.

Usually, the 3D coordinate frame is defined such that its origin is at the center of the first camera so in this sense, yes, K1[I|0] for the first camera and K2[R|t] for the second camera is correct. The first time you triangulate the points, the obtained 3D points are expressed in the 3D coordinate frame defined by the first camera. Therefore, when you want to reproject the 3D points in e.g. view 3 in order to calculate the reprojection error, you need to know the pose of the 3rd camera w.r.t the first view.

Hope this helps.