Units of Rotation and translation from Essential Matrix

edit

Thank you very much. I need little more clarity on convention of the coordinates. So camera origin c is position of camera in world coordinates, which keep changing if the scene changes? My understanding is that after finding Fundamental matrix(findFundamentalMat) from from the common keypoints of Frame1 and Frame2, from which R,t are extracted from Essential matrix. The camera position at Frame1 is subjected to R rotation and t translation will give camera position at Frame2. Is that right or wrong?

Yes, the camera origin c is the position in world coordinates. When extracting the camera matrix P2 = [R|t] of your second view, you always assume, that the world coordinates coincide with the first camera, such that the first camera actually has a camera matrix P1 = [identity|0]. Therefore, R and t represent the motion from the second camera to the first camera (or world coordinate origin) in the second camera's coordinate frame. Finally, you need to invert this motion, to obtain the pose of your second camera w.r.t to the world coordinate system. This blog post is very helpful to understand these geometric relations. So regarding your last question in above comment: It's exactly the other way around.

Thanks. For my Edit 2 : When finding the scaling factor lambda, (sx2, sy2, 1) = K(R|lambda*T)(X,Y,Z) The (X,Y,Z) are coordinates in Frame2 and sx2,sy2 are the pixel coordinates in Frame1 ? Assuming I have (X,Y,Z) for different keypoints , I will get a set of K values, for which I will take a median to estimate it. So Is this scaling factor changing or it is constant throughout motion? When I was doing the above method for a static scene, I was getting the some [R|t] instead of [identity|0] (The values of [R|t] was same for all static scenes incidentally ) , is that due to calculation errors or will the scaling factor make it nearly [identity|0]?

The camera matrix from an essential matrix obtained with two views, is always the camera matrix of the second camera, P2 = K*[R|t] (considering calibration matrix K), based on the assumption that the world coordinate frame has its origin at the first camera center. A 2D point x seen in the second image is given by x = P2*X, where X is a 3D point in world coordinates. So whenever you calculate the camera pose from a new pair of views, the camera pose is always relative to the first view of this pair. So let's say you have ndifferent views. You could e.g. always calculate the essential matrix w.r.t the first view and then, from the first set of triangulated keypoints, you should ideally be able to estimate the correct scale of all other views w.r.t the first (unit) translation.

Ok, So essentially the scale factor is constant. But then getting some [R|t] instead of [identity|0] for static scene , should either be calculation mistake or may be due to SVD . Is there any reason why this is happening? Programmatically I am solving this by ignoring the static frame as the [R|t] had certain values when Frame1= Frame2 . Thank you very much, you have pretty much cleared all my doubts and also added finer details for implementation like checking the feasible rotations and translations.

If the scene is static, then the scale factor of all 3D points should be constant w.r.t. to one specific view. You never get [identitiy|0] from the SVD decomposition of the Essential matrix, because this is the underlying assumption of the first camera that allows you to calculate [R|t] of the second camera. Or am I maybe not understanding your question correctly? So by static scene, do you mean a rigid scene with two images taken from different poses?

By static scene I meant that the Frame1 and Frame2 are the same image i.e in the findFundamentalMat is inputed the exact same points. Which effectively means there is no rotation nor translation from the previous position.(Effectively the first camera and second camera positions are same, as I am assuming the first camera position as [identitiy|0], then the second camera position should also be [identitiy|0], atleast the rotation part may not be identity matrix, but the translation has to be zero )

I think the problem is that the Fundamental matrix is defined by the epipolar geometry, but I can't recall the details right now and what could probably be the cause of your problem.