Fundamental matrix for tiny movements

edit

I want to estimate the rotation between two consequences frames of an calibrated camera where the movement is very small (few degrees and few CMs).

I have found matching points between the two frames accurately. Then, I called findFundamentalMat and computed the Essential Matrix and decompose it using decomposeEssentialMat. I got R1 and R2. However, both of them seems incorrect. They are huge (~50 degree on each axis). I have looked at the Epipolar lines and they looks like:

Although they are correct in term of matching. The overall estimation is not correct. Is there a way to find fundamental matrix that satisfies small movements conditions?