findFundamentalMat not correctly filtering outliers

edit

After detecting keypoints and matching them between two images, I run findFundamentalMat to estimate the Fundamental matrix and also filter the outliers. When I draw the matches using the mask I get from findFundamentalMat, there is sometimes some matches that are not filtered out eventhough they clearly don't fit in the transform.

Here is an example of a good filtering (Left image from robot's camre, right image static):

But without moving the robot, the matches change a lot from one picture to the next (due to the flickering in the light?) And often there is one wrong match or two that are left. I suspect those matches to cause the inconsistency in my estimated Fundamental matrix which can look totally different from one image to the next, even without moving the robot.

Here the yellow and blue line clearly don't fit to the model. Could they cause the fundamental matrix to go totally wrong?