To filter out incorrect your matches you can use epipolar constraint. Use findFundamentalMat function with FM_RANSAC estimation method to estimate fundamental matrix based on matching points from 2 images. RANSAC can deal with situation when you have many outliers.