How to stabilize live feed?

edit

Currently I'm grabbing a frame at the start of my program, then splitting off the center 30% of the image as a "reference" then finding matching points using this reference and the live feed.

I'm using a flann-based matcher and the matching is relatively okay, when using draw-matches the lines between the two frames are generally speaking correct, as seen in this image:

https://i.imgur.com/bZWRMck.png

Then there's two ways to use these keypoint as far as i'm concerned, warpHomography and warpAffine but both give glitchy results where the frame gets warped in random directions instead of being stabilisation.

Because i know code will get asked;

//find matches if there's more than 70 points 
     Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create(DescriptorMatcher::FLANNBASED);
     std::vector< std::vector<DMatch> > knn_matches;
     int size1 = descriptors1.total();
     int size2 = descriptors2.total();
     Mat HInv;

     //cout << size1 << " - " << size2 << endl; 
     if(size1 >=70 && size2 >=70){
        matcher->knnMatch( descriptors1, descriptors2, knn_matches, 2 );
     }

        // //-- Filter matches using the Lowe's ratio test
        const float ratio_thresh = 0.7f;
        std::vector<DMatch> good_matches;
        for (size_t i = 0; i < knn_matches.size(); i++)
        {
            if (knn_matches[i][0].distance < ratio_thresh * knn_matches[i][1].distance)
            {
                good_matches.push_back(knn_matches[i][0]);
            }
        }
        //}
        //-- Localize the object
        std::vector<Point2f> obj;
        std::vector<Point2f> scene;

        for( size_t i = 0; i < good_matches.size(); i++ )
        {
            //-- Get the keypoints from the good matches
            obj.push_back( keypoints1[ good_matches[i].queryIdx ].pt );
            scene.push_back( keypoints2[ good_matches[i].trainIdx ].pt );
        }
            std::vector<Point2f>::iterator prevPoint=obj.begin();
            std::vector<Point2f>::iterator prevPointEnd=obj.begin()+3;
            std::vector<Point2f>::iterator currentPoint=scene.begin();
            std::vector<Point2f>::iterator currentPointEnd=scene.begin()+3;
            Mat A = getAffineTransform(fromTriPoints,toTriPoints);
            cout << A << endl;
            warpAffine(currentFrame, rectifiedFrame, A, currentFrame.size());
        }

Which functions should I be using?