register ir image to distorted rgb

edit

Hi guys, I need some help/hints regarding the following problem. I have an rgb and an ir image of the same scene but in different resolution.

RGB image (420x460):

and the ir corresponding (120x160):

Now I want to register the ir image to the distorted RGB one. The way I am doing it at the moment is by extracting the homography between the two images and then applying the transformation to the ir image. My upper goal is to use this transformation to the depth output and since the ir with the depth sensor output are aligned I am using the ir images since I can have access to more visual information. In the beginning I tried to use an automatic keypoints detector algorithm like SURF (it seems to perform a bit better, in regards to others) however the result is not that good since also the points that if finds are not that accurate as you can see below:

Keypoints detected:

Registration based on the above keypoints (not good at all):

I tried other keypoint detection algorithms as well but the result though it might improved it was not acceptable. Therefore, I decided to provide the matching points manually:

the registered image is by far much better now

and if I overlay the two images

However, you will notice that there is a kind of misplacement (or not good registration) in some spots:

Therefore, I was just wondering if there is a way to get a better result from what I am getting already. Note that I need to keep the distortion in the rgb image, therefore following the normal procedure (chessboard, extracting intrinsic/extrinsic coeffs, undistort, rectify, etc...) is out of the grid for the moment at least.

Thanks.

Adiitional code:

#include <iostream>
#include <stdio.h>
#include <opencv2/opencv.hpp>
#include "opencv2/core.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/features2d.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/calib3d.hpp"
#include "opencv2/xfeatures2d.hpp"

using namespace std;
using namespace cv;
using namespace cv::xfeatures2d;

int main(int argc, char *argv[])
{
    cout << "Hello World!" << endl;

    Mat rgb = imread("../frame_5.png", IMREAD_GRAYSCALE);
    Mat ir = imread("../frame_5a.png", IMREAD_GRAYSCALE);


    if(!rgb.data || rgb.empty())
    {
        cerr << "Problem loading rgb image!!!" << endl;
        return -1;
    }

    if(!ir.data || ir.empty())
    {
        cerr << "Problem loading ir image!!!" << endl;
        return -1;
    }

    Mat ir2 = ir * 2;


    imshow("rgb", rgb);
    imshow("ir", ir);
    imshow("ir2", ir2);


    waitKey();




    //-- Step 1: Detect the keypoints and extract descriptors using SURF
    int minHessian = 400;

    Ptr<SURF> detector = SURF::create( /*minHessian*/600);
//    Ptr<BRISK> detector = BRISK::create(100);


    std::vector<KeyPoint> keypoints_object, keypoints_scene;
    Mat descriptors_object, descriptors_scene;
    detector->detectAndCompute( rgb, Mat(), keypoints_object, descriptors_object );
    detector->detectAndCompute( ir, Mat(), keypoints_scene, descriptors_scene );

    //-- Step 2: Matching descriptor vectors using FLANN matcher
    BFMatcher matcher(NORM_L2, true);
//    FlannBasedMatcher matcher;
    std::vector< DMatch > matches;
    matcher.match( descriptors_object, descriptors_scene, matches );
    double max_dist = 0; double min_dist = 100;
    //-- Quick calculation of max and min distances between keypoints
//    for( int i = 0; i < descriptors_object.rows; i++ )
    for( int i = 0; i < matches.size(); i++ )
    { double ...