Too few inliners obtained from the function cv::SolvePnPRansac

edit

I want to use cv::SolvePnPRansac to obtain the rotation and translation information between two images but most results give a very little amount of inliers. From the matching of the two images, it should not just have such few inliers or even 0.... Even if inliers is not 0, it will not greater than 20, unless occasionally.

I use the function like this:

cv::solvePnPRansac(pts_obj, pts_img, camera_intrinsic_matrix, cv::Mat(), rvec, tvec, false, 100, 8.0, 0.95, inliers );

pts_obj is the 3d points, and pts_img is the image points, intrinsic matrix is from the a function of the camera. Here is the result:

I do not know if the variables pts_obj and pts_img are correct. Here is the code:

std::vector<cv::Point3f> pts_obj;
std::vector< cv::Point2f > pts_img;
cv::Point3f point_obj;
cv::Point2f point_img;
std::vector<cv::DMatch>::iterator iter_goodMatches_start = goodMatches.begin();
std::vector<cv::DMatch>::iterator iter_goodMatches_end = goodMatches.end();
for(; iter_goodMatches_start != iter_goodMatches_end; ++iter_goodMatches_start){
    float x = previous_keypoints[(*iter_goodMatches_start).queryIdx].pt.x;
    float y = previous_keypoints[(*iter_goodMatches_start).queryIdx].pt.y;
    rs::float2 depth_pixel = { x, y };
    ushort depth_value = previous_depth.ptr<ushort>( int(y) )[ int(x) ];
    if(depth_value == 0){
        continue;
    }

    float depth_in_meters = depth_value * scale;// how much this point is away in meters
    rs::float3 depth_point = _depth_intrin.deproject( depth_pixel, depth_in_meters ); // 3D point
    point_obj.x = depth_point.x;
    point_obj.y = depth_point.y;
    point_obj.z = depth_point.z;
    pts_obj.push_back( point_obj );

    point_img = current_keypoints[(*iter_goodMatches_start).trainIdx].pt;
    pts_img.push_back( point_img );
}

Does anyone have this kind of experience?

Thanks.