Why homgraphy matrix doesn't work for any point in the image?

edit

I'm trying to find homography between the following two images.
If I calculate homography based on only corresponding points on the ground then the transformation works well only for the points on the ground. If I calculate homography based on only corresponding points on the heads then the transformation works well only for the points on the head level.

How do I get homography that will work for any point on the image? Is it even possible ?

The code for homography calculation:

vector<pair<Point, Point> > homographyPoints;

vector<cv::Point2f> plane1Points;
vector<cv::Point2f> plane2Points;
for (auto& p : homographyPoints)
{
    plane1Points.push_back(p.first);
    plane2Points.push_back(p.second);
}

homography = findHomography(plane1Points, plane2Points, CV_RANSAC);

The code for using the homography:

vector<cv::Point2f> srcPoints, dstPoints;
srcPoints.push_back(selectedPoint);
cv::perspectiveTransform(srcPoints, dstPoints, params->homography);