Dewrapping results not expected result

edit

I'm trying to iterate through end points of lines, build segments between them and finally dewarp the distortion. I should imagine, that all those "mini"-segments are small images (somthing about 70x300, but it variets.)

This what I done:

//allPoints are created in another class, but the output 
//for std::vector<std::pair<cv::Point, cv::Point>> allPoints is as follows:
//(for the first:)[139, 113][201, 310][223, 339][297, 437][323, 472][381, 465][408, 413][484, 291][505, 151]
//(for the second:)[139, 210][201, 692][223, 664][297, 550][323, 523][381, 544][408, 580][484, 699][505, 254]


for (int i = 0; i < allPoints.size() - 1; i++) {
    cv::Vec3b zero(0, 0, 0);
    cv::Mat_<cv::Vec3b> img_dst(allPoints[i].second.y - allPoints[i].first.y,
        allPoints[i+1].first.x - allPoints[i].first.x);
    std::cout << "\nSize of img_dst: " << img_dst.size() << "\n";

    const cv::Point2f src_pt[] = {
        cv::Point2f(allPoints[i].first),
        cv::Point2f(allPoints[i].second),
        cv::Point2f(allPoints[i + 1].second),
        cv::Point2f(allPoints[i + 1].first) };

    const cv::Point2f dst_pt[] = {
        cv::Point2f(allPoints[i].first),
        cv::Point2f(allPoints[i].second),
        cv::Point2f(allPoints[i + 1].second.x, allPoints[i].second.y),
        cv::Point2f(allPoints[i + 1].first.x, allPoints[i].first.y) };


    const cv::Mat homography_matrix = cv::getPerspectiveTransform(src_pt, dst_pt);

    cv::warpPerspective(matCopy, img_dst, homography_matrix, img_dst.size());

    cv::imshow("Result", img_dst);


    std::pair<vector<int>, std::string> wallNr = WallParser::wallParser(argument);

    int label = wallNr.first[i];
    std::cout << label << " ";

    Main::saveAll(img_dst, "", label);

}

So actually I'm giving the right size and also the right points, which should be dewrap. But I get instead of (created it quicklin ps - in PS it results a bigger image) this one . The image looks like this and all those end points are the end points of the yellow lines:

How can I get that wrapPrespective to dewrap right?