Problem with estimateRigidTransform: mat dst is empty

asked 2017-06-06 03:26:44 -0500

JoeBroesel gravatar image

Hello everyone, I'm new to OpenCV, so it could be that it's just a understanding problem with the estimateRigidTransformation function:

In the following code i find the contours of two rigid translated objects in img1 and 2, but estimateRigidTransformation seems not to work like i thought it would. It would be nice if someone has an idea why the mat dst keeps empty. Thank you!

#include <iostream>
#include <string> 
#include <opencv2/highgui/highgui.
#include <opencv2/video/tracking.hpp>
//Function from https://github.com/opencv/opencv/blob/master/samples/cpp/shape_example.cpp to extract Contours
static std::vector<cv::Point> sampleContour( const cv::Mat& image, int n=300 )
{
   std::vector<std::vector<cv::Point>> contours;
   std::vector<cv::Point> all_points;
   cv::findContours(image, contours, cv::RETR_LIST, cv::CHAIN_APPROX_NONE);
   for (size_t i=0; i <contours.size(); i++)
   {
       for (size_t j=0; j<contours[i].size(); j++)
       {
          all_points.push_back(contours[i][j]);
       }
   }

   // In case actual number of points is less than n
   int dummy=0;
   for (int add=(int)all_points.size(); add<n; add++)
   {
       all_points.push_back(all_points[dummy++]);
   }
   // Uniformly sampling
   std::random_shuffle(all_points.begin(), all_points.end());
   std::vector<cv::Point> sampled;
   for (int i=0; i<n; i++)
   {
       sampled.push_back(all_points[i]);
   }
   return sampled;
}

int main(){
// image reading
cv::Mat templateImage = cv::imread("1.jpg", cv::IMREAD_GRAYSCALE);
cv::Mat queryImage = cv::imread("2.jpg", cv::IMREAD_GRAYSCALE);

// contour extraction
std::vector<cv::Point> queryPoints, templatePoints;
queryPoints = sampleContour(queryImage);
templatePoints = sampleContour(templateImage);

// cast to vector<point2f> https://stackoverflow.com/questions/7386210/convert-opencv-2-vectorpoint2i-to-vectorpoint2f
std::vector<cv::Point2f> queryPoints2f, templatePoints2f;
cv::Mat(queryPoints).convertTo(queryPoints2f, cv::Mat(queryPoints2f).type());
cv::Mat(templatePoints).convertTo(templatePoints2f, cv::Mat(templatePoints2f).type());

cv::Mat R = cv::estimateRigidTransform(templatePoints2f,queryPoints2f,false);
std::cout <<"R:"  << R << std::endl; // R -> empty

/*
 * Solution from https://stackoverflow.com/questions/23373077/using-estimaterigidtransform-instead-of-findhomography
 * let the program crash
 *

cv::Mat H = cv::Mat(3,3,R.type());
H.at<double>(0,0) = R.at<double>(0,0);
H.at<double>(0,1) = R.at<double>(0,1);
H.at<double>(0,2) = R.at<double>(0,2);

H.at<double>(1,0) = R.at<double>(1,0);
H.at<double>(1,1) = R.at<double>(1,1);
H.at<double>(1,2) = R.at<double>(1,2);

H.at<double>(2,0) = 0.0;
H.at<double>(2,1) = 0.0;
H.at<double>(2,2) = 1.0;

std::vector<cv::Point2f> result;
cv::perspectiveTransform(templatePoints2f,result,H);

for(unsigned int i=0; i<result.size(); ++i)
    std::cout << result[i] << std::endl;
*/
 return 0;
}
edit retag flag offensive close merge delete

Comments

I think if you give two contours templatePoints2f,queryPoints2f it means that templatePoints2f[n] must be matched with queryPoints2f [n] Algorithm uses ransac some matching errors are possible

LBerger gravatar imageLBerger ( 2017-06-06 04:03:23 -0500 )edit

Thank you for your response, if i get you right the order of the Points in the vector is relevant?

I noticed that if I canged the line to

    cv::Mat R = cv::estimateRigidTransform(templatePoints2f,templatePoints2f,false);

or

    cv::Mat R = cv::estimateRigidTransform(queryPoints2f,queryPoints2f,false);

i get an output in the form of something like this: R:[1, -6.938148514913645e-16, 7.614916766799279e-14; 6.938148514913645e-16, 1, -1.285599231237722e-13]

instead of this: R:[]

Nevertheless the points are random_shuffled, aren't they? Would it help if i upload the used pictures (rotated apple-shapes from MPEG-dataset)?

JoeBroesel gravatar imageJoeBroesel ( 2017-06-06 04:22:39 -0500 )edit

random_shuffled in a set = if you chose index i in first set in second set you chose index i too. I will test on basic sample.

LBerger gravatar imageLBerger ( 2017-06-06 04:29:25 -0500 )edit

try this :

    vector<Point>  vertex1;
    vector<Point>  vertex2;

    vertex1.push_back(Point(300, 300));
    vertex1.push_back(Point(500, 300));
    vertex1.push_back(Point(500, 500));
    vertex1.push_back(Point(300, 500));
    for (int i = 0; i < vertex1.size(); i++)
        vertex2.push_back(vertex1[i] + Point(200, 200));
    cout<< estimateRigidTransform(vertex1, vertex2, true)<<"\n";
    vertex2.clear();
    for (int i = 0; i < vertex1.size(); i++)
        vertex2.push_back(vertex1[vertex1.size()-i-1] + Point(200, 200));
    cout << estimateRigidTransform(vertex1, vertex2, true)<<"\n";
LBerger gravatar imageLBerger ( 2017-06-06 04:37:11 -0500 )edit

Thanks again! If I edit your dst Mat like described here, i was able to map the points in your example to the right corespondence with

 cv::perspectiveTransform(vertex1,result,H);

which means I am still not 100% sure if the order in vector matters -but it seems to be unstable or luck. Anyhow, if i push more than 5 points in vertex1 dst is getting empty again, even if the bool fullAffine is set to true. I think that the function is limeted to a few points. if there is not other way to estimate a rigid transform in opencv with a lot of unsorted points,can i use findHomography (perhaps as overkill)?

JoeBroesel gravatar imageJoeBroesel ( 2017-06-06 07:53:53 -0500 )edit

May be you can try this.

LBerger gravatar imageLBerger ( 2017-06-06 08:47:04 -0500 )edit

Thank you, i will have a look. As a last question: is the order of points relevant for findHomography?

JoeBroesel gravatar imageJoeBroesel ( 2017-06-06 13:25:51 -0500 )edit
LBerger gravatar imageLBerger ( 2017-06-06 13:32:51 -0500 )edit

thanks for your code! really impressive performance, but it will take a while until I understand everything in detail!

JoeBroesel gravatar imageJoeBroesel ( 2017-06-07 13:16:35 -0500 )edit

A PR is in progress

LBerger gravatar imageLBerger ( 2017-06-07 13:24:11 -0500 )edit