# How to extract Angle , Scale , transition and shear for rotated and scaled object

Problem description

I have rotated and scaled scene and need to correct the scale and rotation , then find rectangle known object in the last fixed image

Input

-Image scense from Camera or scanner
-Normalized (normal scale and 0 degree rotation )templeate image for known object


Requiered output

1-correct the scale and the rotation for the input scene
2-find rectnagle


the following figure explain what is the input and steps to find the output

I 'm using the following sample [Features2D + Homography to find a known object] (http://docs.opencv.org/2.4/doc/tutori...) to find rotated and scaled object .

I used the following code to do the process

//read the input image
if( img_scene.empty() || img_object.empty())
{
}
//Step 1 Find the object in the scene and find H matrix
//-- 1: Detect the keypoints using SURF Detector
int minHessian = 400;
SurfFeatureDetector detector( minHessian );
std::vector<KeyPoint> keypoints_object, keypoints_scene;
detector.detect( img_object, keypoints_object );
detector.detect( img_scene, keypoints_scene );

//-- 2: Calculate descriptors (feature vectors)
SurfDescriptorExtractor extractor;
Mat descriptors_object, descriptors_scene;
extractor.compute( img_object, keypoints_object, descriptors_object );
extractor.compute( img_scene, keypoints_scene, descriptors_scene );

//-- 3: Matching descriptor vectors using FLANN matcher
FlannBasedMatcher matcher;
std::vector< DMatch > matches;
matcher.match( descriptors_object, descriptors_scene, matches );

double max_dist = 0; double min_dist = 100;
//-- Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors_object.rows; i++ )
{
double dist = matches[i].distance;
if( dist < min_dist )
min_dist = dist;
if( dist > max_dist )
max_dist = dist;
}

//-- Draw only "good" matches (i.e. whose distance is less than 3*min_dist )
std::vector< DMatch > good_matches;
for( int i = 0; i < descriptors_object.rows; i++ )
{
if( matches[i].distance < 3*min_dist )
{
good_matches.push_back( matches[i]);
}
}
Mat img_matches;
drawMatches( img_object, keypoints_object, img_scene, keypoints_scene,
good_matches, img_matches, Scalar::all(-1), Scalar::all(-1),
vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );

//Draw matched points
imwrite("c:\\temp\\Matched_Pints.png",img_matches);

//-- Localize the object
std::vector<Point2f> obj;
std::vector<Point2f> scene;
for( int i = 0; i < good_matches.size(); i++ )
{
//-- Get the keypoints from the good matches
obj.push_back( keypoints_object[ good_matches[i].queryIdx ].pt );
scene.push_back( keypoints_scene[ good_matches[i].trainIdx ].pt );
}

Mat H = findHomography( obj, scene, CV_RANSAC );
//-- Get the corners from the image_1 ( the object to be "detected" )
std::vector<Point2f> obj_corners(4);
obj_corners[0] = cvPoint(0,0); obj_corners[1] = cvPoint( img_object.cols, 0 );
obj_corners[2] = cvPoint( img_object.cols, img_object.rows ); obj_corners[3] = cvPoint( 0, img_object.rows );
std::vector<Point2f> scene_corners(4);
perspectiveTransform( obj_corners, scene_corners, H);
//-- Draw lines between the corners (the mapped object in the scene - image_2 )
line( img_matches, scene_corners[0] + Point2f( img_object.cols, 0), scene_corners[1] + Point2f( img_object.cols, 0), Scalar(0, 255, 0), 4 );
line( img_matches, scene_corners[1] + Point2f( img_object.cols, 0), scene_corners[2] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
line( img_matches, scene_corners[2] + Point2f( img_object.cols, 0), scene_corners[3] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
line( img_matches, scene_corners[3] + Point2f( img_object.cols, 0), scene_corners[0] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
//-- Show detected matches
//imshow( "Good Matches & Object detection", img_matches );
imwrite ...
edit retag close merge delete

Perhaps I'm missing something, but why not just do warpPerspective with the WARP_INVERSE_MAP flag? Why do you need to decompose the homography matrix?

( 2016-11-28 12:05:34 -0500 )edit

Thanks @Tetragramm , i don't know that WARP_INVERSE_MAP will recover my original image , i will try it but how get the final object rectangle in the recovered image , which function will do calculate the rectangle.

i was trying to do the mathematics by my self , you saved my time

( 2016-11-28 12:32:42 -0500 )edit

HI @Tetragramm , i have used warpPerspective with the WARP_INVERSE_MAP to get the recovered image after removing the rotation ,scale ,shear and transition , the rotation corrected but the image is shifted , also how i can calculate the recovered image size , i will add images to the question to understand what i mean

( 2016-11-29 10:09:54 -0500 )edit

So, I'm not quite sure what you're trying to do, but I'm sure we can do it all without decomposing the homography.

The output image there looks like you successfully transformed from the scene to the model. If you made the output the same size as the model image, you would just have the scene box matching the model box, ready for denoising or whatever you're doing.

Then to put it back into your image, you just use warpPerspective again, but this time without the INVERSE_MAP flag, and with the BORDER_TRANSPARENT type to place it back where it originally came from.

Am I correct that that is what you wanted to do, just that small box? If you're trying to do something with the whole permit, we'll need to change it up a little.

( 2016-11-29 11:02:38 -0500 )edit

what i'm doing as follow

Input is Scaled and rotated image scene
Output required : Correct the scale and rotation and put the result in the recovered image , i need to get the coordinates of the object image inside the recovered image .
by knowing the object coordinates i will get all text coordinates by using relative relation between them,

( 2016-11-29 11:26:50 -0500 )edit

I had tried warpPerspective + BORDER_TRANSPARENT without INVERSE_MAP but the result is flipped in y axis . using warpPerspective with WARP_INVERSE_MAP gives near to correct result , but the starting point of the recovered image (top left corner) is object top left corner not the scene top left corner, my be we need to recalculate the Homography Matrix to change the top left corner?

( 2016-11-29 12:12:46 -0500 )edit

Sort by » oldest newest most voted

Ah, ok. Create a vector of Point2f, one located at each corner of your model, or each relevant location in your model. Then use the transform function to map those onto your document. The output should be a vector of Point2f that are those points on the document.

So if your input is the corners of the model, then your output is the corners of the box in the image.

If you need to do the inverse, it's just H.inv(), I think.

EDIT: I'm really not sure what you're doing wrong, because I don't have enough of your code, but here. This code does what I think you want done.

Mat warped;
warpPerspective(scene, warped, h, Size(model.cols, model.rows), WARP_INVERSE_MAP);
imshow("Warped", warped);

Mat modifiedScene = scene.clone();
warpPerspective(model, modifiedScene, h, Size(scene.cols, scene.rows), 1, BORDER_TRANSPARENT);
imshow("Modified Scene", modifiedScene);

vector<Point2f> ptsModel, ptsScene;
vector<Point3f> tempPts;
ptsModel.push_back(Point2f(0, 0));
ptsModel.push_back(Point2f(model.cols, 0));
ptsModel.push_back(Point2f(0, model.rows));
ptsModel.push_back(Point2f(model.cols, model.rows));
transform(ptsModel, tempPts, h);
convertPointsHomogeneous(tempPts, ptsScene);

for (int i = 0; i < ptsScene.size(); ++i)
std::cout << ptsScene[i] << "\n";
std::cout << "\n";

line(scene, ptsScene[0], ptsScene[1], Scalar(255, 0, 0));
line(scene, ptsScene[1], ptsScene[3], Scalar(255, 0, 0));
line(scene, ptsScene[2], ptsScene[3], Scalar(255, 0, 0));
line(scene, ptsScene[2], ptsScene[0], Scalar(255, 0, 0));
imshow("Scene", scene);

waitKey();

more

still can not use warpPerspective with WARP_INVERSE_MAP to correct scale and rotation for the whole scene image only part of scene will be corrected, also i used

std::vector<point2f> Recovered_corners(4);
perspectiveTransform( scene_corners, Recovered_corners, H.inv());


Recovered_corners gives the standalone object coordinates not the object coordinates in the recovered scene it seems we need to decompose and adapt H matrix to the new recovered scene , i found the following paper which describe how to decompose Himograhpy matrix H How to decompose Homography i think it's complicated and need some time to understand and implement. now i will use another method to detect and fix the scale and rot

( 2016-11-30 05:49:20 -0500 )edit

Hi @Tetragramm it seems there is little bit miss understand between us, i need to find the corrected scale and rotation for the whole scene , also i need to find rectangle of an object in the final fixed scene , i had updated the question and added image describe the steps and what is expected , also i added the code .

In your code i'm interested in warped image;

warpPerspective(scene, warped, h, Size(model.cols, model.rows), WARP_INVERSE_MAP);

but i need the whole scene to be warpped

then in the final warped image i need to find rectangle of the object see the updated question

( 2016-12-01 06:46:08 -0500 )edit

If you can not see the text in the image which describe the problem and the expected solution use "Ctrl + " in your keyboard to zoom in the image

( 2016-12-01 06:50:36 -0500 )edit

To get your desired final output, you were almost there. You just need to add an offset to your model. You can add it to the third column, first (x) and second (y) rows of the homography. The homography is the warp to the model, but your model is not actually where you want it, so you need to add the offset between where you want it and where it is.

( 2016-12-01 12:21:43 -0500 )edit

Here i will explain method to decompose the transformation matrix H , as in the following two articles
Math ,code

here it's my trial code

//read the input image
if( img_scene.empty() || img_object.empty())
{
}
//Step 1 Find the object in the scene and find H matrix
//-- 1: Detect the keypoints using SURF Detector
int minHessian = 400;
SurfFeatureDetector detector( minHessian );
std::vector<KeyPoint> keypoints_object, keypoints_scene;
detector.detect( img_object, keypoints_object );
detector.detect( img_scene, keypoints_scene );

//-- 2: Calculate descriptors (feature vectors)
SurfDescriptorExtractor extractor;
Mat descriptors_object, descriptors_scene;
extractor.compute( img_object, keypoints_object, descriptors_object );
extractor.compute( img_scene, keypoints_scene, descriptors_scene );

//-- 3: Matching descriptor vectors using FLANN matcher
FlannBasedMatcher matcher;
std::vector< DMatch > matches;
matcher.match( descriptors_object, descriptors_scene, matches );

double max_dist = 0; double min_dist = 100;
//-- Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors_object.rows; i++ )
{
double dist = matches[i].distance;
if( dist < min_dist )
min_dist = dist;
if( dist > max_dist )
max_dist = dist;
}

//-- Draw only "good" matches (i.e. whose distance is less than 3*min_dist )
std::vector< DMatch > good_matches;
for( int i = 0; i < descriptors_object.rows; i++ )
{
if( matches[i].distance < 3*min_dist )
{
good_matches.push_back( matches[i]);
}
}
Mat img_matches;
drawMatches( img_object, keypoints_object, img_scene, keypoints_scene,
good_matches, img_matches, Scalar::all(-1), Scalar::all(-1),
vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );

//Draw matched points
imwrite("c:\\temp\\Matched_Pints.png",img_matches);

//-- Localize the object
std::vector<Point2f> obj;
std::vector<Point2f> scene;
for( int i = 0; i < good_matches.size(); i++ )
{
//-- Get the keypoints from the good matches
obj.push_back( keypoints_object[ good_matches[i].queryIdx ].pt );
scene.push_back( keypoints_scene[ good_matches[i].trainIdx ].pt );
}

Mat H = findHomography( obj, scene, CV_RANSAC );
//-- Get the corners from the image_1 ( the object to be "detected" )
std::vector<Point2f> obj_corners(4);
obj_corners[0] = cvPoint(0,0); obj_corners[1] = cvPoint( img_object.cols, 0 );
obj_corners[2] = cvPoint( img_object.cols, img_object.rows ); obj_corners[3] = cvPoint( 0, img_object.rows );
std::vector<Point2f> scene_corners(4);
perspectiveTransform( obj_corners, scene_corners, H);
//-- Draw lines between the corners (the mapped object in the scene - image_2 )
line( img_matches, scene_corners[0] + Point2f( img_object.cols, 0), scene_corners[1] + Point2f( img_object.cols, 0), Scalar(0, 255, 0), 4 );
line( img_matches, scene_corners[1] + Point2f( img_object.cols, 0), scene_corners[2] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
line( img_matches, scene_corners[2] + Point2f( img_object.cols, 0), scene_corners[3] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
line( img_matches, scene_corners[3] + Point2f( img_object.cols, 0), scene_corners[0] + Point2f( img_object.cols, 0), Scalar( 0, 255, 0), 4 );
//-- Show detected matches
//imshow( "Good Matches & Object detection", img_matches );
imwrite("c:\\temp\\Object_detection_result.png",img_matches);

//Step 2 correct the scene scale and rotation and locate object in the recovered scene
Mat img_Recovered;
//1-decompose find the H matrix
float a = H.at<double>(0,0);
float b = H.at<double>(0,1);
float c = H.at<double>(0,2);
float d = H.at<double>(1,0);
float e = H.at<double>(1,1);
float f = H.at<double>(1,2);

float p = sqrt(a*a + b*b);
float r = (a ...
more

Official site

GitHub

Wiki

Documentation