How to correct/remap a perspective effect after an undistort from camera calibration ?
Good evening,
I would like to remap a trapezoid ROI to a rectangular ROI after correcting a barrel distortion on a frame.
Here is my frame :
From this frame (and many others), I calculated the camera matrix and the distortion coefficients using the function cv::calibrateCamera.
I managed to "draw" the radial distortions which gives me this :
From what I understand, everything in the green area will be remapped into a rectangle with the size of my frame. Everything outside the green area will be considered "cropped".
To undistort the image, I am using cv::initUndistortRectifyMap with matrix R set as identity matrix, as well as cv::remap. The result of this operation is the following :
Now, I would like to remap the trapezoid chessboard into a rectangular chessboard of frame size. I could use cv::getPerspectiveTransform and cv::warpPerspective to remap from trapezoid to rectangle but it would only stretch the image and this is not the result I would expect.
In fact, I would like to know if we could find a much better warp or process knowing the translations and rotations matrices from the camera calibration. The purpose of this is to keep a certain aspect of "depth" and "position" from the image but processing a rectangular ROI (to find contours, get their real positions, etc ...)
Thanks in advance for your help !
EDIT (17/11/17) (see Der Luftmensch comment below) :
I provided two drawings (with my best Paint.NET skills) to express my needs.
I would like to track coins inside a region of interest. The main parameters I would like to track are their sizes and their positions (it is not about object detection here).
Representing the camera and the ROI
To get very good results, I need to calibrate my camera correctly because it has an awful barrel distortion and it distorts the region of interest and the coins shapes in consequence. So once I calibrated my camera, the ROI is still seen as a trapezoid but I would need to remap this ROI into a rectangle of the size of a frame (let's say 640x480).
As I mentionned above, I don't think that using warpPerspective would be a good idea because it would also distort the coins shapes and "fake" their real positions. I don't really know if it is about according ratios correctly or using specific matrices calculations to get the best result. I've heard about translation matrices and rotation matrices but I don't understand how can I get them.
The parameters that I know are the corners of the ROI (because my region doesn't move at all), the rotation angle of the camera (see the second link above), the distance between my camera and the center of the ROI, the world size of the ROI (in centimeters), etc. Coins have different sizes because of the depth field and this is a thing I would care about.
I would like to ...
This is very interesting. Care to share your code?
Would you please add some explanation to your last paragraph "The purpose of this is to keep a certain aspect of "depth" and "position" from the image..". Do you know the image position of the chessboard corners? Do you know the real-world aspect-ratio of the chessboard? If so, it's straightforward to use
cv::findHomography().@sjhalayka Sure, here is the the code : https://gist.github.com/MajorSquirrel...
This is very close to the camera calibration example provided by OpenCV itself, except that I cleaned it to my needs and I added the drawn points (in green) to better visualize what's happening.
@Der Luftmensch I've updated my question above with more details and awesome visual drawings. However, I would like to notice that I've tried using findHomography between reading your answer and posting my update and I've found out that findHomography and getPerspectiveTransform give me the same results. But please, can you take a look at my update ?
Wow, complicated, but not. Thanks for the code!
You have a static camera and a static background (only coins are changing position) and you know the scale of your chessboard in real-world size and you are hoping to identify the size and location objects on the chessboard? I think
cv::warpPerspective()is what you want. Warp the image and your coins will appear as circles which can easily be identified from the background. More, because you know the relationship between pixels and m (or cm, or mm) you can easily compute the real size of the coins. Because you are doing everything on a plane (flat board and flat coins) I don't see any reason to be concerned with a third dimension. Given all of that, why no put the camera directly overhead? The results are likely to be better.@Major Squirrel Any chance that you could simplify the code? I'me expecting the program to load "chess.jpg", which is the only frame, and yet the program expects 3 arguments. It should take no arguments.
@Der Luftmensch I'm sorry to bring an answer two weeks later, I've been quite busy these days.
I've tried using findHomography and warpPerspective by keeping the same ratio between my ROI and my frame, it works well now, thank you for that ! Also, the reason I didn't put the camera directly overhead is because I simply can't, it's a constraint haha.
Is there anyway to improve results of rewarping by calculating matrices manually, given multiple parameters in world coordinates (such as distances, angles, etc), instead of guessing with findHomography ?
@Major Squirrel even I am having the same question. Did you figure out how to do perspective transformation with the known rotational and translational matrices? I.e project the image orthogonal to the camera knowing the intrinsic and extrinsic matrices?
I used the following formulae to obtain the homographic matrix from the camera intrinsic and extrinsic matrices. But when I warped the image using warpPerspective() I am unable to transform it.
Any idea on how to orthogonaly project the image using camera matrices?
Homography H = K [r1 r2 t1] ,where K is intrinsic camera matrix, r1,r2,t1 are column vectors of the extrinsic matrix. (Neglecting r3 because z = 0 according to zhang's paper
Hello @abhijit ! I didn't go further as I use findHomography() to get the matrix instead of going with the rotation and translation matrices. :|