Turning ArUco marker in parallel with camera plane

edit

I have written in this answer some experimentations I did to understand more the concept of homography. Even if this is not really an answer of the original post, I hope it could also be useful to other people and it is a good way for me to summarize all the information I gathered. I have also added the necessary code to check and make the link between the theory and the practice.

What is the homography matrix?

For the theory, just refer to a computer vision course (e.g. Lecture 16: Planar Homographies, ...) or book (e.g. Multiple View Geometry in Computer Vision, Computer Vision: Algorithms and Applications, ...). Quickly, the planar homography relates the transformation between two planes (up to a scale):

This planar transformation can be between:

• a planar object and the image plane (image from here, p9):

• a planar surface viewed by two cameras (image from here, p56 and here, p10):

• a rotating camera around its axis of projection, equivalent to consider that the points are on a plane at infinity (image from here, p11):

How the homography can be useful?

• Camera pose estimation with coplanar points (see here or here, p30), the homography matrix can be estimated using the DLT (Direct Linear Transform) algorithm
• Perspective removal, correction:
• Panorama stitching:

Demo 1: perspective correction

The function findChessboardCorners() returns the chessboard corners location (the left image is the source, the right image is the desired perspective view):

The homography matrix can be estimated with findHomography() or getPerspectiveTransform():

H:
[0.3290339333220102, -1.244138808862929, 536.4769088231476;
 0.6969763913334048, -0.08935909072571532, -80.34068504082408;
 0.00040511729592961, -0.001079740100565012, 0.9999999999999999]

The first image can be warped to the desired perspective view using warpPerspective() (left: desired perspective view, right: left image warped):

Demo 2: compute the homography matrix from the camera displacement

With the function solvePnP(), we can estimate the camera poses (rvec1, tvec1 and rvec2, tvec2) for the two images and draw the corresponding object frames:

• Camera pose for the first camera:
• Camera pose for the second camera:
• Homogeneous transformation between the two cameras:

It is then possible to use the camera poses information to compute the homography transformation related to a specific object plane:

By Homography-transl.svg: Per Rosengren derivative work: Appoose (Homography-transl.svg) CC BY 3.0, via Wikimedia Commons

On this figure, n is the normal vector of the plane and d the distance between the camera frame and the plane along the plane normal. The equation to compute the homography from the camera displacement is:

Where is the homography matrix that maps the points in the first camera frame to the corresponding points in the second camera frame, is the rotation matrix that represents the rotation between the two camera frames and the translation vector between the two camera frames.

Here the normal vector n is the plane normal expressed in the camera frame 1 and can be computed as the cross product of 2 vectors (using 3 non collinear points that lie on the plane) or in ...