I have access to the camera matrix, therefore I can select multiple points on the surface and reconstruct their 3D coordinates on the camera frame. But how do I use those coordinates to build a transformation matrix (mostly rotation, translation is irrelevant, I'd just need to orient my reference frame orthogonally with the surface) My main limitation seems to be that I do not have the correspondent coordinates on the surface points (on object/external reference), therefore can't use cv::estimateAffine3D or cv::findHomography or cv::solvePnP.