Obtaining canonical projected images for single camera given calibration and extrinsic

edit

I'm, trying to implement algorithm find in https://pdfs.semanticscholar.org/8fc2... using openCv, and I can' get around obtaining frontal images from my camera, after calibration and rectification some position are "cropped" and miss image plane. Input image: https://imageshack.com/a/img922/2971/...

Example of one of the wrong position: https://imageshack.com/a/img923/3940/...

cv::Mat distCoeffs;
std::vector<cv::Mat> rvecs, tvecs;
distCoeffs = cv::Mat::zeros(8, 1, CV_64F);

double EPS = 0.00001;
cv::TermCriteria Terms(cv::TermCriteria::COUNT + cv::TermCriteria::EPS, 30, EPS);

int CalibrationControlParameters = 0;

cv::Mat cameraMatrix = cv::Mat::eye(3, 3, CV_64F);

double REPROJECTION = cv::calibrateCamera(Model, KeyPoints, cv::Size(PureImages[0].cols, PureImages[0].rows), 
                                        cameraMatrix, distCoeffs, rvecs, tvecs,
                                        CalibrationControlParameters, Terms);

double alfa = 0;

cv::Mat NewCamera = cv::getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, PureImages[0].size(), alfa,cv::Size(),(cv::Rect*)0,false);

for (int image_num = 0; image_num < PureImages.size(); ++image_num) {

    cv::Mat RotationMatrixCV(3, 3, CV_64FC1);
    cv::Rodrigues(rvecs[image_num], RotationMatrixCV);

    Eigen::Vector3d Translations;
    cv::cv2eigen(tvecs[image_num], Translations);

    cv::Mat CurrentCamera = NewCamera.clone();
    Eigen::Matrix3d CamEig;
    cv::cv2eigen(CurrentCamera, CamEig);

    CamEig(0, 2) += Translations(0);
    CamEig(1, 2) += Translations(1);
    cv::eigen2cv(CamEig, CurrentCamera);

    cv::initUndistortRectifyMap(cameraMatrix, distCoeffs, RotationMatrixCV, CurrentCamera, PureImages[0].size(), CV_32FC1, map1, map2);


    cv::remap(PureImages[image_num], Transformed, map1, map2, 1);

    cv::imwrite(_FolderPath + "perspective" + _PathToImages[image_num], Transformed);
}

Translation can by apply by adding to center of image of CurrentCamera, but I' can't came up with formula with will center all image simultenuisly. Reprojection error is around 0.7 pix ( some images are severly blured), but calibration isn't singular.

Using OpenCv 3.3 and Eigen