I am working on stereo rectification of wide FOV fisheye lens. I want to do stereo rectification in latitude and longitude space so stretching at the edges can be avoided. Hence I am using opencv omndir package with RECTIFY_LONGLATI flag for stereo rectification. I am getting good results but the most of the portion of image is cropped. I want to scale the camera matrix such that no region is cropped while stereo rectification. I don't know how to scale the camera matrix keeping epipolar constraint for stereo rectification. I have found functions for estimating new camera matrix in calibration and fisheye module but not in omnidir module.
I tried to scale (double) the focal length in order to check can i get whole image without cropping. I am getting both left and right image without cropping but stereo rectification is disturbed. I know just scaling the focal length will disturb the rectification. Principal point also need to scaled considering distortions coefficients. I don't know how to do this, hence I was searching for function similar to getoptimalnewcameramatrix in omndir package. I will be helpful If anyone can help me with the maths behind scaling camera matrix and keeping epipolar constraint in stereo rectification.
I am attaching small portion of my code, camera calibration output file and results below.
rectified image without scaling camera matrix
non rectified image with focal length scaling
code snippet.
Mat left_img = imread("../data/left42.png");
Mat right_img = imread("../data/right42.png");
Mat left_img_undistorted, right_img_undistorted;
Mat R1, R2;
cv::omnidir::stereoRectify(rvec, tvec, R1, R2);
int flags_ = cv::omnidir::RECTIFY_LONGLATI;
cv::Size imgSize = left_img.size();
// K1 and K2 are estimated camera matrix from calibration.
// D1 and D2 are estimated distortion coefficients from calibration
// xi11 and xi2 are estimated Mei's coefficients from calibration
Mat K1_ = K1.clone();
Mat K2_ = K2.clone();
// scaling focal length
K1_.at<double>(0,0) = K1_.at<double>(0,0)*2;
K1_.at<double>(1,1) = K1_.at<double>(1,1)*2;
cv::Matx33d Knew(imgSize.width / (3.1415), 0, 0, 0, imgSize.height /(3.1415), 0, 0, 0, 1);
cv::omnidir::undistortImage(left_img, left_img_undistorted, K1_, D1, xi1, flags_, Knew, imgSize, R1);
cv::omnidir::undistortImage(right_img, right_img_undistorted, K2_, D2, xi2, flags_, Knew, imgSize, R2);
Mat final;
hconcat(left_img_undistorted, right_img_undistorted, final);
for( int j = 0; j < final.rows; j += 40 )
line(final, Point(0, j), Point(final.cols, j), Scalar(0, 255, 0), 1, 8);
imwrite("../out/rectified.png", final);
}