OpenCV projectionPoints giving abnormal results

edit

I've did Camera calibration using fisheye model. I did both intrinsic and extrinsic calibration using checkerboard images and got the values K (Camera matrix), D(dist coeffs), R and T. Reprojection error for intrinsic calibration is ~2.45 and for extrinsic calibration it's ~33.62.

Now the task i'm trying to do is to obtain the uv mapping for the 3D points on a particular surface so that i can project the image on to the surface using openGL. To get the 2D uv point corresponding to the 3D point i'm using opencv fisheye models projectPoints() method projectPoints

Image i used for extrinsinsic calibration is of size 3024 * 3024 pixels

Functions i used for intrinsic and extrinsic calibration:

fisheye::calibrate method to get K and D:

double rms = fisheye::calibrate(objectPoints, imagePoints, imageSize, cameraMatrix,
                    distCoeffs, rvecs, tvecs, flags|cv::CALIB_FIX_K4|cv::CALIB_FIX_K5|
                    fisheye::CALIB_RECOMPUTE_EXTRINSIC,
                    criteria);

solvePnP method to get R and T:

bool ok = solvePnP(objectPoints,imagePoints,cameraMatrix,distCoeffs,rvecs,tvecs,false,flags);

I'm getting correct uv points corresponding to the 3D points which are projected on to the image. But when the uv points goes out of the image borders it's returning very deviated and uncorrelatable values like below:

projectPoints function call:

fisheye::projectPoints(mesh_points, uv_points, rvecs, tvecs,
                cameraMatrix, distCoeffs); //here mesh_points are 3D points corresponding to the surface

3D points i've passed to project points function:

1) 0.0 0.0 0.0
2) -11.2 0.0 0.0
3) -11.3 0.0 0.0

UV points i got in the result:

1)  1962.62 1030.9
2)  4734.26 390.099
3) -1781.13 2592.06

So for points (-11.2,0,0) and (-11.3,0,0) the difference between the uv points returned by projectPoints is huge. But ideally as these points are very close in 3D space their uv points should also be close to each other. I'm not able to understand this response from the projectPoints method. The uv point returned for (0,0,0) and (-11.2,0,0) are correct when i compared manually on the image but the uv point corresponding to (-11.3,0,0) is very far and incosistent from other uv points. I feel at the end it's a matrix multiplication that should be happened in project points method and that should be consisent.

Please help us if you've any information regarding this problem.