Fisheye undistorted image, does the camera matrix change?

edit

I am trying to run solvePnP on a series of pre-mapped AruCo markers (stored as a opencv::Aruco::Board) with a fisheye camera, something I have done many times with other cameras.

The intrinsics are from onboard the camera (a realsense t265), so they can be trusted.

Using the images as fisheye, I get a terrible result, with a re-projection error in the thousands.

Using the following to undistort:

Mat camera_matrix = camera_intrinsics_map[sn];
            Mat dist_coeffs = dist_coeffs_map[sn];

            Mat identity = Mat::eye(3, 3, CV_64F);

            // Camera 1 Projection Matrix K[I|0]
            cv::Mat PP1(3, 4, cv::DataType<float>::type);
            camera_matrix.copyTo(PP1.rowRange(0, 3).colRange(0, 3));  

            //dist as 4 x 1
            Mat distortionCoefficients = (Mat1d(4, 1) << dist_coeffs.at<double>(0,0), dist_coeffs.at<double>(1, 0), dist_coeffs.at<double>(2, 0), dist_coeffs.at<double>(3, 0));

            cv::Mat lmapx, lmapy, rmapx, rmapy;
            cv::fisheye::initUndistortRectifyMap(camera_matrix, distortionCoefficients, identity, PP1, Size(848, 800), CV_16SC2, lmapx, lmapy);
            cv::Mat undist_left;
            cv::remap(image, undist_left, lmapx, lmapy, cv::INTER_LINEAR);

I detect the markers and run solvePnP again.

cv::Mat rvec, tvec;

        //fake zero distortion mat because we are rectified

        Mat fake_dist = Mat::eye(4, 1, CV_64F);

        images[time_idx][sn] = undist_left;
        vector<int> ids;
        vector<vector<Point2f>> corners, rejected, undists;

        // detect markers
        aruco::detectMarkers(undist_left, dictionary2, corners, ids, detectorParams2);
        if (ids.size() == 0) continue;

        // estimate board pose
        int markersOfBoardDetected = 0;
        if (ids.size() > 0)
        {
            // get object and image points for the solvePnP function
            Mat objPoints, imgPoints;
            std::vector<cv::Point3f> p3d;
            std::vector<cv::Point2f> p2d;

                           //gets the 3d / 2d correspondences.
            getBoardObjectAndImagePointsLocal(boardMap2, corners, ids, objPoints, imgPoints, p3d, p2d);

            if (objPoints.total() == 0)
                return;

            solvePnP(objPoints, imgPoints, camera_matrix, fake_dist, rvec, tvec, false, cv::SOLVEPNP_ITERATIVE);

The result is better, but nowhere near perfect. I would expect a very good result, as the 3d points are right, the correspondences are perfect and the intrinsics are known.

My question is, because I have undistorted the image, does the change the camera matrix values?

Could this be the issue? Or am I missing something?