cv::fisheye::stereoRectify giving black image.

edit

I am trying to create a depth map from a fisheye stereo camera. I have intrinsic and extrinsic data from the camera SDK, so it can be trusted.

When I undistort the image using cv::fisheye::stereoRectify and then cv::fisheye::initUndistortRectifyMap and cv::remap, I get a black image.

When I print the P1 matrix after cv::fisheye::stereoRectify, it looks odd:

//before [284.49261, 0, 425.80771; 0, 285.6832, 395.49429; 0, 0, 1]

//after [108.90381630035, 0, 530.2634673508817, 0; 0, 108.90381630035, -8615.97446825623, 0; 0, 0, 1, 0]

Am I wrong in thinking that these two matrices should be the same? P1 = K1 * T, is that not correct?

If I use:

    cv::Mat lmapx, lmapy, rmapx, rmapy;
    cv::fisheye::initUndistortRectifyMap(K1, D1, cv::NoArray(), K1, imgSize, CV_16SC2, lmapx, lmapy);

the undistortion looks good. But I am stuck on the Rectify.

My code is:

leftImg = imread(leftFile_ss, cv::IMREAD_GRAYSCALE);
        rightImg = imread(rightFile_ss, cv::IMREAD_GRAYSCALE);

        //load cal data
        cv::Size imgSize(leftImg.cols, leftImg.rows);

        FileStorage fs("T265.xml", FileStorage::READ);
        cv::Mat K1, K2, D1, D2, R, T;
        fs["K1"] >> K1;
        fs["K2"] >> K2;
        fs["D1"] >> D1;
        fs["D2"] >> D2;
        fs["R"] >> R;
        fs["T"] >> T;

        std::cout << K1 << std::endl;

        double balance = 0.0, fov_scale = 1.0;
        cv::Mat R1, R2, P1, P2, Q;
        cv::fisheye::stereoRectify(K1, D1, K2, D2, imgSize, R, T, R1, R2, P1, P2, Q,
        cv::CALIB_ZERO_DISPARITY);

        std::cout << P1 << std::endl;


        cv::Mat lmapx, lmapy, rmapx, rmapy;
        cv::fisheye::initUndistortRectifyMap(K1, D1, R1, P1, imgSize, CV_16SC2, lmapx, lmapy);
        cv::Mat undist_left;
        cv::remap(leftImg, undist_left, lmapx, lmapy, cv::INTER_LINEAR);


        cv::imshow("undL", undist_left);
        cv::waitKey(0);

What am i doing wrong? How can i rectify these two images?

Thank you!