initStereoRectify does not handle distortion correctly

edit

System information (version)

• OpenCV => 3.4.1
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2013 + Python Bindings

Detailed description

When using stereoRectify, the resulting projection matrices depend on the input distortion, which in my understanding should not happen.

I encountered the problem when using stereoRectify on a new pair of cameras. I have used the whole pipeline successfully before on cameras of the same type. This time, the results returned by stereRectify are wrong. The camera centers in the new projection matrices have huge values (see values below).

To locate the problem, I run the rectification and undistortion in two individual steps, instead of using stereRectify. If I first undistort the images and then feed the new undistorted images and a vector filled with zeros as distortion into stereRectify I obtain good results. See the Steps to reproduce.

Steps to reproduce

My data input:

M_L = [[ 430.            0.          200.23631705]
 [   0.          430.          199.36536477]
 [   0.            0.            1.        ]]
M_L.dtype = float64

dist_L = [-0.52501353  0.50232865  0.01204062 -0.03013492 -0.42135838]
dist_L.dtype = float64

M_R = [[ 430.            0.          199.64316951]
 [   0.          430.          199.40004749]
 [   0.            0.            1.        ]]
M_R.dtype = float64

dist_R = [-0.44271713  0.01572439  0.01272985 -0.02997997  0.31224365]
dist_R.dtype = float64

R = [[ 0.99757046  0.06468998 -0.02585312]
 [-0.06438798  0.99784853  0.01234882]
 [ 0.02659635 -0.01065419  0.99958948]]
R.dtype = float64

T = [[-0.23299669]
 [-3.05171467]
 [ 0.43562338]]
T.dtype = float64

The results of stereoRectify using

size = image_L.shape[:2]
R_rectify_L, R_rectify_R, P_L, P_R, Q, validPixROI_L, validPixROI_R = cv.stereoRectify(M_L,
            dist_L, M_R, dist_R, size, R, T)
P_L = [[  4.30000000e+02   0.00000000e+00   2.68848224e+03   0.00000000e+00]
 [  0.00000000e+00   4.30000000e+02  -2.09073371e+03   0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00]]
P_L.dtype = float64

P_R = [[  4.30000000e+02   0.00000000e+00   2.68848224e+03   0.00000000e+00]
 [  0.00000000e+00   4.30000000e+02  -2.09073371e+03  -1.32932034e+03]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00]]
P_R.dtype = float64

The values for pixel centers in P_L and P_R are factor 10 to large. Using these results as in

map1_L, map2_L = cv.initUndistortRectifyMap(M_L, dist_L,
                                            R_rectify_L, P_L, size, cv.CV_32FC1)
image_rectified_L = cv.remap(image_L, map1_L, map2_L, cv.INTER_LINEAR)

map1_R, map2_R = cv.initUndistortRectifyMap(M_R, dist_R,
                                            R_rectify_R, P_R, size, cv.CV_32FC1)
image_rectified_R = cv.remap(image_R, map1_R, map2_R, cv.INTER_LINEAR)

gives completely black images. However, if I separate the undistort step like:

map1_L, map2_L = cv.initUndistortRectifyMap(M_L, dist_L, np.eye(3),
                                            M_L, size, cv.CV_32FC1)
image_interp_L = cv.remap(image_L, map1_L, map2_L, cv.INTER_LINEAR)

map1_R, map2_R = cv.initUndistortRectifyMap(M_R, dist_R, np.eye(3),
                                            M_R, size, cv.CV_32FC1)
image_interp_R = cv.remap(image_R, map1_R, map2_R, cv.INTER_LINEAR)

and than use stereoRectify on the undistorted images as in:

R_rectify_L, R_rectify_R, P_L, P_R, Q, validPixROI_L, validPixROI_R = cv.stereoRectify(M_L, 
                                    np.zeros((1,5)), M_R, np.zeros((1,5)), size, R, T)

map1_L, map2_L = cv.initUndistortRectifyMap(M_L, np.zeros((1,5)),
                                            R_rectify_L, P_L, size, cv ...