First time here? Check out the FAQ!
I am having similar issues, my guess is that the maps generated with initUndistortRectifyMap() are only used for inverse