Camera calibration and aspect ratio

edit

The correct way is not using CALIB_FIX_ASPECT_RATIO flag or set this flag and apply aspect ratio in initial camera matrix. It is strange that first approach doesn't work in your case. Can you show your calibration code?

Also, for better accuracy I would recomend you calibrate camera with native resolution and then rescale camera matrix (distortion coefficents remain the same). For recovering scale (and maybe crop) parameters take some shots of calibration rig with native and working resolution. Then find transformation between corners coordinates on different resolution. The transformation will have a form:

new_x = (x - crop_x) * scale_x
new_y = (y - crop_y) * scale_y

Also note that crop_x and crop_y should be integer values and scale_x and scale_y are probably rational values. Finaly rescale camera matrix parameters:

new_fx = fx * scale_x
new_fy = fy * scale_y
new_cx = (cx - crop_x) * scale_x
new_cy = (cy - crop_y) * scale_y

UPD I just thought that your problem may be related to the wrong parameter newCameraMatrix in initUndistortRectifyMap. Use getOptimalNewCameraMatrix for this parameter.