Looks like your intrinsics are not good. Try to calibrate intrinsics for each camera with calibrateCamera() and look at the undistorted image quality. If it's good, use these intrinsics: pass CV_CALIB_FIX_INTRINSIC to flags combination in stereoCalibrate(). Don't forget, that you must have the same image size from each camera, when using stereoCalibrate().