Strange streching effects in photo due to transformation by matrix

edit

I am working on stitching algorithm and and sometime there is strange streching of photo. It is cause by low number of keypoints BUT I am interested what cause it in this matrix:

and the effect of this matrix is this: https://imgur.com/a/dwzm9Af (sorry can not upload image here, it seems too big)

can you suggest how to reduce this effect or what cause it?