1 | initial version |
So I believe I see your problem.
You are applying the average motion to each new frame. Simplifying to one dimension, if your motion were
0, -5, 0, -5, 0, -5, 0, -5
Your average motion would be -2.5 You then correct every frame by +2.5, and everything is still jittery. What you need is to correct everything by the difference from -2.5, or rather, the difference from your average transformation matrix.
Hopefully that's enough to get you started.