If you are trying to estabilize the video, there is a stabilization module in OpenCV. You can check that out.
But if is not related to that, and you want a measure of the shakyness of the camera, you could do something like:
- Find and compute Features in consecutive frames.
- Match those features for every pair of consecutive frames.
- Cross-validate and eliminate bad matches.
- Find a Homography transformation between the features in one frame to the ones of the second frame, and reject outliers with RANSAC to achieve robustness for moving objects and matching errors.
- You can verify the rotation angle of the camera or maybe compute the magnitude of the rotation part of the homography matrix as a shakyness strength measure.
Every step has an OpenCV method, except for step 3 I think. But you can cross validate by yourself.