Large scale and rotation is difficult for OF algorithms like the LK or Farneback. However, you may use previous estimation to warp current images toward the template image or the other way around.
I tested the speed of the LK and Farneback OF method in OpenCV. For calculating a dense flow field, the Farneback approach is in fact much faster than the LK algorithm. The reason that the LK is faster is because a sparse flow is usually computed.
