As Steven suggested, the VOT challenge paper is an excellent source to start looking for a state-of-the-art tracker.
This is the VOT2017 paper and here there is more related material.
I'm giving a chance to CSRT (in the paper it's called CSRDCF++ - Discriminative Correlation Filter with Channel and Spatial Reliability) because it should be the winner of the real-time category and it has already been merged into OpenCV. I say "it should" because, as stated in the paper, the actual winner is SiamFC (Fully-Convolutional Siamese Networks for Object Tracking) but looking at the videos, CSRT seems to be a little bit more accurate during tracking.
«The best performing real-time trackers is CSRDCF++ (A.40), but this tracker is co-authored by the VOT organizers. According to the VOT winner rules, the winning real-time tracker of the VOT2017 is SiamFC (A.21).»
And this is a short video showing the performace (impressive IMHO) of the CSRT algorithm.
Take a look at the VOT tracking challenge results 2016: http://data.votchallenge.net/vot2016/...
Great suggestion, thank you! I think that maybe CSRT (already in OpenCV contrib) maybe what I was looking for.