Real-Time Multi-Object Tracking with Learning

edit

My Goal is to have real-time MultiTracker with Learning.
I used Kalman filter to track an object but i found errors in the estimation while tracking.
The object was not been tracked continuously. I want to implement some learning mechanism along with Tracking.
One way i thought of doing this is,
1) calculate the average HSV of a particular roi then store that HSV value in a vector(Scalar or Vec3b)
2) Compare the new HSV value (average from some ROI) with all previous HSV values present in vector collection.
3)If the new HSV value did not match with the HSV values in vector then track this as a new separate object.
4) else if the new roi matched HSV values in vector, then it is said to be the same object present in the roi. continue tracking old object.
5) Have some regular time based checking to remove old HSV values in vector.

I tried KCF, MIL e.t.c they are not realtime. can you recommend any realtime learning mecahnism or ways to improve above proposed one.