object detection and tracking at known background

edit

Hi! I am trying to code to track an object. Here are some information.

1. The background is complicating (many edges, objects) but mostly static (only slow changes such as sunlight). The object is mostly always moving and it moves much faster than the background does.
2. The approx. size and shape of the object is known. It is a stick with a known size. So I can expect the min/max size of the object when it appears to the image.
3. The stick (object) changes its location and often rotates a bit.
4. The color of the stick is mostly known. But its color is similar to the background of it.

I have tried to train LBP, but it did not work well due to true negatives. (I even tried to train the background as for the negative images without the object). Thus, I moved to a very classic method. color filtering, edge detection, and etc. Probably it will first filter with a color range, remove blobs which are inside an object, find contours with a similar shape. Here my questions is...

1. For removing blobs, I just have contours how. What should I do to fill small blobs? (_, contours, _) = cv2.findContours(frame, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
2. I have a rectangle frame which overs the object which may be rotated. How could I calculate the ratio of width and height? I got the rectangle from ... rect = np.int32(cv2.boxPoints(cv2.minAreaRect(cnt)))
3. The most important question is.... Do you think there is any good way/model to achieve my goal? Deep learning would not be possible considering the object's image is not easy to get.