calcOpticalFlowPyrLK losing single user-defined tracking point

asked 2013-12-15 18:32:13 -0500

lhotchkiss gravatar image

I am trying to write a python program to track a user's head movement based on a single point between their eyes. I am using OpenCV 2.4.5 and calcOpticalFlowPyrLK. I set the initial point using HAAR points, and the initial point is correct, but after the first call to calcOpticalFlowPyrLK the program is now tracking a completely different point. I'm not very familiar with computer vision, so I can't figure out why its losing the initial point.

I'm using python 2.7, Fedora 18, and OpenCV 2.4.5.

My code:

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

# loading the classifiers
hcf = cv2.CascadeClassifier('/usr/share/OpenCV/haarcascades/haarcascade_frontalface_default.xml')
hce = cv2.CascadeClassifier('/usr/share/OpenCV/haarcascades/haarcascade_eye.xml')

# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 100,
                       qualityLevel = 0.3,
                       minDistance = 7,
                       blockSize = 7 )

# Parameters for lucas kanade optical flow
lk_params = dict( winSize  = (15,15),
                  maxLevel = 2,
                  criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

# Create some random colors
color = np.random.randint(0,255,(100,3))

# Take first frame and find corners in it
ret, old_frame =
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)

# Create a mask image for drawing purposes
mask = np.zeros_like(old_frame)

first = True
    #Get new frame
    ret,frame =
    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    #First iteration, calcualte the face
        first = False
        faces = hcf.detectMultiScale(frame_gray, minSize=(100, 100))
        face = faces[0]
        print face[0],face[1],face[2],face[3]

        p0 = [[((face[0] + (face[2]/2)),(face[1] + (face[3]/3)))]]
        p0 = np.float32(np.asarray(p0))
        print "p0", p0

    #cv2.rectangle(frame,(face[0], face[1]), (face[0] + face[2], face[1] + face[3]),(155, 255, 25),2)
    #Draw circle at initial point,((face[0] + (face[2]/2)),(face[1] + (face[3]/3))),5,(155, 255, 25))

    # calculate optical flow
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

    # Select good points
    if st==1:
        good_new = p1[st==1]
        good_old = p0[st==1]

        # draw the tracks
        for i,(new,old) in enumerate(zip(good_new,good_old)):
            a,b = new.ravel()
            c,d = old.ravel()
            mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)

    k = cv2.waitKey(30) & 0xff
    if k == 27:

    # Now update the previous frame and previous points
    old_gray = frame_gray.copy()
    if st==1:
        p0 = good_new.reshape(-1,1,2)

edit retag flag offensive close merge delete


Does your tracker lose track of points if they are corners detected by cv2.goodFeaturesToTrack? I ask this because Lucas-Kanade optical flow generally works well with corner points. The point that you specified might be featureless (i.e. flat with same textures and no strong gradients). See here for more details:

pyro gravatar imagepyro ( 2013-12-15 19:53:39 -0500 )edit

When I use that code from that link, the tracker identifies many points, and seems to follow them correctly. I actually used that code to base my code off of. The point that I specify in my code is the area directly between the eyebrows above the bridge of the nose, which in general doesn't have any distinct features. After setting the initial point to this location, the tracking point immediately jumps to another area, often my hair, eyebrow, or side of my head. Is there a different tracking function that would work better for an area like that?

lhotchkiss gravatar imagelhotchkiss ( 2013-12-15 20:31:30 -0500 )edit

You may try median-flow tracker. It works by tracking a grid of points (not a single point) in a defined image patch. Each grid point is tracked using Lucas-Kanade optical flow, and the median of this vector field determines location of the patch in the consecutive frame. More details and Python code here: If you still need something more advanced you may consider OpenTLD ( which combines tracking, online learning and detection to robustly track objects. Infact, Median flow tracker is a part of this framework. Hope this helps!

pyro gravatar imagepyro ( 2013-12-16 23:08:14 -0500 )edit