exclude moving objects in camera frame

edit

I have some LEDs which are blinking at a frequency half the frame rate of camera. Which means ideally in one frame LED is on and in another frame the LED is off. I need to find the positions of LED lights in camera feed. I capture N number of frames in grayscale mode, compute absolute difference between the frames, threshold the resulting image and check for contours. This algorithm works fine when there is no object moving in the camera feed. If some object is moving this algorithm fails. Can any one please suggest what can I do to exclude moving objects in this algorithm? If there is another way ( completely different and better ) to detect the position of LED lights please suggest that as well. Here is my code to detect the LED positions in camera frame. ( where framesToProcess is N number of frames captured quickly and contourCenter is indexed Cartesian coordinates that are candidate for LED points.)

void getContourCenters(vector<Mat>  &framesToProcess, vector<pointI>& contourCenter)
{
size_t j = 0;

for (int i = 1; i < framesToProcess.size(); i++)
{



        Mat tempDifferenceImage, tempThresholdImage;
        vector< vector<Point> > contours;
        vector<Vec4i> hierarchy;
        Rect objectBoundingRectangle = Rect(0, 0, 0, 0);
        absdiff(framesToProcess[i - 1], framesToProcess[i], tempDifferenceImage);
        threshold(tempDifferenceImage, tempThresholdImage, SENSITIVITY_VALUE, 255, THRESH_BINARY);
        blur(tempThresholdImage, tempThresholdImage, Size(BLUR_SIZE, BLUR_SIZE));
        findContours(tempThresholdImage, contours, hierarchy, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

        for (int k = 0; k < contours.size(); ++k)
        {
            objectBoundingRectangle = boundingRect(contours[k]);
            int xpos = objectBoundingRectangle.x + objectBoundingRectangle.width / 2;
            int ypos = objectBoundingRectangle.y + objectBoundingRectangle.height / 2;
            contourCenter.push_back(mp(xpos, ypos, j++));
        }
    }

}