Bounding box ranking / vehicle, pedestrian detection

edit

I have a set of images with hundreds of boxes scattered which aren't shown below. I would like to 'rank' these boxes in terms of how likely it is to contain either a car or a pedestrian similar to what is done in 'Edge Boxes'. The ranked boxes will then be passed to an actual classifier.

Currently, I seem to be getting a lot of boxes around trees and walls. The method needs to be really really fast and so I thought of simply trying to match it with a template that results in the low score on trees / buildings and higher score on cars / pedestrians. I think there is no faster way than direct template matching ?

I was wondering if anyone can provide a suggestion / paper for me to read up on. The focus is on speed.