surf object/scene dependences

edit

Have created many object detection programs (SURF &SIFT) and they all work but don't do a good job of finding my objects in the scene. Is there some dependencies or relationships I am not addressing when defining pixels in the object and scene? Maybe some suggestions I need to address.