Object detection using ORB gives more than acceptable number of false positive and false negatives

edit

I have a template image and a bunch of test images to detect if the template image is present in them, by matching the features detected by ORB. I use the following piece of code

import cv2
import numpy as np

template_gray = cv2.imread(template_path, 0)
feature_detector = cv2.ORB_create()
temp_kp, temp_des = feature_detector.detectAndCompute(template_gray, None)
threshold = len(temp_kp)/4

matcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)

test_gray = cv2_imread(<test_img_path>, 0)
test_kp, test_des = feature_detector.detectAndCompute(test_gray, None)
matches = matcher.match(temp_des, test_des)
if len(matches) > threshold: print("Object present")
else: print("Object missing")

I have tried many different combinations of threshold values. But none is working reliably across all test images. Too high of a threshold usually yields a lot of false negatives and too low of it gives too many false positives.

Can anyone tell me if there is a way to improve upon this situation or if this is the fundamental limitation of object detection by feature matching?