Ok some basic comments on traincascade algorithm and your application problems:
- Do not use to large images for model training. A hand rule that I am using is never to let the largest dimension go over 80 pixels. It will take ages in larger stages for training and basically you will run out of processing memory rather soon. I would suggest reducing your dimensions to the half of the original images when creating the vector file.
- You only specified standard options, however, hidden from you are the parameters -maxHitRate 0.995 and -maxFalseAlarmRate 0.5. The last one means that if a false alarm rate of 0.5^(number of stages) is reached, training is stopped, because that is exactly what you asked from the algorithm. If you want, you can decrease the number, but keep in mind that training will possible increase exponential.
- Actually all parameters could be seen here.
But even before that, it could be that your detector has enough at 4 stages to make a difference between positive and negative images. For this you will need to test your model. Possibly you will need way more negatives, to train for a varying and cluttered background, but it can give an idea about the accuracy of your detector.