Ask Your Question
1

How to plot the Precision-Recall Curve (PRC) for a Cascade classifier correctly?

asked 2016-06-15 22:35:04 -0600

updated 2016-06-23 07:07:13 -0600

I am following OpenCV 3 Blueprints' Chapter 5 Object Detection for Industrial Applications by Steven Puttenams

https://www.packtpub.com/application-...

The training was successfully completed and I computed the F1 score to be approximately 0.86 as validated on the training dataset of 523 images (just to test if the training was okay).

Ex.

[root@cobalt workspace]# opencv_traincascade -vec npimages.vec -data output_lbp2/ -numPos 444 -numNeg 888 -bg negatives.txt -numStages 20 -featureType LBP -precalcValBufSize 2048 -precalcIdxBufSize 2048 -w 24 -h 24

However, my main problem is plotting the Precision-Recall Curve (PRC) curve, as suggested in Chapter 5, we could use the ff. to output the score:

detector.detectMultiScale(equalize, objects, levels, scores, 1.05, 1, 0, Size(), Size(), true);

But the score I'm getting have very small variations:

FILENAME    NUM_RECT    X   Y   WIDTH   HEIGHT  SCORE
/home/cobalt/Data/IMG_20160610_170847.jpg   1   190 287 68  68  -1.08848
/home/cobalt/Data/IMG_20160610_170925.jpg   1   186 294 68  68  -1.06534
/home/cobalt/Data/IMG_20160610_170957.jpg   1   189 286 68  68  -1.06534
/home/cobalt/Data/IMG_20160610_171038.jpg   1   191 289 67  67  -0.998512
    ... ... ..  ..  ..  ...
/home/cobalt/Data/IMG_20160610_205103.jpg   1   190 291 68  68  1.82761
/home/cobalt/Data/IMG_20160610_205106.jpg   1   190 291 68  68  1.82761
/home/cobalt/Data/IMG_20160610_205122.jpg   1   194 297 68  68  1.82761

And this gives rather small values for precision-recall:

Ex. coordinates.txt

0.00761905  0.00761905  -1.09
0.00763359  0.00761905  -1.08
0.00763359  0.00761905  -1.07
0.00766284  0.00761905  -1.06
0.00766284  0.00761905  -1.05
0.00766284  0.00761905  -1.04
... ..  ...
0.00952381  0.00761905  1.77
0.00952381  0.00761905  1.78
0.00952381  0.00761905  1.79
0.00952381  0.00761905  1.8
0.00952381  0.00761905  1.81
0.00952381  0.00761905  1.82

With the values above, I cannot get a decent PRC curve. I'm not quite sure what's the problem here.... Though, I can compute the Precision-Recall values using the detect_simple.cpp as I modified it here:

https://gist.github.com/melvincabatuan/45a0de3624e99a5c34d308d4a0b99b45

Ex. Output:

The number of true positives (TP) are: TP = 628.
The number of false positives (FP) are: FP = 185.
The number of false negatives (FN) are: FN = 7.
Precision = 0.772448.
Recall = 0.988976.
F1 Score = 0.867403.
edit retag flag offensive close merge delete

Comments

Did you solve this problem? I'm obtaining the same results but I don't know why. Thanks

Pessanha24 gravatar imagePessanha24 ( 2016-07-17 12:09:45 -0600 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2016-07-20 03:00:54 -0600

updated 2016-07-20 03:12:58 -0600

Hello guys, keep in mind that the forum sometimes gets more than 100 new posts each day, so keeping track of ones interested for me can only be achieved if people put a notice. This can be done by adding the @ sign followed with my username here without spaces in between! I will take a look at this, based on the info @Pessanha24 sent me through the official github repo for the book. Solutions will also be posted there for future reference!


As a reply to the actual post

  • Could you somehow provide me with your model, test data en annotations file?
  • Why would the smaller variations be bad? It depends on how similar the test images are I guess.
  • Did you use the provided script of the book for calculating the precision-recall values? Or did you reprogram it yourself?
  • Can you take a look at the retrieved number of TP FP FN for each image and see if that resembles the detection output?

Happy to hear from you soon!

edit flag offensive delete link more

Comments

Also I notice you changed the condition loops quite heavily. Not sure why you did that, but the original code worked fine for me. Care to comment on that?

StevenPuttemans gravatar imageStevenPuttemans ( 2016-07-20 03:15:28 -0600 )edit

Question Tools

1 follower

Stats

Asked: 2016-06-15 22:35:04 -0600

Seen: 982 times

Last updated: Jul 20 '16