Ask Your Question
0

Precision-recall evaluation

asked 2013-05-04 11:39:27 -0500

Shin gravatar image

Is this

<a href=http://i39.tinypic.com/2n0cw2.jpg">

a possibile outcome for a precision-recall graph or you think i'm doing something wrong? (The blue line is relative to a one-step classifier, the other line refers to the same classifier with the difference that this time a second step is added in the classification process).

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
3

answered 2013-05-05 07:53:35 -0500

unxnut gravatar image

Your precision seems to improve slightly as the recall improves. Typically, the precision will go down as the recall improves. So, yes, there seems to be something suspect. And the precision seems to be very high to begin with while recall is still varying over a good portion albeit with very high value.

The ideal situation will be where precision and recall are both 1.0 and in that case, you will not have a graph like the one where recall varies with constant precision.

edit flag offensive delete link more

Comments

@unxnut Hi unxnut. Thank you for your comment. I was wondering if you could take a quick look to my confusion matrixes. You can find a screenshot of my spreadsheet here http://tinypic.com/r/qssdxw/5

Shin gravatar imageShin ( 2013-05-05 12:12:33 -0500 )edit

I looked at your numbers but cannot tell how you are computing precision and recall. Could you tell me how you did it so that I can see if there is any issue in your definition of those two terms.

unxnut gravatar imageunxnut ( 2013-05-05 12:58:18 -0500 )edit

@unxnut: Precision = TP/(TP+FP), Recall = TP/(TP+FN)

Shin gravatar imageShin ( 2013-05-05 13:07:20 -0500 )edit

Recall definition is not correct. I assume TP is the "Total Positives". Recall will be TP + Missed Positives. In other words, the total number of positives in the system.

unxnut gravatar imageunxnut ( 2013-05-05 13:27:19 -0500 )edit

I'm actually a little bit confused. Could you elaborate more or make an example? If i'm following your reasoning, you mean Recall = # True Positives / # Total Positives, but total should indeed be equal to (#True Positives + #False Negatives), which corresponds to my definition from earlier.

Shin gravatar imageShin ( 2013-05-05 13:47:32 -0500 )edit

You are right. The double negatives confused me :-(

unxnut gravatar imageunxnut ( 2013-05-05 13:51:12 -0500 )edit

I just went back and looked at your image again. I eyeballed a few values and they seem correct. Now, the next issue will be how you have plotted those values in your original image (the one on this page). Could you elaborate on that image?

unxnut gravatar imageunxnut ( 2013-05-05 13:56:49 -0500 )edit

For the "apple" classifier (a 1-vs-all classifier) i take precision and recall for different k values. I then plot them like in http://i40.tinypic.com/3145rme.jpg

Shin gravatar imageShin ( 2013-05-05 14:05:23 -0500 )edit

Define k. Is this the number of instances?

unxnut gravatar imageunxnut ( 2013-05-05 14:34:57 -0500 )edit

Number of clusters (first step is a Bag-of-words classification approach)

Shin gravatar imageShin ( 2013-05-05 15:07:16 -0500 )edit
1

answered 2013-05-06 10:51:18 -0500

I think there is something strange: you are plotting recall and precision for different kmeans parameter, but, this parameters doesn't seem related to your precision.

I think you should plot your decision parameter (the one used in confusion matrix), eg. if you want to recognize an apple, and every time you "see" something you say: "this is an apple", your recall is 100% but your precision should be quite low (depending on how many apples you have in your test).

A contrario, if you almost never said "this is an apple", you probably have a good precision, but a low recall. T

he idea of this graph, as I understand it, is to show "how good is your estimation". If I use the analogy of biometric:

How many bad guys are entering in my house if I want to be sure that all good guys are able to enter?

And help the "user" to find a good compromise... This is the basic goal of ROC curves, see the Wikipedia page here.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

Stats

Asked: 2013-05-04 11:39:27 -0500

Seen: 1,266 times

Last updated: May 06 '13