8000 average_precision_score does not return correct AP when all negative ground truth labels · Issue #8245 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

average_precision_score does not return correct AP when all negative ground truth labels #8245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
varunagrawal opened this issue Jan 30, 2017 · 22 comments · Fixed by #19085
Closed

average_precision_score does not return correct AP when all negative ground truth labels #8245

varunagrawal opened this issue Jan 30, 2017 · 22 comments · Fixed by #19085
Labels
Bug module:metrics

Comments

@varunagrawal
Copy link
Contributor

Description

average_precision_score does not return correct AP when y_true is all negative labels.

Steps/Code to Reproduce

One can run this piece of dummy code:

sklearn.metrics.ranking.average_precision_score(np.array([0, 0, 0, 0, 0]), np.array([0.1, 0.1, 0.1, 0.1, 0.1]))

It returns nan instead the correct value with the error:

RuntimeWarning: invalid value encountered in true_divide
recall = tps / tps[-1]

Expected Results

As per this Stackoverflow answer, Recall = 1 when FN=0, since 100% of the TP were discovered and Precision = 1 when FP=0, since no there were no spurious results.

Actual Results

Current output is:

/usr/local/lib/python3.5/dist-packages/sklearn/metrics/ranking.py:415: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
Out[201]: nan

Versions

Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609]
NumPy 1.12.0
SciPy 0.18.1
Scikit-Learn 0.18.1
@varunagrawal varunagrawal changed the title average_precision_score does not return correct AP when y_true is all negative labels average_precision_score does not return correct AP when all negative ground truth labels Jan 30, 2017
@varunagrawal
Copy link
Contributor Author
varunagrawal commented Jan 30, 2017 •
edited
8000 Loading

If someone can help me understand how to correctly do the False Positive calculation, I can submit a Pull Request.

For this bug, recall will be 1 since there are 0 True Positives and False Negatives. However, to calculate precision, I need to understand the correct way to find False Positives. The above sample results in _binary_clf_curve returning the same number of False Positives as the number of samples. Should I simply use that as the cue and say that is TPs=0 and FPs=len(y_true), then precision is 1?

I am not sure I quite understand how the threshold calculation is being performed here.

@amueller
Copy link
Member

Can you check if #7356 fixes this?

@lesteve
Copy link
Member
lesteve commented Feb 1, 2017

Can you check if #7356 fixes this?

No it doesn't. I think we just need to have to do something like this to handle this edge case:

recall = 1 if tps[-1] == 0 else tps / tps[-1]

@varunagrawal if you do a PR please add a non-regression test with only zeros in y_true.

@varunagrawal
Copy link
Contributor Author

@lesteve can you please also specify what needs to be done for precision? Or should that be as is?

@lesteve
Copy link
Member
lesteve commented Feb 1, 2017 •
edited
Loading

I believe the code works as it is. You can add a test with only 1s in y_true to make sure that precision is 1 in this case.

@varunagrawal varunagrawal mentioned this issue Feb 3, 2017
Closed
@varunagrawal
Copy link
Contributor Author

@lesteve hoping you can take a look at the PR.

@gvishal
Copy link
gvishal commented Oct 8, 2017

Updates on this. Is this merged?

@varunagrawal
Copy link
Contributor Author

The current functionality of average_precision has changed. I'm planning to submit a new PR for that. Will close this when the other PR is ready.

@gvishal
Copy link
gvishal commented Oct 9, 2017

The standard TREC Eval is able to compute AP and other metrics on the same data.

@mendozamanu mendozamanu mentioned this issue May 16, 2019
Closed
@kangkang59812
Copy link

so how did you solve it ?

@varunagrawal varunagrawal mentioned this issue Aug 9, 2019
Closed
@crypdick
Copy link
Contributor

What's the status on this?

@MarkCarbonell98
Copy link

Same problem here with sklearn 0.23.2

@varunagrawal varunagrawal mentioned this issue Dec 31, 2020
Merged
@varunagrawal
Copy link
Contributor Author

Finally got the PR in. Sorry about the delay.

@Lorenz92
Copy link

Any update on this issue?

@mcever
Copy link
mcever commented Sep 20, 2021

Just updated sklearn and this still appears to be an issue?

@varunagrawal
Copy link
Contributor Author

It's been 5 years now with this issue. I've opened the PR and updated it countless times but the only blocker is the approving review.

@Lorenz92
Copy link

Hi @lesteve, are you planning to approve this PR? Please let us know. Thank you!

@mcever
Copy link
mcever commented Sep 21, 2021

@varunagrawal thanks for submitting a fix :) can you link your update so that I can copy it to my own code base? Also, I think I'm finding that this issue also presents itself when a class has no examples. For example, obviously the following example y_true will trigger this bug

   [0., 0., 0., 0., 0.],
   [0., 1., 1., 0., 0.],
   [0., 0., 1., 0., 0.],
   [1., 1., 0., 0., 0.],

But I think it will still be present even removing the all 0 vector because the final column is all 0s. Do you also find this to be the case? If so, does your PR cover this case? If not, perhaps there's something wrong with my own code.

@ekosman
Copy link
ekosman commented Oct 24, 2021

Hey there. Is this issue going to be fixed?

@redsphinx
Copy link
redsphinx commented May 2, 2022 •
edited
Loading

Hi it's 2022 now and this is still an issue, any plans for fixing it?

This issue persists for when y_true is all -1's or all 0's

@jeremiedbb
Copy link
Member

Hi, it's 2022 now and this is issue has been fixed (see #19085). I encourage you to test the release candidate for version 1.1.0: ```pip install scikit-learn==1.1.0rc1``.

@redsphinx
Copy link

pip install scikit-learn==1.1.0rc1

Ok this works both with all -1 and 0, Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug module:metrics
Projects
None yet
Milestone
No milestone
0