8000 precision_score shows incorrect value · Issue #10843 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

precision_score shows incorrect value #10843

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
animeshprasad opened this issue Mar 20, 2018 · 6 comments · Fixed by #13151
Closed

precision_score shows incorrect value #10843

animeshprasad opened this issue Mar 20, 2018 · 6 comments · Fixed by #13151

Comments

@animeshprasad
Copy link

Description

precision_score shows incorrect value

Steps/Code to Reproduce

A=np.array([[0,0,1],[0,1,0],[0,0,1]])
B=A
precision_score(A,B, average=None)
array([ 0., 1., 1.])

Expected Results

array([ 1., 1., 1.])

Actual Results

array([ 0., 1., 1.])

Versions

import platform; print(platform.platform())
Darwin-14.5.0-x86_64-i386-64bit
import sys; print("Python", sys.version)
('Python', '2.7.10 (default, Jul 14 2015, 19:46:27) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]')
import numpy; print("NumPy", numpy.version)
('NumPy', '1.13.3')
import scipy; print("SciPy", scipy.version)
('SciPy', '1.0.0')
import sklearn; print("Scikit-Learn", sklearn.version)
('Scikit-Learn', '0.18.1')
@jnothman
Copy link
Member

You might have expected [nan, 1, 1,] too. We raise a warning to tell you that we will set it to 0.

@qinhanmin2014
Copy link
Member

I think the problem is that we do not raise a warning when there's only negative labels. E.g.

precision_score([0, 0, 0], [0, 0, 0])

I vote for a warning to tell users that we set precision, recall, fbeta_score and support to 0 in this case.

@jnothman
Copy link
Member
jnothman commented Mar 21, 2018 via email

@qinhanmin2014
Copy link
Member

A warning is indeed raised for the snippet presented in the issue description.

Sorry, I haven't noticed that.

So my opinion here is:
(1) Keep the current behavior as I suppose there's no clear definition of precision_score when there's only negative labels.
(2) Also raise similar warning for precision_score([0, 0, 0], [0, 0, 0])

@johannfaouzi
Copy link
Contributor

Maybe raising a warning like "XXX is ill-defined and being set to 0.0 where there is only one label in both true and predicted samples." where XXX would be precision/recall/f_score and returning 0 could be added when there is only label in both y_true and y_pred.

It could be added right after this, when present_labels.size == 1 and XXX would be determined by the argument warn_for.

scikit-learn/sklearn/metrics/classification.py

Lines 1027 to 1030 in 158c7a5

y_type, y_true, y_pred = _check_targets(y_true, y_pred)
check_consistent_length(y_true, y_pred, sample_weight)
present_labels = unique_labels(y_true, y_pred)

@johannfaouzi
Copy link
Contributor

Currently, precision_score([0, 0, 0], [0, 0, 0]) returns 1. If 0 is returned, some tests fail (1 is expected) like test_averaging_multilabel_all_ones :

scikit-learn/sklearn/metrics/tests/test_common.py

Lines 937 to 946 in 4e87d93

def test_averaging_multilabel_all_ones():
y_true = np.ones((20, 3))
y_pred = np.ones((20, 3))
y_score = np.ones((20, 3))
y_true_binarize = y_true
y_pred_binarize = y_pred
for name in METRICS_WITH_AVERAGING:
yield (check_averaging, name, y_true, y_true_binarize,
y_pred, y_pred_binarize, y_score)

and check_averaging calls _check_averaging :

scikit-learn/sklearn/metrics/tests/test_common.py

Lines 833 to 838 in 4e87d93

# No averaging
label_measure = metric(y_true, y_pred, average=None)
assert_array_almost_equal(label_measure,
[metric(y_true_binarize[:, i],
y_pred_binarize[:, i])
for i in range(n_classes)])

The multilabel case returns 1 and the current binary case returns also 1 (and returns 0 if the changes are made).

Are these tests supposed to be modified thus? @qinhanmin2014

@qinhanmin2014 qinhanmin2014 mentioned this issue Feb 12, 2019
Merged
@jnothman jnothman mentioned this issue Feb 13, 2019
Merged
@qinhanmin2014 qinhanmin2014 mentioned this issue Mar 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ENH/FIX Replace jaccard_similarity_score by sane jaccard_score jnothman/scikit-learn
4 participants
@jnothman @animeshprasad @qinhanmin2014 @johannfaouzi
0