8000 precision_score shows incorrect value · Issue #10843 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]

Skip to content

precision_score shows incorrect value #10843

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
animeshprasad opened this issue Mar 20, 2018 · 6 comments · Fixed by #13151
Closed

precision_score shows incorrect value #10843

animeshprasad opened this issue Mar 20, 2018 · 6 comments · Fixed by #13151

Comments

@animeshprasad
Copy link

Description

precision_score shows incorrect value

Steps/Code to Reproduce

A=np.array([[0,0,1],[0,1,0],[0,0,1]])
B=A
precision_score(A,B, average=None)
array([ 0., 1., 1.])

Expected Results

array([ 1., 1., 1.])

Actual Results

array([ 0., 1., 1.])

Versions

import platform; print(platform.platform())
Darwin-14.5.0-x86_64-i386-64bit
import sys; print("Python", sys.version)
('Python', '2.7.10 (default, Jul 14 2015, 19:46:27) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]')
import numpy; print("NumPy", numpy.version)
('NumPy', '1.13.3')
import scipy; print("SciPy", scipy.version)
('SciPy', '1.0.0')
import sklearn; print("Scikit-Learn", sklearn.version)
('Scikit-Learn', '0.18.1')

@jnothman
Copy link
Member

You might have expected [nan, 1, 1,] too. We raise a warning to tell you that we will set it to 0.

@qinhanmin2014
Copy link
Member

I think the problem is that we do not raise a warning when there's only negative labels. E.g.

precision_score([0, 0, 0], [0, 0, 0])

I vote for a warning to tell users that we set precision, recall, fbeta_score and support to 0 in this case.

@jnothman
Copy link
Member
jnothman commented Mar 21, 2018 via email

@qinhanmin2014
Copy link
Member

A warning is indeed raised for the snippet presented in the issue description.

Sorry, I haven't noticed that.

So my opinion here is:
(1) Keep the current behavior as I suppose there's no clear definition of precision_score when there's only negative labels.
(2) Also raise similar warning for precision_score([0, 0, 0], [0, 0, 0])

@johannfaouzi
Copy link
Contributor

Maybe raising a warning like "XXX is ill-defined and being set to 0.0 where there is only one label in both true and predicted samples." where XXX would be precision/recall/f_score and returning 0 could be added when there is only label in both y_true and y_pred.

It could be added right after this, when present_labels.size == 1 and XXX would be determined by the argument warn_for.

y_type, y_true, y_pred = _check_targets(y_true, y_pred)
check_consistent_length(y_true, y_pred, sample_weight)
present_labels = unique_labels(y_true, y_pred)

@johannfaouzi
Copy link
Contributor

Currently, precision_score([0, 0, 0], [0, 0, 0]) returns 1. If 0 is returned, some tests fail (1 is expected) like test_averaging_multilabel_all_ones :

def test_averaging_multilabel_all_ones():
y_true = np.ones((20, 3))
y_pred = np.ones((20, 3))
y_score = np.ones((20, 3))
y_true_binarize = y_true
y_pred_binarize = y_pred
for name in METRICS_WITH_AVERAGING:
yield (check_averaging, name, y_true, y_true_binarize,
y_pred, y_pred_binarize, y_score)

and check_averaging calls _check_averaging :

# No averaging
label_measure = metric(y_true, y_pred, average=None)
assert_array_almost_equal(label_measure,
[metric(y_true_binarize[:, i],
y_pred_binarize[:, i])
for i in range(n_classes)])

The multilabel case returns 1 and the current binary case returns also 1 (and returns 0 if the changes are made).

Are these tests supposed to be modified thus? @qinhanmin2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants
0