10000 Implement binary/multiclass classification metric - Spherical Payoff by KaikeWesleyReis · Pull Request #18970 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Implement binary/multiclass classification metric - Spherical Payoff #18970

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Implement binary/multiclass classification metric - Spherical Payoff #18970

wants to merge 3 commits into from

Conversation

KaikeWesleyReis
Copy link

What does this implement? Explain your changes.

This PR implements spherical payoff score for binary/multiclass classification.

This metric is most used in ecological modelling for Decision Making models such as Bayesian Networks, where you want to maximize the correct predicted classes and the model's confidence in that output.

It's measures the model's confidence, through the probability, to predict the correct category and have a defined interval: [0, 1]. Best possible score is 1.0, and the worst is 0.0. It's calculated as the average over all samples.

The benefits to use this score:

  • See if your model have confidence in predicted answers
  • Support other useful metrics such as Confusion Matrix related, given the fact that measures the output probabilities
  • The other metrics equivalents (scoring rules metrics, read more here) as brier score or log loss it's only available for binary problems. This metric doesn't have this issue.

Any other comments?

The equation was implemented based on this 2007 paper reference:
equation

Other point is the fact that was not implemented yet for open source community, the only reference that I found was in a paid program call Netica in this tutorial.

Abs,

@jnothman
Copy link
Member
jnothman commented Dec 7, 2020 via email

@KaikeWesleyReis
Copy link
Author

Hi @jnothman,
In fact, you are correct: This metric is a well defined/used in Bayesian Networks and ecological decision modeling papers, but is not used (yet) in ML community.

Besides that, spherical payoff cousin, Brier score (they came from the same scoring family), will probably have your multi class version here.

Summarizing, I understand your POV and thanks for the report!

Base automatically changed from master to main January 22, 2021 10:53
@lorentzenchr lorentzenchr mentioned this pull request Nov 19, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
module:metrics
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@KaikeWesleyReis @jnothman
0