added multiclass_log_loss metric #1125
Conversation
|
Thanks, I do think that's useful. I will try to review the code later. |
|
So one (somewhat) related issue is that one cannot optimize this type of metric using GridSearchCV. The problem is that the grid search always assumes that you want to score using model.predict() rather than predict_proba(). It's obviously easy to temporarily hack the code to allow this, but I was wondering if people had any desire for a better implementation of such a feature. Thoughts? |
|
I have two recommendations:
|
|
@kyleabeauchamp The issue with grid search was discussed here: #1014. |
|
@kyleabeauchamp thanks for your recommendations, changed the files accordingly... |
|
|
||
| def multiclass_log_loss(y_true, y_pred, eps=1e-15): | ||
| """Multi class version of Logarithmic Loss metric. | ||
| https://www.kaggle.com/wiki/MultiClassLogLoss |
There was a problem hiding this comment.
I would rather use more standard refences such as elements of statistical learning and wikipedia.
The link to the thread seems pretty out of place. Also, I guess alternative names should be mentioned. This is the multinomial logistic regression loss, right? aka softmax loss aka max entropy?
|
Hey @ephes, are you still working on this? Or are you to busy with the competition ;) |
|
Yes, I'm too busy. The competition is eating up all of my spare time atm :). But I do plan to work on this again next week, when the competition is over. |
|
Ok, no worries! I'll just use your branch until then. Good luck! |
|
How is this PR going @ephes ? I can try to help if needed :) |
|
Ping myself: this should get merged. |
|
iirc the documentation and testing needs some work. |
|
Could it be called |
|
no, it should ;) |
|
I have some time tomorrow, I hope I can finish it then. |
|
I agree that this PR needs some work. Binary log loss can also be used for OVR. You just need to sum up the losses of each class. So we might want to prefer two different functions, one for binary log loss and one for multiclass log loss. |
|
Close this one in favour of #2013. |
Don't know whether this is helpful, just practicing :)...