[go: up one dir, main page]
Scribd
Upload
69 views54 pages

How Good Is Your Model?: Andreas Müller

The document discusses evaluating machine learning models and selecting the best hyperparameters. It introduces concepts like accuracy not always being the best evaluation metric due to issues like class imbalance. It discusses using more nuanced metrics like precision, recall, F1 score calculated from a confusion matrix. It shows how to obtain these metrics and plot a confusion matrix using scikit-learn. It also discusses tuning hyperparameters of models using grid search cross-validation to select parameters that perform best.

Uploaded by

rickshark
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
69 views54 pages

How Good Is Your Model?: Andreas Müller

The document discusses evaluating machine learning models and selecting the best hyperparameters. It introduces concepts like accuracy not always being the best evaluation metric due to issues like class imbalance. It discusses using more nuanced metrics like precision, recall, F1 score calculated from a confusion matrix. It shows how to obtain these metrics and plot a confusion matrix using scikit-learn. It also discusses tuning hyperparameters of models using grid search cross-validation to select parameters that perform best.

Uploaded by

rickshark
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 54

How good is your

model?
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

Andreas Müller
Core developer, scikit-learn
Classi cation metrics
Measuring model performance with accuracy:
Fraction of correctly classi ed samples

Not always a useful metric

SUPERVISED LEARNING WITH SCIKIT-LEARN


Class imbalance example: Emails
Spam classi cation
99% of emails are real; 1% of emails are spam

Could build a classi er that predicts ALL emails as real


99% accurate!

But horrible at actually classifying spam

Fails at its original purpose

Need more nuanced metrics

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

SUPERVISED LEARNING WITH SCIKIT-LEARN


Diagnosing classi cation predictions
Confusion matrix

Accuracy:

SUPERVISED LEARNING WITH SCIKIT-LEARN


Metrics from the confusion matrix
Precision

Recall

F1score:

High precision: Not many real emails predicted as spam

High recall: Predicted most spam emails correctly

SUPERVISED LEARNING WITH SCIKIT-LEARN


Confusion matrix in scikit-learn
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

knn = KNeighborsClassifier(n_neighbors=8)

X_train, X_test, y_train, y_test = train_test_split(X, y,


test_size=0.4, random_state=42)

knn.fit(X_train, y_train)

y_pred = knn.predict(X_test)

SUPERVISED LEARNING WITH SCIKIT-LEARN


Confusion matrix in scikit-learn
print(confusion_matrix(y_test, y_pred))

[[52 7]
[ 3 112]]

print(classification_report(y_test, y_pred))

precision recall f1-score support


0 0.95 0.88 0.91 59
1 0.94 0.97 0.96 115
avg / total 0.94 0.94 0.94 174

SUPERVISED LEARNING WITH SCIKIT-LEARN


Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Logistic regression
and the ROC curve
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

Hugo Bowne-Anderson
Data Scientist, DataCamp
Logistic regression for binary classi cation
Logistic regression outputs probabilities

If the probability ‘p’ is greater than 0.5:


The data is labeled ‘1’

If the probability ‘p’ is less than 0.5:

The data is labeled ‘0’

SUPERVISED LEARNING WITH SCIKIT-LEARN


Linear decision boundary

SUPERVISED LEARNING WITH SCIKIT-LEARN


Logistic regression in scikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

logreg = LogisticRegression()

X_train, X_test, y_train, y_test = train_test_split(X, y,


test_size=0.4, random_state=42)

logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)

SUPERVISED LEARNING WITH SCIKIT-LEARN


Probability thresholds
By default, logistic regression threshold = 0.5

Not speci c to logistic regression


k-NN classi ers also have thresholds

What happens if we vary the threshold?

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


The ROC curve

SUPERVISED LEARNING WITH SCIKIT-LEARN


Plotting the ROC curve
from sklearn.metrics import roc_curve
y_pred_prob = logreg.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)

plt.plot([0, 1], [0, 1], 'k--')


plt.plot(fpr, tpr, label='Logistic Regression')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Logistic Regression ROC Curve')
plt.show();

SUPERVISED LEARNING WITH SCIKIT-LEARN


Plotting the ROC curve

logreg.predict_proba(X_test)[:,1]

SUPERVISED LEARNING WITH SCIKIT-LEARN


Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Area under the ROC
curve
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

Hugo Bowne-Anderson
Data Scientist, DataCamp
Area under the ROC curve (AUC)
Larger area under the ROC curve = better model

SUPERVISED LEARNING WITH SCIKIT-LEARN


Area under the ROC curve (AUC)
Larger area under the ROC curve = better model

SUPERVISED LEARNING WITH SCIKIT-LEARN


Area under the ROC curve (AUC)
Larger area under the ROC curve = better model

SUPERVISED LEARNING WITH SCIKIT-LEARN


Area under the ROC curve (AUC)
Larger area under the ROC curve = better model

SUPERVISED LEARNING WITH SCIKIT-LEARN


AUC in scikit-learn
from sklearn.metrics import roc_auc_score

logreg = LogisticRegression()

X_train, X_test, y_train, y_test = train_test_split(X, y,


test_size=0.4, random_state=42)

logreg.fit(X_train, y_train)

y_pred_prob = logreg.predict_proba(X_test)[:,1]

roc_auc_score(y_test, y_pred_prob)

0.997466216216

SUPERVISED LEARNING WITH SCIKIT-LEARN


AUC using cross-validation
from sklearn.model_selection import cross_val_score
cv_scores = cross_val_score(logreg, X, y, cv=5,
scoring='roc_auc')

print(cv_scores)

[ 0.99673203 0.99183007 0.99583796 1. 0.96140652]

SUPERVISED LEARNING WITH SCIKIT-LEARN


Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Hyperparameter
tuning
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

Hugo Bowne-Anderson
Data Scientist, DataCamp
Hyperparameter tuning
Linear regression: Choosing parameters

Ridge/lasso regression: Choosing alpha

k-Nearest Neighbors: Choosing n_neighbors

Parameters like alpha and k: Hyperparameters

Hyperparameters cannot be learned by tting the model

SUPERVISED LEARNING WITH SCIKIT-LEARN


Choosing the correct hyperparameter
Try a bunch of different hyperparameter values

Fit all of them separately

See how well each performs

Choose the best performing one

It is essential to use cross-validation

SUPERVISED LEARNING WITH SCIKIT-LEARN


Grid search cross-validation

SUPERVISED LEARNING WITH SCIKIT-LEARN


Grid search cross-validation

SUPERVISED LEARNING WITH SCIKIT-LEARN


Grid search cross-validation

SUPERVISED LEARNING WITH SCIKIT-LEARN


GridSearchCV in scikit-learn
from sklearn.model_selection import GridSearchCV

param_grid = {'n_neighbors': np.arange(1, 50)}

knn = KNeighborsClassifier()

knn_cv = GridSearchCV(knn, param_grid, cv=5)

knn_cv.fit(X, y)

knn_cv.best_params_

{'n_neighbors': 12}

knn_cv.best_score_

0.933216168717

SUPERVISED LEARNING WITH SCIKIT-LEARN


Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Hold-out set for nal
evaluation
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

Hugo Bowne-Anderson
Data Scientist, DataCamp
Hold-out set reasoning
How well can the model perform on never before seen data?

Using ALL data for cross-validation is not ideal

Split data into training and hold-out set at the beginning

Perform grid search cross-validation on training set

Choose best hyperparameters and evaluate on hold-out set

SUPERVISED LEARNING WITH SCIKIT-LEARN


Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N

You might also like