How to visualize validation loss during each epoch with `make_pipeline` and metadata routing in scikit-learn? #30656

acmoudleysa · 2025-01-16T16:03:28Z

acmoudleysa
Jan 16, 2025

The Problem

I’m working with a custom MLPestimator in scikit-learn and want to visualize the validation loss during each epoch while training. Here's a snippet of the MLPestimator :

class MLPEstimator(BaseEstimator):
    def __init__(self, hidden_dims: List = [128, 64, 32], ...):
        self.hidden_dims = hidden_dims
        ...

    def fit(self, X, y, val_data: Tuple[NDArray, NDArray] = None):  # Accept val_data argument
        X, y = self._cast_torch(X, y, self.device)
        if val_data is not None:
            X_val, y_val = self._cast_torch(*val_data, self.device)  # Cast validation data

        # Setup and training steps
        for epoch in range(self.epochs):
            ...
            if val_data is not None:
                self.model.eval()
                with torch.no_grad():
                    val_pred = self.model(X_val)
                    val_loss = self.compute_loss(val_pred, y_val)
                    print(f"Epoch {epoch+1}: Validation Loss: {val_loss.item()}")
                self.model.train()

This allows me to visualize the validation loss at each epoch and train my model.

The challenge

I want to integrate this process with sklearn's make_pipeline, so I don’t have to manually apply preprocessing steps like scaling. Here's the current implementation:

from sklearn.pipeline import make_pipeline
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from custom_models import MLPEstimator
import sklearn
sklearn.set_config(enable_metadata_routing=True)

X, y = load_digits(return_X_y=True, n_class=5)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)

model_example = MLPEstimator(task="classification", epochs=500).set_fit_request(val_data=True)
pipeline = make_pipeline(StandardScaler(), model_example)

pipeline.fit(X_train, y_train, val_data=(X_val, y_val))  # Pass validation data during fit

sklearn.set_config(enable_metadata_routing=False)

It seems like it works but the issue is that the validation data is only passed to MLPestimator during the fit method, meaning it bypasses the preprocessing steps (like scaling). This causes a mismatch in how the training and validation data are processed, leading to incorrect results

The question

Is there a built-in way to visualize the validation loss during training while using make_pipeline? Also, how can I ensure the validation data passes through the preprocessing steps (like scaling) before being passed to the model for validation loss evaluation?

I understand metadata routing wasn't originally designed for this, but it seems like it could help here. Before building a custom solution, I wanted to check if there is already a better way to handle this within scikit-learn or if metadata routing can be leveraged effectively in this case.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

How to visualize validation loss during each epoch with `make_pipeline` and metadata routing in scikit-learn? #30656

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

How to visualize validation loss during each epoch with make_pipeline and metadata routing in scikit-learn? #30656

Uh oh!

Uh oh!

acmoudleysa Jan 16, 2025

The Problem

The challenge

The question

Replies: 0 comments

How to visualize validation loss during each epoch with `make_pipeline` and metadata routing in scikit-learn? #30656

acmoudleysa
Jan 16, 2025