Why absolute values of class weights do matter? #30891

satyrmipt · 2025-02-24T17:00:14Z

satyrmipt
Feb 24, 2025

The question of how to choose absolute values of class weights was never covered in my uni lectures so I thought that only relative values do matter and class_weight={0:1, 1:2, 2:4} is equal to class_weight={0:0.5, 1:1, 2:2}. But you could simply check this is not true. The code below solves the same task in loop with different weights while ratios of every two weights stay the same:

import plotly.express as px
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
import numpy as np
print(sklearn.__version__) # 1.5.1

x, y = load_iris(return_X_y =True)
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.2, random_state=314)
train_x.shape, test_x.shape, train_y.shape, test_y.shape

class_weight={0:1, 1:2, 2:4}
plotx, ploty, ploty_train =[], [], []
for i in range(1,10000, 5):
    i=i/100
    curr_class_weight={k:v/i for k,v in class_weight.items()}
    model=LogisticRegression(random_state=314, class_weight=curr_class_weight, max_iter=1550000)
    model.fit(train_x, train_y)
    pred_y=model.predict(test_x)
    pred_train_y=model.predict(train_x)
    plotx.append(i*100)
    ploty.append(accuracy_score(test_y, pred_y))
    ploty_train.append(accuracy_score(train_y, pred_train_y))
    # print(curr_class_weight, accuracy_score(train_y, pred_train_y), accuracy_score(test_y, pred_y), sep='\t')
px.line(x=plotx, y=ploty)
px.line(x=plotx, y=ploty_train)

As you could see we get different results.

Why absolute values of weights do matter?
How can I calculate right (from ML POV) weights in case when i can not use compute_sample_weight or compute_class_weight directly. E.g. i have a problem when we need to predict client's bankruptcy. Some clients are more valuable then others and we have to predict VIP clients in first place, so every client has weight from 1 to 1000 defined by marketing department.

M-Ehtesham-Ul-Hassan-Malik · 2025-03-03T20:31:44Z

M-Ehtesham-Ul-Hassan-Malik
Mar 3, 2025

The Importance of Absolute Values in Class Weights

The absolute values of class weights do matter because they directly affect the optimization process of logistic regression. While the relative ratios of weights influence the decision boundary, the absolute values impact the gradient updates and convergence behavior of the model.

1. Why Do Absolute Values of Weights Matter?

Impact on the Loss Function Scaling

In weighted logistic regression, class weights adjust the loss function to prioritize certain samples.
The loss function is scaled by the absolute weight values, which means changing them affects the magnitude of gradients and learning dynamics.
If weights are scaled down, the model updates more slowly, and if scaled up, the model over-prioritizes those classes.

Regularization & Optimization Effects

Regularization (like L2 in LogisticRegression) is not weight-invariant.
If you divide all weights by a constant factor, it changes the regularization strength, leading to different decision boundaries.

Numerical Stability

Very small or very large class weights can lead to instability during training.
For example, using class_weight = {0: 0.01, 1: 0.02, 2: 0.04} vs. {0: 1, 1: 2, 2: 4} will lead to different optimization behaviors, even though the ratios are the same.

2. How to Calculate Correct Class Weights Without `compute_sample_weight`?

Since you cannot use compute_sample_weight or compute_class_weight directly, you can use a manual approach based on inverse class frequency or domain knowledge.

1. Inverse Class Frequency Method (if working with imbalanced classes)

If your dataset has an imbalance, you can set class weights as:

w𝑖 = N / N𝑖

where:

N = Total number of samples
N𝑖 = Number of samples in class 𝑖
w𝑖 = Weight for class 𝑖

Example Implementation

import numpy as np
from collections import Counter

# Sample labels
y_train = np.array([0, 1, 1, 2, 2, 2, 2])  # Example class distribution
class_counts = Counter(y_train)
total_samples = len(y_train)

# Compute class weights
class_weights = {cls: total_samples / count for cls, count in class_counts.items()}
print(class_weights)

2. Custom Weights Based on Business Value (Your VIP Clients Example)

Since your VIP clients have weights from 1 to 1000, you can normalize them to avoid extreme effects:

Option 1: Min-Max Scaling

w𝑖 = ((Weight𝑖 - min) / max - min) + 1

Option 2: Log-Scaling to Prevent Extreme Values

w𝑖 = log(weight i+1)

Example Implementation

import numpy as np

# Assume marketing assigns weights from 1 to 1000
client_weights = np.array([10, 50, 100, 500, 1000])  

# Normalize using log-scale
normalized_weights = np.log(client_weights + 1)
print(normalized_weights)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why absolute values of class weights do matter? #30891

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Why absolute values of class weights do matter? #30891

satyrmipt Feb 24, 2025

Replies: 1 comment

M-Ehtesham-Ul-Hassan-Malik Mar 3, 2025

The Importance of Absolute Values in Class Weights

1. Why Do Absolute Values of Weights Matter?

Impact on the Loss Function Scaling

Regularization & Optimization Effects

Numerical Stability

2. How to Calculate Correct Class Weights Without compute_sample_weight?

1. Inverse Class Frequency Method (if working with imbalanced classes)

Example Implementation

2. Custom Weights Based on Business Value (Your VIP Clients Example)

Option 1: Min-Max Scaling

Option 2: Log-Scaling to Prevent Extreme Values

Example Implementation

satyrmipt
Feb 24, 2025

M-Ehtesham-Ul-Hassan-Malik
Mar 3, 2025

2. How to Calculate Correct Class Weights Without `compute_sample_weight`?