[go: up one dir, main page]
Scribd
Upload
17 views16 pages

Video 14 - Binary Cross Entropy Loss

The document discusses the usage of Binary Cross Entropy Loss in neural networks, focusing on encoding class labels as probability distributions and the differences between categorical and binary cross entropy loss. It highlights the importance of matching network architecture with the classification task and choosing appropriate activation functions. Additionally, it provides examples of implementing single-label and multi-label classifiers in PyTorch and emphasizes the need for different variants of cross entropy loss based on the task type.

Uploaded by

shubham jha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views16 pages

Video 14 - Binary Cross Entropy Loss

The document discusses the usage of Binary Cross Entropy Loss in neural networks, focusing on encoding class labels as probability distributions and the differences between categorical and binary cross entropy loss. It highlights the importance of matching network architecture with the classification task and choosing appropriate activation functions. Additionally, it provides examples of implementing single-label and multi-label classifiers in PyTorch and emphasizes the need for different variants of cross entropy loss based on the task type.

Uploaded by

shubham jha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Usage of Binary Cross Entropy Loss

Antonio Rueda-Toicen
Learning goals

● Encode class labels as probability distributions


● Understand differences between categorical and binary cross entropy loss
● Describe differences between sigmoid and softmax activation functions
● Design network architectures with agreement between the classification task,
output layer, activation function, and loss function
The logarithm function and cross entropy
Predicting a single label from multiple classes with a
neural network

MNIST has a single label per image


Single label classification: n probabilities for n classes

model = nn.Sequential(
# We get raw logits as output
nn.Linear(input_size, 3),
)

criterion = nn.CrossEntropyLoss()
loss = criterion(predictions, targets)
Binary Cross Entropy

Used to model events such as

Compare with categorical cross entropy


when having a single class and a single data point
Encoding class labels as probability distributions for binary and
multilabel classification
Single label binary classifiers
Language-guided detection with Grounding DINO

Image from Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
Single label binary classifiers in PyTorch

model = nn.Sequential(
model = nn.Sequential( nn.Linear(input_size, 1),
# Raw logits output # Convert to probabilities
nn.Linear(input_size, 1) ) nn.Sigmoid()
)
criterion = nn.BCEWithLogitsLoss()
loss = criterion(logits, targets) criterion = nn.BCELoss()
loss = criterion(predictions, targets)
Multilabel classification on satellite images

https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/
Multilabel classifiers

p=0.9
Multilabel classifiers in PyTorch

model = nn.Sequential( model = nn.Sequential(


nn.Linear(input_size, num_classes),
# Raw logits
# Convert to probabilities
nn.Linear(input_size, num_classes),
nn.Sigmoid()
)
criterion = nn.BCEWithLogitsLoss()
criterion = nn.BCELoss() loss = criterion(predictions, targets)
loss = criterion(predictions, targets)
Multilabel classification on satellite images

https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/
Summary
Cross entropy and the final activation function
● Cross entropy measures the mismatch between predicted and true distributions
● Softmax handles single-label problems with multiple classes
● Sigmoid handles both binary and multi-label cases
Network design
● Match output layer size to number of classes
● Consider single-label vs multi-label requirements for final layer
● Choose activation function based on classification type (e.g. single label vs multi-label)
Different tasks require different variants of cross entropy loss
● Use nn.BCELoss or nn.BCEWithLogitsLoss for binary or multi-label classification
tasks
● Use nn.CrossEntropyLoss for single label tasks
Further reading
LogSoftmax vs Softmax
● https://discuss.pytorch.org/t/logsoftmax-vs-softmax/21386

You might also like