Unit 5 Deep Learning

Below is a comprehensive explanation of the key topics under Deep Learning ,

including:
 Deep Feed Forward Network
 Regularization
 Training Deep Models
 Dropout
 Convolutional Neural Networks (CNNs)
 Recurrent Neural Networks (RNNs)
 Deep Belief Networks (DBNs)
For each, I’ll provide: ✅ Definition ,
🧠 Procedure / Algorithm ,
💡 Applications , and
📌 Examples

1. Deep Feed Forward Network (DFFN)

✅ Definition:
A Deep Feed Forward Network is a type of artificial neural network with multiple
hidden layers where information flows in one direction — from input to output —
without any cycles or feedback loops.
Also known as a Multi-Layer Perceptron (MLP) when deep.

🧠 Structure:
 Input Layer : Receives raw data.
 Hidden Layers : Multiple layers of neurons using activation functions like ReLU
or sigmoid.
 Output Layer : Produces final prediction (e.g., class label or continuous value).

🔁 Procedure / Algorithm:
1. Forward Propagation :
 Input → Hidden Layer 1 → Hidden Layer 2 → Output
 Each neuron computes weighted sum + applies activation function.
2. Loss Computation :
 Use loss function like MSE (regression) or Cross-Entropy (classification).
3. Backpropagation :
 Compute gradients using chain rule.
 Update weights via optimization algorithm (e.g., SGD, Adam).
4. Iterate :
 Repeat until convergence or fixed number of epochs.

💡 Applications:
 Image classification
 Speech recognition
 Time series forecasting
 Anomaly detection
📌 Example:
Predicting whether a customer will churn based on features like age, usage, and billing
history.

1
2. Regularizations
✅ Definition:
Regularization techniques are used to prevent overfitting by adding constraints or
penalties to the model.

🧰 Common Techniques:
METHOD DESCRIPTION
L1 Regularization (Lasso) Adds penalty proportional to absolute value of weights
L2 Regularization (Ridge) Adds penalty proportional to square of weights
Dropout Randomly drops neurons during training
Early Stopping Stops training when validation performance plateaus

🔁 How Used:
Add regularization term to loss function:
Total Loss=Original Loss+λ⋅Regularization Term

💡 Applications:
 Improving generalization
 Reducing variance
 Preventing overfitting on small datasets

📌 Example:
Using L2 regularization in a deep CNN to improve accuracy on test images.

3. Training Deep Models

✅ Definition:
Training deep models involves optimizing the parameters of a neural network with many
layers using gradient-based optimization methods.

🧠 Key Components:
 Initialization : Xavier or He initialization for weights
 Activation Functions : ReLU, Leaky ReLU, Tanh
 Optimization Algorithms : SGD, Adam, RMSProp
 Batch Size : Number of samples per update
 Learning Rate : Step size in parameter updates

🔁 Training Procedure:
1. Initialize weights randomly
2. For each epoch:
 Divide data into batches
 Forward propagate inputs
 Compute loss
 Backpropagate gradients
 Update weights using optimizer
3. Validate and stop early if performance plateaus

💡 Applications:

2
  Object detection
 Machine translation
 Generative modeling

📌 Example:
Training a deep DFFN to classify handwritten digits using the MNIST dataset.

4. Dropout
✅ Definition:
Dropout is a regularization technique that randomly deactivates a fraction of neurons
during training to prevent overfitting and improve generalization.

🧠 How It Works:
 During training, each neuron has a probability p of being dropped.
 During inference, all neurons are active but scaled down by factor p .

🔁 Algorithm:
At each training step:
1. Randomly select a subset of neurons to deactivate.
2. Perform forward and backward pass.
3. Restore full network for evaluation.

💡 Applications:
 CNNs and RNNs
 Preventing co-adaptation of neurons
 Improving robustness

📌 Example:
Applying dropout in a CNN before the final classification layer to reduce overfitting.

5. Convolutional Neural Network (CNN)

✅ Definition:
A CNN is a deep learning architecture designed to process grid-like data (e.g., images).
It uses convolutional layers to automatically learn spatial hierarchies of features.

🧠 Architecture:
 Convolutional Layers : Extract local patterns using filters
 Pooling Layers : Reduce spatial dimensions
 Fully Connected Layers : Final classification

🔁 Procedure:
1. Apply filters to input image to extract features
2. Use pooling (max/average) to downsample
3. Flatten feature maps and feed into fully connected layers
4. Train using backpropagation and optimization

💡 Applications:
 Image classification
 Object detection

3
  Facial recognition
 Medical imaging

📌 Example:
Using a CNN like VGG16 to classify images into categories such as dog, cat, car, etc.

6. Recurrent Neural Network (RNN)

✅ Definition:
An RNN is a type of neural network designed for sequential data. It maintains a hidden
state that captures information about previous elements in the sequence.

🧠 Variants:
 Vanilla RNN
 Long Short-Term Memory (LSTM)
 Gated Recurrent Unit (GRU)

🔁 Algorithm:
At each time step t :
ht=f(W⋅xt+U⋅ht−1)
Where:
 ht : hidden state at time t
 xt : input at time t
 f : activation function

💡 Applications:
 Language modeling
 Machine translation
 Speech recognition
 Time series forecasting

📌 Example:
Using an LSTM to generate text character-by-character based on a given seed.

7. Deep Belief Network (DBN)

✅ Definition:
A DBN is a generative probabilistic model composed of multiple layers of stochastic,
latent variables. It's typically built using Restricted Boltzmann Machines (RBMs).

🧠 Architecture:
 Unsupervised pre-training layer by layer using RBMs
 Supervised fine-tuning using backpropagation

🔁 Training Procedure:
1. Pre-train each layer using contrastive divergence
2. Stack trained RBMs to form DBN
3. Fine-tune using labeled data and backpropagation

💡 Applications:
 Feature extraction

4
  Dimensionality reduction
 Handwriting recognition
 Collaborative filtering

📌 Example:
Using DBN for unsupervised feature learning in document classification tasks.

📊 Summary Table
TOPIC PURPOSE KEY APPLICATION EXAMPLE
ALGORITHM
Deep Feed General-purpose Backpropagation, Classification, Customer Churn
Forward Network deep learning Adam Regression Prediction
Regularizations Prevent L1/L2, Dropout, Improve CNN for Image
overfitting Early Stopping generalization Classification
Training Deep Optimize Gradient Descent, All DL tasks Training a
Models parameters Batch Training language model
Dropout Regularization Randomly disable Reduce overfitting Before final
neurons layer in CNN
Convolutional Image processing Convolution, Max- Image Face
Neural Network Pooling Classification Recognition
Recurrent Neural Sequence LSTM, GRU Language Chatbot
Network modeling Modeling
Deep Belief Generative RBM stacking Feature Extraction Document
Network modeling Classification

📝 Final Notes:
 Deep Feed Forward Networks are foundational in deep learning.
 CNNs excel in vision tasks due to their ability to capture spatial structure.
 RNNs (especially LSTMs and GRUs) are essential for sequential data.
 Dropout and other regularizations help combat overfitting.
 DBNs are more niche today but historically important for unsupervised learning.

🔹 What is a Perceptron?
A perceptron is the simplest type of artificial neural network model, introduced by Frank
Rosenblatt in 1958. It is a binary linear classifier that maps input features to a binary
output (0 or 1), based on a linear combination of inputs and a threshold (or bias). It is the
foundation of many modern deep learning systems.

5
6
7
8