Building Blocks of DNN
CS-421 Machine Learning
Umair bin Mansoor
DHA Su¤a University
June 2020
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 1/6
Building Blocks of DNN
Let us revise and consider calculations for a single layer l.
Forward propagation:
Input: A[l 1 ] , Output: A[l ] , Z[l ]
Z[l ] = W [l ] A[l 1 ] + b[l ]
A[l ] = g [l ] Z[l ]
Backward propagation:
Input: dA[l ] , Z[l ] , Output: dA[l 1 ] , dW [l ] , db[l ]
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 2/6
Building Blocks of DNN
Let us see the Backward propagation steps in detail:
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN
Let us see the Backward propagation steps in detail:
dZ[l ] = dA[l ] g [l ] Z[l ]
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN
Let us see the Backward propagation steps in detail:
dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN
Let us see the Backward propagation steps in detail:
dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T
db[l ] = dZ[l ]
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN
Let us see the Backward propagation steps in detail:
dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T
db[l ] = dZ[l ]
dA[l 1 ] = W[l ]T dZ[l ]
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.
Getting to know the optimum value of the hyperparameters can
become another optimization problem.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?
Hyperparameters are parameters that are responsible for determining
W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.
Getting to know the optimum value of the hyperparameters can
become another optimization problem.
In order to have a deep knowledge about them requires a greater
understanding of how they a¤ect each other.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
Applied DNN is Emperical in Nature
- Hyperparameter tuning is a
repetitive process.
- There are a number of applications
to DNN e.g. Computer vision,
Speech, NLP, etc. currently, all
of them require tuning of
hyperparameters for optimal
performance.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 5/6
Human Brain v.s. Arti…cial Neural Network
No one knows how a biological neuron works, not yet.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network
No one knows how a biological neuron works, not yet.
All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network
No one knows how a biological neuron works, not yet.
All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.
What we have discovered is an e¢ cient method of learning using this
structure.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network
No one knows how a biological neuron works, not yet.
All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.
What we have discovered is an e¢ cient method of learning using this
structure.
This led us to think that our brain works in a similar fashion.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
