CS 229 – Machine Learning https://stanford.

edu/~shervine

VIP Cheatsheet: Deep Learning r Learning rate – The learning rate, often noted η, indicates at which pace the weights get
updated. This can be fixed or adaptively changed. The current most popular method is called
Adam, which is a method that adapts the learning rate.

r Backpropagation – Backpropagation is a method to update the weights in the neural network

Afshine Amidi and Shervine Amidi by taking into account the actual output and the desired output. The derivative with respect
to weight w is computed using chain rule and is of the following form:
August 23, 2018 ∂L(z,y) ∂L(z,y) ∂a ∂z
= × ×
∂w ∂a ∂z ∂w

Neural Networks As a result, the weight is updated as follows:

Neural networks are a class of models that are build with layers. Commonly used types of neural ∂L(z,y)
networks include convolutional and recurrent neural networks. w ←− w − η
∂w
r Architecture – The vocabulary around neural networks architectures is described in the
figure below:
r Updating weights – In a neural network, weights are updated as follows:

• Step 1: Take a batch of training data.

• Step 2: Perform forward propagation to obtain the corresponding loss.

• Step 3: Backpropagate the loss to get the gradients.

By noting i the ith layer of the network and j the j th hidden unit of the layer, we have:
• Step 4: Use the gradients to update the weights of the network.
[i] [i] T [i]
zj = wj x + bj

where we note w, b, z the weight, bias and output respectively. r Dropout – Dropout is a technique meant at preventing overfitting the training data by
dropping out units in a neural network. In practice, neurons are either dropped with probability
r Activation function – Activation functions are used at the end of a hidden unit to introduce p or kept with probability 1 − p.
non-linear complexities to the model. Here are the most common ones:

Sigmoid Tanh ReLU Leaky ReLU Convolutional Neural Networks

1 ez − e−z r Convolutional layer requirement – By noting W the input volume size, F the size of the
g(z) = g(z) = g(z) = max(0,z) g(z) = max(z,z)
1 + e−z ez + e−z convolutional layer neurons, P the amount of zero padding, then the number of neurons N that
with   1 fit in a given volume is such that:

W − F + 2P
N = +1
S

r Batch normalization – It is a step of hyperparameter γ, β that normalizes the batch {xi }.

By noting µB , σB
2 the mean and variance of that we want to correct to the batch, it is done as

follows:

xi − µ B
xi ←− γ p +β
r Cross-entropy loss – In the context of neural networks, the cross-entropy loss L(z,y) is 2 +
σB
commonly used and is defined as follows:
h i
L(z,y) = − y log(z) + (1 − y) log(1 − z) It is usually done after a fully connected/convolutional layer and before a non-linearity layer and
aims at allowing higher learning rates and reducing the strong dependence on initialization.

Stanford University 1 Fall 2018

CS 229 – Machine Learning https://stanford.edu/~shervine

Recurrent Neural Networks • We iterate the value based on the values before:

r Types of gates – Here are the different types of gates that we encounter in a typical recurrent
" #
neural network:
X
0 0
Vi+1 (s) = R(s) + max γPsa (s )Vi (s )
a∈A
s0 ∈S
Input gate Forget gate Output gate Gate
Write to cell or not? Erase a cell or not? Reveal a cell or not? How much writing?
r Maximum likelihood estimate – The maximum likelihood estimates for the state transition
probabilities are as follows:
r LSTM – A long short-term memory (LSTM) network is a type of RNN model that avoids #times took action a in state s and got to s0
the vanishing gradient problem by adding ’forget’ gates. Psa (s0 ) =
#times took action a in state s

Reinforcement Learning and Control

r Q-learning – Q-learning is a model-free estimation of Q, which is done as follows:
The goal of reinforcement learning is for an agent to learn how to evolve in an environment. h i
Q(s,a) ← Q(s,a) + α R(s,a,s0 ) + γ max Q(s0 ,a0 ) − Q(s,a)
r Markov decision processes – A Markov decision process (MDP) is a 5-tuple (S,A,{Psa },γ,R) a0
where:
• S is the set of states
• A is the set of actions
• {Psa } are the state transition probabilities for s ∈ S and a ∈ A

• γ ∈ [0,1[ is the discount factor

• R : S × A −→ R or R : S −→ R is the reward function that the algorithm wants to

maximize

r Policy – A policy π is a function π : S −→ A that maps states to actions.

Remark: we say that we execute a given policy π if given a state a we take the action a = π(s).
r Value function – For a given policy π and a given state s, we define the value function V π
as follows:
h i
V π (s) = E R(s0 ) + γR(s1 ) + γ 2 R(s2 ) + ...|s0 = s,π

∗
r Bellman equation – The optimal Bellman equations characterizes the value function V π
of the optimal policy π ∗ :
∗ ∗
X
V π (s) = R(s) + max γ Psa (s0 )V π (s0 )
a∈A
s0 ∈S

Remark: we note that the optimal policy π ∗ for a given state s is such that:
X
π ∗ (s) = argmax Psa (s0 )V ∗ (s0 )
a∈A
s0 ∈S

r Value iteration algorithm – The value iteration algorithm is in two steps:

• We initialize the value:

V0 (s) = 0

Stanford University 2 Fall 2018