Photogrammetry & Robotics Lab
Intro to Neural Networks
Part 1: Network Basics
Cyrill Stachniss
The slides have been created by Cyrill Stachniss.
1
5 Minute Preparation for Today
https://www.ipb.uni-bonn.de/5min/
2
Image Classification
input classifier output
“cat”
“5”
3
Semantic Segmentation
“a label for
each pixel”
4
Neural Networks
§ Machine learning technique
§ Often used for classification, semantic
segmentation, and related tasks
§ First ideas discussed in the 1950/60ies
§ Theory work on NNs in the 1990ies
§ Increase in attention from 2000 on
§ Deep learning took off around 2010
§ CNNs for image tasks from 2012 on
5
Part 1
Neural Networks Basics
6
Neural Network
What is a neuron? What is a network?
fundamental unit connected elements
(of the brain)
neural networks are connected
elementary (computing) units
7
Biological Neurons
Biological neurons are the fundamental
units of the brain that
§ Receive sensory input from the
external world or from other neurons
§ Transform and relay signals
§ Send signals to other neurons and
also motor commands to the muscles
8
Artificial Neurons
Artificial neurons are the fundamental
units of artificial neural networks that
§ Receive inputs
§ Transform information
§ Create an output
9
Neurons
§ Receive inputs / activations from
sensors or other neurons
§ Combine / transform information
§ Create an output / activation
10
Neurons as Functions
We can see a neuron as a function
§ Input given by
§ Transformation of the input data can
be described by a function
§ Output
11
Neural Network
§ NN is a network/graph of neurons
§ Nodes are neurons
§ Edges represent input-output
connections of the data flow
input computations output
12
Neural Network as a Function
§ The whole network is again a function
§ Multi-layer perceptron or MLP is often
seen as the “vanilla” neural network
input layer hidden layers output layer
13
Neural Networks are Functions
§ Neural networks are functions
§ Consist of connected artificial neurons
§ Input layer takes (sensor) data
§ Output layer provides the function
result (information or command)
§ Hidden layers do some computations
input layer hidden layers output layer
14
Different Types of NNs
§ Perceptron
§ MLP – Multilayer perceptron
§ Autoencoder
§ CNN – Convolutional NN
§ RNN – Recurrent NN
§ LSTM – Long/short term memory NN
§ GANs – Generative adversarial network
§ Graph NN
§ Transformer
§ ...
15
[Image courtesy: van Veen]
16
[Image courtesy: van Veen]
17
Multi-layer Perceptron (MLP)
18
Multi-layer Perceptron
Seen as a Function
input layer hidden layers output layer
19
Image Classification Example
“cat”
input function that maps label
image images to labels
20
What is the Network’s Input?
An image consists
of individual pixels.
image 21
What is the Network’s Input?
An image consists
of individual pixels.
pixel intensities
Each pixel stores
an intensity value.
image 22
What is the Network’s Input?
pixel intensities
An image consists
of individual pixels.
Each pixel stores
an intensity value.
image 23
What is the Network’s Input?
An image consists
of individual pixels.
Each pixel stores
an intensity value.
We have N+1 such
intensity values.
24
What is the Network’s Input?
Arrange all the
intensity values
in a N+1 dim vector.
25
What is the Network’s Input?
Arrange all the
intensity values
in a N+1 dim vector.
26
What is the Network’s Input?
Arrange all the
intensity values
in a N+1 dim vector.
27
Input Layer of the Network
This vector is
the input layer
of our network!
28
What is the Network’s Output?
“cat”
29
What is the Network’s Output?
Is it a...
cat or a
dog or a
human or a
...?
30
What is the Network’s Output?
Is it a...
cat or a
dog or a
human or a
...?
indicator
vector
31
What is the Network’s Output?
Is it a...
cat or a
dog or a
human or a
...?
indicator
vector
32
What is the Network’s Output?
Is it a...
cat or a
dog or a
human or a
...?
we are
never
certain...
33
Output of the Network
the output layer is vector
indicating an activation/
likelihood for each label
34
Image Classification
“cat”
largest
value
pixels intensities output layer is a
are the values of vector of likelihoods
the input layer for the possible labels
35
Multi-layer Perceptron
Let’s Look at a Single Neuron
36
Multi-layer Perceptron
Let’s Look at a Single Neuron
37
Perceptron (Single Neuron)
output
How does this
inputs function look like?
38
Perceptron (Single Neuron)
output activation
for the next layer
activations from weights
previous layer
39
Function Behind a Neuron
(input) activations
weights
bias
activation function
output activation
40
Function Behind a Neuron
A neuron gets activated ( ) through
§ A weighted sum of input activations
§ A bias activation
§ An activation function
41
Similarity to Convolutions?
§ A neuron is similar to a convolution
§ Remember linear shift-invariant
kernels used as local operators
This part looks like the convolutions
used for defining local operators
Additionally: activation function and bias
42
Activation Function
§ Biological neurons are either active
or not active
§ We can see this as a step function:
“activated”
“no activation”
§ Bias tells us where the activation
happens
43
Activation Function
“activated”
“no activation”
§ We can model this behavior through
§ Non-smooth functions (eg, steps) have
disadvantages later down the line...
44
Sigmoid Activation Function
§ Common activation function is a
sigmoid (also called logistic function)
§ Smooth function
§ Squeezes values to [0,1]
45
ReLU Activation Function
§ Most commonly used one is the so-
called “rectified linear unit” or ReLU
§
§ Often advantages for deep networks
46
Neuron Activation
§ A neuron is only activated if
“no activation” “activated” (identity)
§ If
§ the weighted activations are larger than the
negative bias
47
Common Activation Functions
There are different activation functions
§ sigmoid()
§ ReLU()
§ tanh()
§ atan()
§ softplus()
§ identity()
§ step-function()
§ …
ReLU is often used
48
Illustration
[Courtesy of S. Sharma]
49
Function Behind a Neuron
§ Neuron gets activated if the weighted
sum of input activations is large
enough (larger than the negative bias)
§ This is the case for all neurons in the
neural network
50
For All Neurons...
These are a lot of values! 51
Let’s Use a Matrix Notation
52
Each Layer Can Be Expressed
Through Matrix Multiplications
layer 1 layer 0
53
Do It Layer by Layer...
54
Do It Layer by Layer...
input = layer 0
layer 1
layer 2
layer k = output
That not much more than linear algebra...
55
Feedforward Networks
§ MLPs are feedforward networks
§ The information flows form left to right
§ There are no loops
§ Such networks are called feedforward
networks
§ There exist other variants (eg, RNNs)
56
Example:
Handwritten Digit Recognition
57
Handwritten Digit Recognition
=5
28x28 pixel image
[Image courtesy: Nielsen] 58
Handwritten Digit Recognition
[Image courtesy: Nielsen/Lecun] 59
A Basic MLP Recognizing Digits
[Image courtesy: Nielsen] 60
Images to Digits - A Mapping
from 784 to 10 Dimensions
28x28
pixel
input
images output
vector
(784 dim)
(10 dim)
[Partial image courtesy: Nielsen] 61
What Happens in the Layers?
62
What Happens in the 1st Layer?
pixel
values
784 input activations = pixel intensities
784 weights = weights for pixel intensities
63
What Happens in the 1st Layer?
784 input activations = pixel intensities
784 weights = weights for pixel intensities
treat activations and weights as images
-1 0 +1
white
black
(rest doesn’t
matter)
pixel values weights effect on the
weighted sum
64
What Happens in the 1st Layer?
-1 0 +1
pixel
values weights tell us
what matters for
activating the neuron!
individual “weight images” for a neuron
support individual patterns in the image
65
Link to Local Operators Defines
Through Convolutions
§ Direct link to defining
image operators through
-1 0 +1 convolutions
Here:
§ Global (not local) operators
§ Weight matrix does not
weights tell us (yet) “slide over image”
what matters
66
Weights & Bias = Patterns
§ Weights define the patterns to look
for in the image
§ Bias tells us how well the image must
match the pattern
§ Activation functions “switches the
neuron on” if it matches the pattern
67
What Happens in the 2nd Layer?
§ The weights in layer 2 tell us which
1st layer patterns should be combined
§ The deeper we go, the more patterns
get arranged and combined
§ The last layer decides, which final
patterns make up a digit
68
What Happens in the Layers?
raw simple combined patterns
pixels patterns patterns to digits
[Image courtesy: Nielsen] 69
No Manual Features
Compared to several other classifiers,
this network also includes the feature
computations – it operates directly on
the input data, no manual features!
raw simple combined patterns
pixels patterns patterns to digits
[Image courtesy: Nielsen] 70
Classification Performance
Such a simple MLP achieves a correct
classification for ~96% of the examples
[Partial image courtesy: Nielsen] 71
Classification Performance
§ A simple MLP achieves a classification
accuracy of ~96%
§ Note that there are tricky cases
§ That is a good performance for a
simple model!
§ Improved networks achieve ~99%
72
How to Design a
Neural Network?
73
How to Make the Network
Compute What We Want?
§ So far, the network is a recipe for
sequentially performing computations
§ Structure and parameters are the
design choices
§ How to set them?
Learning!
74
Summary – Part 1
§ What are neurons and neural networks
§ Lots of different networks exists
§ Focus: multi-layer perceptrons (MLP)
§ Activations, weights, bias
§ Networks have many parameters
§ “It’s just a bunch of matrices and
vectors”
§ MLP for simple image classification
§ Part 2: Learning the parameters
75
Literature & Resources
§ Online Book by Michael Nielsen, Chapter 1:
http://neuralnetworksanddeeplearning.com/chap1.html
§ Nielsen, Chapter 1, Python3 code:
https://github.com/MichalDanielDobrzanski/DeepLearningPython
§ MNIST database:
§ http://yann.lecun.com/exdb/mnist/
§ Grant Sanderson, Neural Networks
https://www.3blue1brown.com/
§ Alpaydin, Introduction to Machine Learning
76
Slide Information
§ The slides have been created by Cyrill Stachniss as part of the
photogrammetry and robotics courses.
§ I tried to acknowledge all people from whom I used
images or videos. In case I made a mistake or missed
someone, please let me know.
§ Huge thank you to Grant Sanderson (3blue1brown) for
his great educational videos that influenced this lecture.
§ Thanks to Michael Nielsen for his free online book & code
§ If you are a university lecturer, feel free to use the course
material. If you adapt the course material, please make sure
that you keep the acknowledgements to others and please
acknowledge me as well. To satisfy my own curiosity, please
send me email notice if you use my slides.
Cyrill Stachniss, cyrill.stachniss@igg.uni-bonn.de
77
