A Gentle Introduction
to Neural Networks
(with Python)
Tariq Rashid @postenterprise
EuroPython Bilbao July 2016
Background
Ideas
DIY
Handwriting
Thoughts
and a
live demo!
Background
Start With Two Questions
locate people in this photo
add these numbers
+
+
+
+
+
+
2403343781289312
2843033712837981
2362142787897881
3256541312323213
9864479802118978
8976677987987897
8981257890087988
= ?
AI is Huge!
Googles and Go
Ideas
Simple Predicting Machine
Simple Predicting Machine
Kilometres to Miles
random starting parameter
try a model - this one is linear
Kilometres to Miles
not great
Kilometres to Miles
better
Kilometres to Miles
worse !
Kilometres to Miles
best yet !
Key Points
1.
2.
xactly? Try
e
s
rk
o
w
g
in
how someth
w
o
n
k
.
t
n
o
D
parameters
le
b
ta
s
ju
d
a
model with
.
parameters
e
th
e
n
fi
re
or to
Use the err
Garden Bugs
Classifying Bugs
Classifying Bugs
Classifying Bugs
Classifying Bugs
Key Points
ings.
1.
Classifying
thing
th
predicting
e
k
li
a
d
in
k
s is
Learning from Data
Example
Width
Length
Bug
3.0
1.0
ladybird
1.0
3.0
caterpillar
Learning from Data
Learning from Data
not a good separator
Learning from Data
shift the line up
just above the training data point
Learning from Data
How Do We Update The Parameter?
error = target - actual
E = (A + A)x - Ax
A = E / x
Hang On!
Oh no!
each update ignores
previous examples
Calm Down the Learning
A = L (E / x)
learning rate
Calm Down the Learning
learning rate = 0.5
Key Points
1.
you
d - ensures
o
o
g
is
g
in
your learn
pact of
im
s
e
c
u
d
Moderating
re
nd
your data, a
ll
a
m
o
fr
rn
lea
.
training data
y
is
o
n
r
o
outliers
Boolean Logic
IF I have eaten my vegetables AND I am still hungry
THEN I can have ice cream.
IF its the weekend OR I am on annual leave THEN Ill
go to the park.
Input A
Input B
AND
OR
Boolean Logic
Boolean Logic
XOR Puzzle!
Input A
Input B
XOR
XOR Solution!
Use more than one node!
Key Points
gle
ith just a sin
w
d
e
lv
o
s
e
nt b
roblems ca
p
e
m
o
S
1.
r classifier.
simple linea
g together to
in
rk
o
w
s
e
nod
e multiple
s
u
n
a
c
u
o
s.
2. Y
se problem
e
th
f
o
y
n
a
solve m
Brains in Nature
Brains in Nature
brain 0.4 grams
11,000 neurons
natures brains can eat, fly, navigate, fight,
communicate, play, learn
.. and theyre
resilient
302 neurons
37 billion neurons
(humans 20 billion)
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons
https://faculty.washington.edu/chudler/facts.html
Brains in Nature
Brains in Nature
logistic function
y = 1 / (1 + e-x)
Brains in Nature
Artificial Neuron
Artificial Neural Network .. finally!
Pause.
...
Where Does The Learning Happen?
link weight?
sigmoid function slope?
Key Points
things, and
d
te
a
c
ti
is
h
do sop
ct
brains can
l
ra
and imperfe
tu
e
a
g
a
N
m
a
d
1.
to
ly resilient
are incredib
omputing.
c
l
a
n
io
it
d
a
like tr
signals .. un
d
artly inspire
p
s
in
ra
b
l
a
logic
to copy bio
g
in
ry
T
.
2
etworks.
n
l
a
r
u
e
n
l
artificia
its
arameter p
le
b
ta
s
ju
e ad
ights are th
e
w
k
in
L
s.
3.
ing happen
rn
a
le
e
th
where
Feeding Signals Forward
Feeding Signals Forward
Feeding Signals Forward
Matrix Multiplication
Matrix Multiplication
weights
incoming signals
WI = X
dot product
Key Points
e
tions can b
la
u
lc
a
c
rd
a
no
iplication,
any feedforw
lt
m
u
e
m
h
T
ix
tr
a
.
1
as m
concisely
d
e
s
s
re
p
x
e
network.
e
th
e
p
a
h
s
t
matter wha
n do matrix
a
c
s
e
g
a
u
g
lan
.
rogramming
p
e
m
nd quickly
o
a
S
y
tl
n
2.
ie
ic
n really eff
multiplicatio
Network Error
Network Error
Internal Error
Internal Error
Matrices Again!
Key Points
we
guide how
to
r
o
r
r
e
e
th
ber we use
m
e
m
e
k weights.
R
n
li
.
r
1
te
e
m
dels para
refine a mo
e
is easy - th
s
e
d
o
n
t
u
tp
or at the ou
nd actual
a
d
e
ir
s
2. The err
e
d
etween the
difference b
outputs.
obvious. A
t
n
is
s
e
d
o
n
l
or at interna
portion to
rr
o
r
e
p
e
h
in
T
it
t
li
.
p
3
to s
pproach is
a
c
ti
is
r
u
e
h
hts.
the link weig
be
e error can
th
g
n
ti
a
g
a
back prop
too!
4. and
ultiplication
m
ix
tr
a
m
as a
expressed
Yes, But How Do We Actually Update The Weights?
Aaarrrggghhh !!
Perfect is the Enemy of Good
landscape is a complicated difficult mathematical function ..
with all kinds of lumps, bumps, kinks
Gradient Descent
smaller gradient .. youre closer to the
bottom take smaller steps?
Key Points
e
of finding th
y
a
w
l
a
c
ti
c
a pra
t descent is
n
ie
d
a
r
G
1.
nctions.
fu
lt
u
c
fi
if
d
f
minimum o
by
rshooting
e
v
o
f
o
e
c
n
ha
shallower.
avoid the c
ts
n
e
a
g
c
t
u
n
o
ie
Y
d
ra
2.
eg
r steps if th
e
ll
a
m
s
g
in
tak
tion
ifficult func
d
a
is
rk
o
etw
nt
f a neural n
o
r
o
rr
dient desce
e
ra
e
g
h
e
b
y
a
3. T
m
eights so
of the link w
will help ...
Climbing Down the Network Error Landscape
We need to find this gradient
Error Gradient
E = (desired - actual)2
school level calculus (chain rule)
dE/dwij = - ej . oj . (1 - oj) . oi
previous node
A gentle intro to calculus
http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/a-gentle-introduction-to-calculus.html
Updating the Weights
move wjk in the opposite direction to
the slope
remember that learning rate
DIY
Python Class and Functions
Neural Network
Class
Initialise
Train
Query
set size, initial weights
do the learning
query for answers
Python has Cool Tools
matrix maths
numpy
scipy
matplotlib
notebook
Function - Initialise
# initialise the neural network
def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes
# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
# learning rate
self.lr = learningrate
# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)
pass
numpy.random.normal()
random initial weights
Function - Query
combined weighted signals into hidden
layer
then sigmoid applied
# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs
numpy.dot()
similar for output layer
Function - Train
# train the neural network
def train(self, inputs_list, targets_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T
same feed forward as before
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
output layer errors
hidden layer errors
# output layer error is the (target - actual)
output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.
transpose(hidden_outputs))
# update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.
transpose(inputs))
pass
update weights
Handwriting
Handwritten Numbers Challenge
MNIST Datasets
MNIST dataset:
60,000 training data examples
10,000 test data examples
MNIST Datasets
label
784 pixels
values
28 by 28 pixel image
Output Layer Values
Experiments
96% is very good!
weve only used simple ideas
and code
random processes
do go wonky!
More Experiments
98%
is amazing!
Thoughts
Peek Inside The Mind Of a Neural Network?
Peek Inside The Mind Of a Neural Network?
this isnt done very
often
Thanks!
live demo!
Finding Out More
makeyourownneuralnetwork.blogspot.co.uk
github.com/makeyourownneuralnetwork
www.amazon.co.uk/dp/B01EER4Z4G
twitter.com/myoneuralnet
slides goo.gl/JKsb62
Raspberry Pi Zero
It all works on a Raspberry Pi Zero
and it only costs 4 / $5 !!
