5

CS7113
Machine Learning
W. M . Kalani Sriya Ilmini
Department of Computer Science
Faculty of Computing
Sir John Kotalawela Defence University
Rathmalana

1
Perceptron Learning Rule
Lecture 5

2
Objectives
• Determine the weight matrix and bias for perceptron networks with
many inputs.
• Explain what a learning rule is.
• Developing the perceptron learning rule.
• Discussing the advantages and limitations of the single layer
perceptron.

3
Development
• Introduced a neuron model by Warren McCulloch & Walter
Pitts[1943].
• Main features
• Weighted sum of input signals is compared to a threshold to determine the
output.
• 0 if weighted_sum <0
• 1 if weighted sum >=0
• Able to compare any logical arithmetic function
• No training method was available.

4
Development…
• Perceptron was developed by Frank Rosenblatt [1950]
• Neurons were similar to those of McCulloch & Pitts.
• Key feature- introduced a learning rule.
• Proved that learning rule is always converged to correct weights if
weights exist for the problem.
• Simple and automatic.
• No restriction on initial weights- random

5
Learning Rules
• Procedure for modifying the weights and biases of a network to
perform a specific task.
• Supervised Learning- Network is provided with a set of examples of proper network behavior
(inputs/ outputs)
• Reinforcement Learning- Network is only provided with a grade, or score, which indicates
network performance.
• Unsupervised Learning- Only network inputs are available to the learning algorithm. Network
learns to categorize (cluster) the inputs.

6
Perceptron Architecture

7
Perceptron Architecture…

8
Perceptron Architecture…
• Therefore, if the inner product of the ith row of the weight matrix with
the input vector is greater than or equal to −𝒃𝒊 the output will be 1,
otherwise the output will be 0. 0 if weighted_sum <0
1 if weighted sum >=0

• Each neuron in the network divides the input space into two regions.

9
Single- Neuron Perceptron

10
Decision Boundary
• Decision boundary

• All points on the decision boundary have the same inner product with
the weight vector.
• Decision boundary is orthogonal to weight vector.
• for any two points in the decision boundary.
•
• Weight vector is orthogonal to

11
Direction of the Weight Vector
• Any vector in the shaded region
will have an inner product
greater than –b and
• Vectors in the un-shaded region
will have inner product less than
–b.
• Therefore the weight vector
will always point toward the
region where the neuron output
is 1.

12
Graphical Method
• Design of a perceptron to implement the AND gate.

13
Graphical Method…
• First select a decision boundary that separate dark circles and light
circles.
• Next choose a weight vector that is orthogonal to the decision
boundary.
• The weight vector can be any length.
• Infinite no of possibilities.
• One choice is

14
Graphical Method…
• Finally, we need to find the bias, b.
• Pick a point on the decision boundary (say )

• Testing

15
Multiple- Neuron Perceptron
• Each neuron will have its own decision boundary.

• A single neuron can classify input vector into two categories.

16
Perceptron Learning Rule
• Supervised training
• Provided a set of examples of proper network behavior

where 𝒑𝑞 − input to the network and 𝒕𝑞 − corresponding output

17
Starting Point
• Random initial weight

• Present 𝒑1 to the network:

18
Tentative Learning Rule
• We need to alter the weight
vector so that is points more
toward 𝒑1 , so that in the future
it has a better chance of
classifying 𝒑1 .

19
Tentative Learning Rule…
• One approach would be to set equal to
• This rule cannot find a solution always.

20
Tentative Learning Rule…
• Another possibility would be add to
• This rule can be stated as

21
Second Input Vector

22
Third Input Vector

23
Unified Learning Rule

24
Unified Learning Rule…

25
Multiple- Neuron Perceptron

26
Apple/ Banana Example

27
Second Iteration

28
Check

29
Perceptron Rule Capability

The perceptron rule will always converge to

30
Perceptron Limitations

31
Thank You!