MEEM 1713 – ARTIFICIAL INTELLIGENCE

Chapter 3:
Artificial Neural Network
Part 4 – Matlab Toolbox and ANN Applications

Assoc. Prof. Ts. Dr. Herman Wahid

Department of Control & Mechatronics Engineering (CMED)
Faculty of Electrical Engineering
Universiti Teknologi Malaysia.
 Chapter Outline

3.13 Matlab NN Toolbox

3.13.1 Using Fitting Tool

3.13.2 Using Pattern Recognition Tool

3.14 Examples of ANN Applications

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 2
 3.13
Matlab NN Toolbox

www.utm.my
3
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 3.13.1

www.utm.my
Using Fitting Tool

MEEM 1713 – ARTIFICIAL INTELLIGENCE 8

 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 3.13.2
Using Pattern

www.utm.my
Recognition Tool
MEEM 1713 – ARTIFICIAL INTELLIGENCE 24
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 3.14
Examples of NN

www.utm.my
Applications
MEEM 1713 – ARTIFICIAL INTELLIGENCE 39
 NN Applications

6.4.1 Money/Currency Recognition Machines

6.4.2 Postcode Recognition

6.4.3 Intelligent Database Engine

6.4.4 Electricity Load Forecasting (CAIRO)

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 AI Applications in Industry

Expert Expert
systems systems
Fuzzy logic Fuzzy logic
Neural Neural
networks networks
…………. ………….

Intelligent Cognition
Man- Algorithms, Execution Tasks
Machine computations

www.utm.my
Interface

Perception Intelligent machine

(Sensors)
MEEM 1713 – ARTIFICIAL INTELLIGENCE 41
 AI Applications
(IT)
Decision
Support
Systems

Consumer
Products
AI

www.utm.my
Industrial Medical
Products Applications

MEEM 1713 – ARTIFICIAL INTELLIGENCE 42

 3.14.1
Money/Currency

www.utm.my
Recognition Machines
MEEM 1713 – ARTIFICIAL INTELLIGENCE 43
 CURRENCY RECOGNITION
• Several companies: NCR,
Glory, ..
• Example by Glory Ltd., Japan
• Currency recognition has a
wide variety of applications
such as:
 Banks (Counting, savings, etc.)
 ATMs (Depositing, etc)

www.utm.my
 Parking
 Currency Exchange
 Fraud detection
44
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 45
 • Currently used techniques: Humans, Expert
system, Rule-based system, etc.
• Neural networks can provide a simpler way in
designing the system

Image Feature
Processing Extraction

www.utm.my
Recognition Neural
Results Networks

MEEM 1713 – ARTIFICIAL INTELLIGENCE 46

 Technologies involve:
• Sensors
• Servos/motor control
• DSP/PC
• Image processing
• Feature extraction
• Neural networks
• Electronics
• Software

www.utm.my
• Machine design

47
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Prototype design

www.utm.my
48
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Technique to find best threshold
value (part of image processing)

www.utm.my
49
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Feature Extraction

www.utm.my
50
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Design considerations
• How many types of inputs
should be considered for
each denomination?

• 4 Types
 face up - head
 face up - tail
 face down - head
 face down – tail

• The target during training

www.utm.my
of the neural network are
the value of the banknote.

MEEM 1713 – ARTIFICIAL INTELLIGENCE 51

 • The ANN configuration is as shown in the figure below.
• Once the error is minimized, training is stopped and the machine is
tested with untrained banknotes.

• Even with a small training set, it was observed that the machine was
capable of recognizing the different banknotes.

www.utm.my
52
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Australian $20

USA $50

UK Pound 20

Canada $10

Hong Kong $10

www.utm.my
After training NN will
identify the correct currency
MEEM 1713 – ARTIFICIAL INTELLIGENCE 54
 Prototype Currency
Recognition Machine by
Neural Networks

www.utm.my
55
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 3.14.2
Postcode Recognition

www.utm.my
(CAIRO)
MEEM 1713 – ARTIFICIAL INTELLIGENCE 56
 6.3 Postcode Recognition by Fuzzy ARTMAP NN

• As labour gets more expensive, the need for automation arise.

• It was reported that using manual sorting post offices in many

countries are running at a loss.

• Post office automation involves the use of high technology such as

image processing techniques, intelligent character recognition using
neural networks, and high speed sorting machines.

• At CAIRO, a machine that reads handwritten postcodes based on the

fuzzy ARTMAP neural networks has been developed.

• Uses a CCD camera for capturing image of the specially-designed

www.utm.my
enveloped, the trained Fuzzy ARTMAP recognises and interprets the
images into corresponding ASCII codes, then sort the letter into the
appropriate bin.

57
UTM
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 • Components of the Automated Letter Sorting System

Windows NT 4.0

PRM
DDE Main Module
WiT

Parallel Port

www.utm.my
ICD MSM

MEEM 1713 – ARTIFICIAL INTELLIGENCE 58 58

 Image Capturing Device (ICD)
• To acquire a monochrome image of the envelope into the
computer in real-time.
• It consists of:
– Sony Charged Couple Device (CCD) camera
– Camera adapter
– DT3155 Frame grabber board

Main Module
• Provides a graphical user interface (GUI) to interact with the
user.
• Allows the user to choose the range of postcode that

www.utm.my
belongs to a certain bin.
• Monitor and control of the operation.
• Developed using Borland C++Builder.

MEEM 1713 – ARTIFICIAL INTELLIGENCE 59 59

 Postcode Recognition Module (PRM)
• Core module of the system.
• Instructs the ICD to capture image.
• Informs the MM of the recognition result.
• Image processing part is developed using WiT 5.0.2, an Image
processing software.
• Fuzzy ARTMAP neural network is developed using Visual C++ 4.2.
Light source
Tray CCD

Mail Sorting Module (MSM) Camera

• Has 5 bins for mail sorting, ‘A’ to ‘E’.

Bin D
Envelope

• Transport the letters to the designated bins. Bin B

www.utm.my
• Solenoids act as pushers to push the letters
into the desired bins.
• connected to the PC via the parallel port.

MEEM 1713 – ARTIFICIAL INTELLIGENCE 60 60

 CAI RO’s I ntelligent P ost-Code
Recognition M achine
CCD Frame
Camera Grabber
Marzuki Khalid
CAIRO, UTM,
Kuala Lumpur

SORTING
MECHANISM
5 4 1 0 0
Fuzzy ARTMap
Handwritten Codes Algorithm

SORT

www.utm.my
Bin#1 Bin#2 Bin#3 Bin#n
MEEM 1713 – ARTIFICIAL INTELLIGENCE 61
 Hand-written Character Recognition using
Fuzzy ARTMAP Neural Networks
Objective: To develop a system using Fuzzy ARTMAP neural networks which
can recognise hand-written characters.

Pre-processing

Feature Extraction

www.utm.my
Trained Fuzzy ARTMAP Neural Networks

Classification of character in ASCII

MEEM 1713 – ARTIFICIAL INTELLIGENCE 62 62
 Adaptive Resonance Theory (ART)
Attentional Subsystem Orienting
F2 yj Subsystem
+ +
Gain
+
Control
wji
Reset
Gain - Signal
Control - xi +
+ +
F1 +

Input
• An unsupervised neural networks architecture.

www.utm.my
• Attempt to address the stability-plasticity
dilemma.
• Faster convergence rate compare to conventional
neural networks like back-propagation MLP.
MEEM 1713 – ARTIFICIAL INTELLIGENCE 63 63
 Fuzzy ARTMAP Algorithm
Inter-ART module
ARTa ARTb
Fab module
module

F2a ρ F2b
reset reset

F1a F1b

I = (I0, I0c) O = (O0, O0c)

F0a ρa match tracking F0b ρb

I0 O0

www.utm.my
• A supervised neural networks applying two
Fuzzy ART neural networks and an inter-ART module.
• Synergize ART neural networks with fuzzy logic.
MEEM 1713 – ARTIFICIAL INTELLIGENCE 64 64
 Configuration of the Image Processing Software
for the Intelligent Post-Code Recognition

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 65
 www.utm.my
Samples of Envelopes with Handwritten Postcodes

MEEM 1713 – ARTIFICIAL INTELLIGENCE 66

 CAIRO’s
Intelligent
Postcode
Recognition and
Envelope Sorting
Machine
Sorter

www.utm.my
MINDEX’97-Silver
Prize

MEEM 1713 – ARTIFICIAL INTELLIGENCE 67

 www.utm.my
Example of GUI on Screen
MEEM 1713 – ARTIFICIAL INTELLIGENCE 68 68
 Experiments Done
• Envelope’s Background Colour
– white, yellow, pink, blue, green.
• Foreground Colour
– blue, red, green, black, pencil’s grey
• Stroke Thickness
• Character Size

• Position

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 69
 Conclusion of Research
• The Fuzzy ARTMAP neural network, has been successfully
applied to solve the hand-written character recognition
problem.

• A comparative study of the Fuzzy ARTMAP and MLP prove

that the former is better in the context of hand-written
character recognition problem.

• An Automated Letter Sorting Machine is developed based on

the Fuzzy ARTMAP.

www.utm.my
• CAIRO are in the midst of getting a contract from Pos
Malaysia as a consultant for their post office automation
system.

MEEM 1713 – ARTIFICIAL INTELLIGENCE 70

 3.14.3
Intelligent Database Engine

www.utm.my
(CAIRO)
MEEM 1713 – ARTIFICIAL INTELLIGENCE 71
 4.5 Intelligent Database Engine
UTM

• In many large organizations, databases are used to keep information

and for management systems.

• However, most of these database systems are rather static such that
they only store information and the user will use the information to
make their own decisions.

• Intelligent database systems are currently being developed such that

the system itself can help the user to make decisions based on
experience or past information stored in the database.

• At CAIRO, UTM, we are developing an Intelligent Database Engine that

uses neural network techniques to aid the decision making process.

www.utm.my
• Based on the available data and past experience on the decision
making process, the neural network is trained to learn the experience.

72
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 • Once trained, the neural network can be used to look at new data and
make the necessary decisions.

• However, neural networks cannot provide the reasons for such

decision and this is a drawback as many users would want to know
how such decisions were made by the computer system.

• To overcome such problem, we design our Intelligent Database Engine

(IDE) with a data mining algorithm called “A Priori” to mine the data
such that the reasons or rules are given on why such decisions were
made.

• Thus, our IDE is made of two components:

– (1). a neural network component that make decisions
– (2). a data mining component that provides rules/reasons supporting the

www.utm.my
neural network decisions

• The IDE can be applied in many kinds of databases such as loan

approval, job appraisal, tender reccommendations, etc.
73
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Design of an Intelligent Database
System

1. Loan Approval Process (LAP)

(done Manually)

2. LAP
(Assist by Info Management System)

www.utm.my
3. Intelligent Database System

MEEM 1713 – ARTIFICIAL INTELLIGENCE

 Loan Approval System (Manual)

Manager

Applicants

Bank
Evaluation

www.utm.my
Form Standard

Yes/No

MEEM 1713 – ARTIFICIAL INTELLIGENCE 75 75

 Loan Approval System
(Assisted by Intelligent Database Engine)

I.D.E.

Applicants

Bank
Evaluation

www.utm.my
Form Standard

Yes/No
Manager
MEEM 1713 – ARTIFICIAL INTELLIGENCE 76 76
 Current Database Management Systems
Current System Weaknesses:
• Human Intensive
• Time Consuming
• Bias

Current System Limitations:

• Not intelligent enough to make decisions

www.utm.my
• Unable to discover knowledge/rules or anomalies

77
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Design of an Intelligent Database System

1. Loan Approval Process LAP

(Manual)

2. LAP
(Info Management System)

Artificial Intelligence + Data Mining

www.utm.my
Neural Network + Algorithm Apriori
3. Intelligent Database System

MEEM 1713 – ARTIFICIAL INTELLIGENCE 78

 Methodology:

• Phase 1: Data Capturing

• Phase 2: Data Pre-processing
• Phase 3: Intelligent Decision Making (Neural Network)
• Phase 4: Rule/Anomaly Discovery (Algorithm Apriori)
• Phase 5: Output Presentation

www.utm.my
79
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 3:
How does Neural Network make decision?

Training Reasoning
θ
Age
w11
X1 f(S)
Y/R/N
Ocp
S
w1n
X2 Σ y
w21
w2n
Gender

www.utm.my
w31
Xn Σ
w3n e
Adaptive 80
MEEM 1713 – ARTIFICIAL INTELLIGENCE Algorithm
 Some Inputs (Evaluation Criteria)

• Age
• Salary
• Occupation
• Loan Amount
• Loan Purpose
• Mortgage
• Guarantor’s information
• House (Rent/Own)
• Other properties (shares)

www.utm.my
• Yearly taxes and insurance expenses
• Asset Balance (checking, savings, personal property)

81
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 3:
How does Neural Network make
decision?
Pre-processing
Numeric
35
Y= 1 Y
37
Age
45
Y= 0 R

Binary
Y= -1 N

www.utm.my
M= 1
Gender
F= 0

82
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 3:
How does Neural Network make
decision?
Professional
Professor
Sf Engineer 100 Y= 1
Doctor Y
Symbolic
Y= 0
Skilled Labor 010
Ocp Mechanic
R
Electrician
Plumber Y= -1
N

www.utm.my
Unskilled Labor
Laborer 001
Farmhand
Trucker

83
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 4:
How does Apriori extract
Rule/Anomaly?
AID Age Ocp Salary Loan Loan Result
Amount Purpose if x then y
1 30 SA 2500 20000 Car Y
2 30 Teacher 2000 5000 Computer Y If age=30 then
3 45 House
wife
0 40000 Car N Result=Y
4 30 Manager 4500 45000 Car Y
5 42 Driver 2000 70000 House N x: Age=30
y: Result=Y
3
Freq(x)
Minimum Support 50% X 100
Total records

www.utm.my
5

Freq(xy) 3
Minimum Confidence 60% X 100
Freq(x) 3

MEEM 1713 – ARTIFICIAL INTELLIGENCE 84 84

 Phase 4:
How does Apriori Extract
Rule/Anomaly?
• Pass 1
x: age=30 x: loan amount=20000
x: ocp=SA x: ocp=teacher
• Pass 2
x: age=30 and loan amount=20000
x: ocp=SA and salary=2500
• Pass 3

www.utm.my
x: age=30 and loan amount=20000
and loan purpose=car
85
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 4:
How does Apriori Extract
Rule/Anomaly?
For loan approval:
y: result=Y
or
y: result=N
if (age=30 and ocp=SA and …) then (result=Y)

www.utm.my
if (salary=2500 and age=35) then (result=Y)
if (salary=0) then (result=N)
86
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Phase 5:
Output Presentation:

Neural Algorithm
Network Apriori

www.utm.my
If… then…
Approve/ If… then…
Not Approve If… then…

MEEM 1713 – ARTIFICIAL INTELLIGENCE 87 87

 Outcome Expected of this
Project
• The system is able to learn the
knowledge to make decision despite the
absence of human expertise or priori
knowledge.
(Neural Network)

www.utm.my
88
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Outcome Expected of this
Project
• The accuracy or the applicability of the
result can be checked using a set of rules.
(Algorithm Apriori)

if age=30 and ocp=SA then result=Y

if salary=0 then result=N

www.utm.my
…
...
89
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Benefits

• Improve efficiency
– Save management cost (time & manpower)
– Speed up the evaluation process and
increase productivity
• Improve effectiveness
– Avoiding bias
– Avoiding mistake due to human emotional
factor

www.utm.my
– Strong adaptation, system can be modify
easily to adapt to changes

90
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 13.4.4

www.utm.my
Electricity Load Forecasting
MEEM 1713 – ARTIFICIAL INTELLIGENCE 91
 4.8 Electricity Load Forecasting

• At the National Load Despatch Centre (NLDC) of TNB, the generation

of electricity is always predicted ahead of time.

• This is because if the demand is less and too much electricity is

produced will result in a waste as large amount of electricity cannot
be stored.

• Thus, the NLDC monitor and control closely the generation of MW

according to the load demand.

• In this research, we use ANN to predict electricity load demand based

on past data.

www.utm.my
• The ANN can be trained to predict the MW generated at next hour or
a few hours in advance.
92
UTM
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 • ANN input candidates would be:
– MW - current hour + previous hour load demands
– time, date and year
– outside temperature
– type of day - Sunday, Monday, etc.
– type of day - National/Public holiday
– weather conditons - windy, cloudy ??

• The ANN training approach resembles a regression approach where

the load profile is learnt.

• The more information that is given to the ANN will make it more
intelligent or more precise in the MW prediction.

www.utm.my
93
MEEM 1713 – ARTIFICIAL INTELLIGENCE
 Example of the
ANN model for
load forecasting
Prediction
of MW

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 94 94
 CAIRO’s Neural Network
Software for Load Forecasting

www.utm.my
MEEM 1713 – ARTIFICIAL INTELLIGENCE 95 95