[go: up one dir, main page]

0% found this document useful (0 votes)
144 views8 pages

NN Learning and Expert Systems

This document discusses neural network learning and expert systems. It begins with an introduction to connectionist models and two fundamental neural network models: multilayer perceptrons and backpropagation networks. It then covers topics like representation issues, learning algorithms for single and multilayer models, unsupervised learning, backpropagation, and using neural networks for expert systems.

Uploaded by

Zendo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
144 views8 pages

NN Learning and Expert Systems

This document discusses neural network learning and expert systems. It begins with an introduction to connectionist models and two fundamental neural network models: multilayer perceptrons and backpropagation networks. It then covers topics like representation issues, learning algorithms for single and multilayer models, unsupervised learning, backpropagation, and using neural networks for expert systems.

Uploaded by

Zendo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

Neural Network Learning and Expert Systems

Stephen I . Gallant

Foreword xii i

I Basics 1

l Introduction and Important Definitions 3

1 .1 Why Connectionist Models? 3


1 .2 The Structure of Connectionist Models 11
1 .3 Two Fundamental Models : Multilayer Perceptron s
(MLP's) and Backpropagation Networks (BPN's) 17
1 .4 Gradient Descent 19
1 .5 Historic and Bibliographic Notes 23
1 .6 Exercises 27
1 .7 Programming Project 29

2 Representation Issues 31

2.1 Representing Boolean Functions 31


2 .2 Distributed Representations 39
2 .3 Feature Spaces and ISA Relations 42
2 .4 Representing Real-Valued Functions 48
2 .5 Example: Taxtime! 55
2 .6 Exercises 56
2 .7 Programming Projects 59

H Learning in Single-Layer Models 61

3 Perceptron Learning and the Pocket Algorithm 63


3 .1 Perceptron Learning for Separable Sets of Trainin g
Examples 63
3 .2 The Pocket Algorithm for Nonseparable Sets o f
Training Examples 74
*3 .3 Khachiyan's Linear Programming Algorithm 87
3 .4 Exercises 88
3 .5 Programming Projects 92

4 Winner-Take-All Groups or Linear Machines 95


4.1 Generalizes Single-Cell Models 96
4.2 Perceptron Learning for Winner-Take-All Groups 98

4.3 The Pocket Algorithm for Winner-Take-All Groups 98


*4.4 Kessler's Construction, Perceptron Cycling, and th e
Pocket Algorithm Proof 99
4.5 Independent Training 102
4 .6 Exercises 10 3
4 .7 Programming Projects 10 3

5 Autoassociators and One-Shot Learning 10 5


5.1 Linear Autoassociators and the Outer-Produc t
Training Rule 10 5
5 .2 Anderson's BSB Model 109
5 .3 Hopfield's Model 11 0
5 .4 The Traveling Salesman Problem 11 2
5 .5 The Cohen-Grossberg Theorem 11 5
*5 .6 Kanerva's Model 11 6
*5 .7 Autoassociative Filtering for Feedforward Networks 11 7
5 .8 Concluding Remarks 11 8
5 .9 Exercises 11 9
5 .10 Programming Projects 12 1

6 Mean Squared Error (MSE) Algorithms 123


6.1 Motivation 123
6.2 MSE Approximations 123
6 .3 The Widrow-Hoff Rule or LMS Algorithm 125
6.4 ADALINE 12 7
6.5 Adaptive Noise Cancellation 12 8
6.6 Decision-Directed Learning 12 9
6 .7 Exercises 13 1
6.8 Programming Projects 13 1

7 Unsupervised Learning 13 3
7 .1 Introduction 13 3
7 .2 k-Means Clustering 13 4
7 .3 Topology-Preserving Maps 13 6
7 .4 ART1 143
7 .5 ART2 146

7 .6 Using Clustering Algorithms for Supervised Learning 149


7 .7 Exercises 150
7 .8 Programming Projects 15 1

III Learning in Multilayer Models 15 3

8 The Distributed Method and Radial Basis Functions 15 5


8 .1 Rosenblatt's Approach 15 6
8 .2 The Distributed Method 15 7
8 .3 Examples 162
8 .4 How Many Cells? 16 3
8 .5 Radial Basis Functions 16 5
8 .6 A Variant: The Anchor Algorithm 16 7
8 .7 Scaling, Multiple Outputs, and Parallelism 16 8
8 .8 Exercises 170
8 .9 Programming Projects 17 1

*9 Computational Learning Theory and the BRD Algorithm 17 3


9 .1 Introduction to Computational Learning Theory 17 3
9 .2 A Learning Algorithm for Probabilistic Bounde d
Distributed Concepts 17 8
9 .3 The BRD Theorem 180
9 .4 Noisy Data and Fallback Estimates 18 3
9 .5 Bounds for Single-Layer Algorithms 189
9 .6 Fitting Data by Limiting the Number of Iterations 189
9 .7 Discussion 19 1
9 .8 Exercise 19 3
9 .9 Programming Project 19 3

10 Constructive Algorithms 19 5
10.1 The Tower and Pyramid Algorithms 19 5
10.2 The Cascade-Correlation Algorithm 19 8
10.3 The Tiling Algorithm 200
10.4 The Upstart Algorithm 20 1
10.5 Other Constructive Algorithms and Pruning 20 3
10.6 Easy Learning Problems 205

10.7 Exercises 20 8
10.8 Programming Projects 20 9

11 Backpropagation 21 1
11 .1 The Backpropagation Algorithm 21 2
11 .2 Derivation 21 7
11 .3 Practical Considerations 21 9
11 .4 NP-Completeness 22 4
11 .5 Comments 22 5
11 .6 Exercises 22 7
11 .7 Programming Projects 22 8

12 Backpropagation: Variations and Applications 23 1


12.1 NETtalk 23 1
12.2 Backpropagation through Time 23 3
12.3 Handwritten Character Recognition 23 5
12.4 Robot Manipulator with Excess Degrees of Freedom 23 9
12.5 Exercises 244
12.6 Programming Projects 244

*13 Simulated Annealing and Boltzmann Machines 245


13.1 Simulated Annealing 245
13.2 Boltzmann Machines 246
13 .3 Remarks 25 1
13 .4 Exercises 25 1
13 .5 Programming Project 25 1

IV Neural Network Expert Systems 25 3

14 Expert Systems and Neural Networks 25 5


14.1 Expert Systems 25 5
14.2 Neural Network Decision Systems 26 4
14.3 MACIE, and an Example Problem 26 7
14.4 Applicability of Neural Network Expert Systems 28 1
14.5 Exercise 28 2
14 .6 Programming Projects 282

15 Details of the MACIE System 28 3


15.1 Inferencing and Forward Chaining 28 3
15.2 Confidence Estimation 28 9
15.3 Information Acquisition and Backward Chaining 29 0
15.4 Concluding Comment 29 1
15.5 Exercises 29 1
15.6 Programming Projects 29 3

16 Noise, Redundancy, Fault Detection, and Bayesian


Decision Theory 29 5
16.1 The High Tech Lemonade Corporation's Problem 29 5
16.2 The Deep Model and the Noise Model 29 8
16.3 Generating the Expert System 30 0
16.4 Probabilistic Analysis 302
*16.5 Noisy Single-Pattern Boolean Fault Detection Problems 305
*16 .6 Convergence Theorem 308
16 .7 Comments 31 0
16 .8 Exercises 31 0
16.9 Programming Projects 31 1

17 Extracting Rules from Networks 31 5


17 .1 Why Rules? 31 5
17 .2 What Kind of Rules? 31 5
17 .3 Inference Justifications 31 7
17 .4 Rule Sets 322
17 .5 Conventional + Neural Network Expert Systems 325
17 .6 Concluding Remarks 328
17 .7 Exercises 329
17 .8 Programming Projects 329

Appendix: Representation Comparisons 33 1


A .1 DNF Expressions and Polynomial Representability 33 1
A.2 Decision Trees 33 8
A.3 76-)L Diagrams 342
A.4 Symmetric Functions and Depth Complexity 343

A.5 Concluding Remarks 34 6


A.6 Exercises 34 6

Bibliography 34 9
Index 361

You might also like