Soft Computing

Unit -3
Dr. Pallavi Malavath

Dr. Pallavi Malavath

 Syllabus
• Associative Memory: Auto associative
Memory, Hetero-associative Memory,
Bidirectional Associative Network,
Applications of Associative Memory
• Associate Models: Hopfield Network,
Boltzmann Network, Simulated Annealing,
Applications of Networks

Dr. Pallavi Malavath

 Associative Memory
• Associative memory is a pattern warehouse that stores
a set of patterns as memories.
• The network will recall the pattern associated with it
from the memory when an input pattern is given.
• The pattern may be the same as the key pattern or it
can be a different pattern.
• On the basis of pattern retrieval, the associative
network can be classified into auto associative memory
and Hetero associative memory also known as auto-
correlators and hetero-correlators.

Dr. Pallavi Malavath

• These type of memories are called content
addressable memories(CAM).
• The input pattern is correlated to the patterns stored
in the memory.
• The correlation helps to retrieve the pattern closer to
the input pattern .
• The memory should consist of unique patterns so
that there may not be any ambiguity in the retrieval
of the patterns

Dr. Pallavi Malavath

• This is Hetero-
associative memory
because input pattern x
is different from the
recall pattern y.
• If x and y are same,
then the model is an
auto-associative
memory.

Dr. Pallavi Malavath

• These models use feed forward network or
recurrent network and are called as neural
associative network.
• The selection of network depends on the
association capability.
• Hebbian rules are used for learning.
• Auto correlators are useful for the retrieval of
images even when they are distorted( noisy).

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Auto-associative Memory
• Associative net uses Hebb rule, which is the product of input
xi and output yi or an outer product rule, which is an
alternate method of Hebb rule.
• These associative net has same number of neurons in the
input and output layers as the training and the target output
vectors are same.
• The association between the patterns is stored by their
weights.
• The performance of the net is based on its recall of patterns
from the noisy input.
• The initial weights are set to ‘0’.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
Dr. Pallavi Malavath
Dr. Pallavi Malavath
 Hetero-associative Memory
• Hetero-associative Memory retrieves the
pattern from the memory where the input
and the target patterns are different.
• These networks are trained using the Hebb or
delta rule.
• The correlation between the input pattern and
the output pattern is determined by the
weights.

Dr. Pallavi Malavath

• These networks respond well in noisy and
incomplete patterns.
• The input layer has n nodes and the output layer has
m nodes representing the patterns.
• The interconnection between the nodes in the layers
are linked by weights that are defined using the Hebb
rule.
• The training algorithm is similar to the auto
associative memory.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Bidirectional Associative Memory
(BAM)
• Bidirectional Associative Memory (BAM) is
a supervised learning model in Artificial Neural
Network.
• This is hetero-associative memory, for an input
pattern, it returns another pattern which is
potentially of a different size. This phenomenon is
very similar to the human brain.
• A Recurrent Neural Network (RNN) is needed to
receive a pattern of one set of neurons as an
input and generate a related, but different,
output pattern of another set of neurons.

Dr. Pallavi Malavath

 Why BAM is required?
• The main objective to introduce such a network
model is to store hetero-associative pattern pairs.
This is used to retrieve a pattern given a noisy or
incomplete pattern.
• BAM Architecture: When BAM accepts an input of n-
dimensional vector X from set A then the model
recalls m-dimensional vector Y from set B. Similarly
when Y is treated as input, the BAM recalls X.

Dr. Pallavi Malavath

BAM Architecture

Dr. Pallavi Malavath

• BAM can be classified into 3 types:
• 1) Discrete BAM
• 2) Continuous BAM
• 3) Exponential BAM
• The BAM net resembles the single layer
perceptron network and the interconnection
between the layers in the net is bi-
directional.

Dr. Pallavi Malavath

• Signals can be sent from the layer X to layer Y with W
as the weight matrix and from the layer Y to layer X
with the weight
. matrix Wt.
• The layers are not called as the input and output
layers as the input/output pattern can be given at
the either side.
• During the training process, the weights are adjusted
in both forward and backward movement of signals
and this oscillation will converge when they reach
the equilibrium

Dr. Pallavi Malavath

Dr. Pallavi Malavath
Dr. Pallavi Malavath
• A step function is used as an activation
function.
• The bipolar value improves the performance
of Hebbain net.
• Activation function at layer Y:

Dr. Pallavi Malavath

• This signal is send to layer X.
• Activation function at layer X:

Dr. Pallavi Malavath

 Continuous BAM
• Continuous Bidirectional Associative Memory (BAM)
is an extension of the original BAM model, which is
a type of neural network designed to associate
patterns bi-directionally.
• Continuous BAM is particularly useful for
applications where the input and output patterns
are represented by continuous values rather than
binary or discrete ones.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Features of Continuous BAM
• Bidirectional Mapping: The network learns
associations in both directions between two
pattern sets, X and Y. Once trained, it can
retrieve Y given X, or vice versa.
• Continuous Activation Functions: Unlike the
traditional BAM, which uses binary thresholds,
the continuous version employs smooth
activation functions, such as sigmoid or
hyperbolic tangent, to handle real-valued
inputs and outputs.
Dr. Pallavi Malavath
• Energy Function: Continuous BAM networks
define an energy function that decreases
during the association process.
• This guarantees convergence to a stable state
making the network suitable for memory
retrieval tasks.

Dr. Pallavi Malavath

• Learning Rule: The associative weights W
between X and Y are updated using a
generalized Hebbian learning rule:

•Feedback Dynamics: The continuous BAM

uses feedback loops to iteratively refine the
states of neurons in the X-layer and Y-layer
until they converge to stable patterns.

Dr. Pallavi Malavath

 Applications
• Continuous BAM networks are applied in
fields where continuous-valued pattern
associations are necessary, such as:
• Signal processing
• Control systems
• Pattern recognition
• Data compression
• Robotics

Dr. Pallavi Malavath

 Exponential Bidirectional Associative
Memory
• Exponential Bidirectional Associative Memory
(EBAM) is a neural network model that
extends the traditional Bidirectional
Associative Memory (BAM).
• It uses an exponential function to encode and
decode associative patterns, enhancing the
network's ability to store and retrieve
information efficiently.

Dr. Pallavi Malavath

 Features of EBAM:
• Bidirectional Mapping: EBAM, like BAM, supports
two-way (bidirectional) associations between input
patterns (X) and output patterns (Y). It allows
retrieving X from Y and vice versa.
• Exponential Function: The core enhancement in
EBAM is its use of an exponential activation function.
This improves the network's ability to handle non-
linear associations, increasing its capacity and
robustness in pattern recognition.

Dr. Pallavi Malavath

• Energy-Based Model: Similar to BAM, EBAM is often
described in terms of an energy function. The exponential
function helps minimize this energy during learning, leading
to faster convergence and improved stability.
• Learning Rule: The Hebbian learning rule (or its variants) is
typically used to train the associative weights. In EBAM, the
weights (WWW) are updated to strengthen the relationship
between co-occurring patterns:

Dr. Pallavi Malavath

• Storage Capacity: The exponential modification
enhances storage capacity compared to
traditional BAM, allowing EBAM to store and
retrieve a larger number of patterns.
• Applications: EBAM is particularly useful in fields
like:
– Pattern recognition
– Associative memory systems
– Data classification and clustering
– Signal processing

Dr. Pallavi Malavath

• Advantages of EBAM:
• Improved Accuracy: Handles complex and
non-linear relationships better than BAM.
• Faster Convergence: The exponential
dynamics allow quicker stabilization during
retrieval.
• Increased Capacity: Capable of storing a larger
number of associative patterns.

Dr. Pallavi Malavath

 Applications of Associative Memory
• Character recognition.
• Fault detection.
• Categorization and classification problems.
• Feature extraction
• Analysis of document images and identification of
object within the image.
• Detection analogies in a structural safety
management.
• Design for VLSI implementation
Dr. Pallavi Malavath
 Hopfield Network
• A Hopfield Network is a type of recurrent
artificial neural network that serves as an
associative memory system.
• It is generally used in performing auto-
association and optimization tasks. It is
calculated using a converging interactive
process and it generates a different response
than our normal neural nets.

Dr. Pallavi Malavath

• Discrete Hopfield Network
• It is a fully interconnected neural network where each
unit is connected to every other unit. It behaves in a
discrete manner, i.e. it gives finite distinct output,
generally of two types:
• Binary (0/1)
• Bipolar (-1/1)
• The weights associated with this network are symmetric
in nature and have the following properties.
• wij=wji
• wii=0

Dr. Pallavi Malavath

 Structure & Architecture of Hopfield Network

• Each neuron has an inverting and a non-

inverting output.
• Being fully connected, the output of each
neuron is an input to all other neurons but not
the self.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Boltzmann Network
• What is a Boltzmann machine?
• A Boltzmann machine is a type of stochastic neural
network that can learn complex data distributions
through its network of interconnected nodes.
• How does a Boltzmann machine work?
• It works by using a learning algorithm to adjust the
weights of the connections, optimizing the network
to represent complex patterns in data through
probability distribution.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Overview of Boltzmann Machines
• Introduced by Geoffrey Hinton and Terry
Sejnowski in 1985.
• Uses a network of neurons (or units) that are
symmetrically connected.
• The network follows an energy-based model
inspired by the Boltzmann distribution in
thermodynamics.
• It operates using stochastic (randomized)
updates to find optimal solutions.
Dr. Pallavi Malavath
 Structure of a Boltzmann Machine
A Boltzmann Machine consists of:
• Visible Units (V): Represent the input and output of
the system.
• Hidden Units (H): Used to capture dependencies and
patterns in the data.
• Bidirectional Weighted Connections: Every unit is
connected to others (fully connected network).
• Energy Function (E): The probability of a state is
determined by its energy level.

Dr. Pallavi Malavath

Dr. Pallavi Malavath
 Learning in Boltzmann Machines
• Learning occurs through adjusting weights to
minimize energy and maximize the likelihood
of observed data.
• Uses Gibbs Sampling and Contrastive
Divergence (CD) for training.
• The network updates neuron states using the
sigmoid activation function.

Dr. Pallavi Malavath

 Applications in Soft Computing
• Optimization Problems (e.g., Travelling
Salesman Problem, Job Scheduling).
• Feature Learning & Dimensionality
Reduction.
• Image Recognition & Processing.
• Natural Language Processing (NLP).
• Recommendation Systems.

Dr. Pallavi Malavath

 Pros & Cons of Boltzmann Machines (BMs)
• Pros (Advantages)
• Efficient in Optimization – Useful for solving
complex combinatorial problems(computationally
difficult- Traveling Salesman Problem, Sudoku &
Puzzles, Knapsack Problem)
• Supports Unsupervised Learning – Can learn
patterns without labeled data.
• Probabilistic Model – Handles uncertainty and
stochastic learning well.
• Good for High-Dimensional Data – Effective in
feature extraction & dimensionality reduction.

Dr. Pallavi Malavath

• Cons (Disadvantages)
• High Computational Cost – Full BMs require
exponential time to train.
• Slow Convergence – Training with Gibbs
Sampling is inefficient.
• Difficult to Train – Gradient calculations are
complex and unstable.
• Requires Large Datasets – Needs significant
data to perform well.

Dr. Pallavi Malavath

 Simulated Annealing
• What is Simulated Annealing?
• Simulated Annealing (SA) is a probabilistic
optimization algorithm inspired by the annealing
process in metallurgy, where materials are heated
and then cooled slowly to remove defects and
achieve a stable structure.
• In computing, SA is used to find near-optimal
solutions for difficult combinatorial and optimization
problems.

Dr. Pallavi Malavath

Simulated Annealing

Dr. Pallavi Malavath

• SA searches for the global optimum by
allowing occasional moves to worse solutions
to escape local optima.
• It gradually reduces the probability of
accepting worse solutions as the
"temperature" decreases.

Dr. Pallavi Malavath

 How Simulated Annealing Works?

• Initialize: Start with an initial solution and set

a high temperature (T).
• Generate New Solution: Make a small random
change to the current solution.
• Evaluate the Change:
– If the new solution is better, accept it.

Dr. Pallavi Malavath

• If the new solution is worse, accept it with a
probability

•Cool Down: Gradually decrease the temperature T

(e.g., using T=0.9TT = 0.9TT=0.9T).
•Repeat Steps 2–4 until T ≈ 0 or a stopping criterion is
met.

Dr. Pallavi Malavath

 Applications of Associative Memory
• Recalling and Reconstruct corrupted patterns.
• Large-Scale computational intelligence
systems.
• Handwriting recognition software.
• Medical image processing.
• Classification and prediction of data analysis.
• VLSI design, Routing in Network etc…

Dr. Pallavi Malavath