Autoencoders

Autoencoders
• Autoencoders are a specific type of feedforward neural networks
where the input is the same as the output.
• They compress the input into a lower-dimensional code and then
reconstruct the output from this representation.
• The code is a compact “summary” or “compression” of the input,
also called the latent-space representation.
• The primary applications are dimensionality reduction and specific
data compression.
Autoencoders
• An autoencoder consists of 3 components: encoder, code and
decoder. The encoder compresses the input and produces the code,
the decoder then reconstructs the input only using this code.

To build an autoencoder we need 3 things: an encoding method,

decoding method, and a loss function to compare the output with the
target.
Autoencoders: Properties
• Data-specific: Autoencoders are only able to meaningfully compress
data similar to what they have been trained on. Since they learn
features specific for the given training data, they are different than a
standard data compression algorithm like gzip. So we can’t expect an
autoencoder trained on handwritten digits to compress landscape
photos.
• Lossy: The output of the autoencoder will not be exactly the same as
the input, it will be a close but degraded representation. If you want
lossless compression they are not the way to go.
• Unsupervised: Autoencoders are considered an unsupervised learning
technique since they don’t need explicit labels to train on. But to be
more precise they are self-supervised because they generate their
own labels from the training data.
 Architecture

Both the encoder and decoder are fully-connected feedforward neural

networks, Note that the decoder architecture is the mirror image of the encoder.
This is not a requirement but it’s typically the case. The only requirement is the
dimensionality of the input and output needs to be the same.
Hyperparameters
• Code size: number of nodes in the middle layer. Smaller size results in more compression.
• Number of layers: the autoencoder can be as deep as we like. In the figure above there
are 2 layers in both the encoder and decoder, without considering the input and output.
• Number of nodes per layer: the autoencoder architecture we’re working on is called
a stacked autoencoder since the layers are stacked one after another. Usually stacked
autoencoders look like a “sandwitch”. The number of nodes per layer decreases with
each subsequent layer of the encoder, and increases back in the decoder. Also the
decoder is symmetric to the encoder in terms of layer structure. As noted above this is
not necessary and we have total control over these parameters.
• Loss function: we either use mean squared error (mse) or binary crossentropy. If the
input values are in the range [0, 1] then we typically use crossentropy, otherwise we use
the mean squared error. For more details check out this video.
• Autoencoders are trained the same way as ANNs via backpropagation.
Autoencoder Components
Autoencoders consists of 4 main parts:
• Encoder: In which the model learns how to reduce the input dimensions
and compress the input data into an encoded representation.
• Bottleneck: which is the layer that contains the compressed
representation of the input data. This is the lowest possible dimensions
of the input data.
• Decoder: In which the model learns how to reconstruct the data from the
encoded representation to be as close to the original input as possible.
• Reconstruction Loss: This is the method that measures measure how
well the decoder is performing and how close the output is to the original
input.
Training of an Auto-encoder for data compression
• Step 1: Encoding the input data
Training of an Auto-encoder for data compression
• Step 2: Decoding the input data
 Training of an Auto-encoder for data compression
• Step 3: Backpropagating the error

The above-described training process is

reiterated several times until an acceptable
level of reconstruction is reached.
After the training process, only the encoder
part of the Auto-encoder is retained to
encode a similar type of data used in the
training process.
Autoencoders
The different ways to constrain the network are:-
• Keep small Hidden Layers: If the size of each hidden layer is kept as small
as possible, then the network will be forced to pick up only the representative
features of the data thus encoding the data.
• Regularization: In this method, a loss term is added to the cost function
which encourages the network to train in ways other than copying the input.
• Denoising: Another way of constraining the network is to add noise to the
input and teaching the network how to remove the noise from the data.
• Tuning the Activation Functions: This method involves changing the
activation functions of various nodes so that a majority of the nodes are
dormant thus effectively reducing the size of the hidden layers.
Variations of autoencoders
The different variations of Auto-encoders are:-
• Denoising Auto-encoder: This type of auto-encoder works on a
partially corrupted input and trains to recover the original
undistorted image. this method is an effective way to constrain the
network from simply copying the input.
• Sparse Auto-encoder: This type of auto-encoder typically contains
more hidden units than the input but only a few are allowed to be
active at once. This property is called the sparsity of the network.
The sparsity of the network can be controlled by either manually
zeroing the required hidden units, tuning the activation functions or
by adding a loss term to the cost function.
Variations of autoencoders

• Convolutional Auto-encoder: Convolutional auto-encoders are a type

of autoencoder that use convolutional neural networks (CNNs) as their
building blocks. The encoder consists of multiple layers that take a
image or a grid as input and pass it through different convolution
layers thus forming a compressed representation of the input. The
decoder is the mirror image of the encoder it deconvolves the
compressed representation and tries to reconstruct the original image.
• Variational Auto-encoder: This type of auto-encoder makes strong
assumptions about the distribution of latent variables and uses
the Stochastic Gradient Variational Bayes estimator in the training
process.
Denoising autoencoders
Sparse autoencoders