This member-only story is on us. Upgrade to access all of Medium.

Member-only story

What is an encoder decoder model?

Encoder Decoder is a widely used structure in deep learning and
through this article, we will understand its architecture

Nechu BM · Follow
Published in Towards Data Science · 4 min read · Oct 7, 2020

321 1

Photo by Michael Dziedzic on Unsplash

In this post, we introduce the encoder decoder structure in some cases

known as Sequence to Sequence (Seq2Seq) model. For a better
understanding of the structure of this model, previous knowledge on RNN is
helpful.

When do we use an encoder decoder model?

1-Image Captioning
Encoder decoder models allow for a process in which a machine learning
model generates a sentence describing an image. It receives the image as the
input and outputs a sequence of words. This also works with videos.
 ML output: ‘Road surrounded by palm trees leading to a beach’, Photo by Milo Miloezger on Unsplash

2-Sentiment Analysis
These models understand the meaning and emotions of the input sentence
and output a sentiment score. It is usually rated between -1 (negative) and 1
(positive) where 0 is neutral. It is used in call centers to analyse the evolution
of the client’s emotions and their reactions to certain keywords or company
discounts.

Image by the author

3-Translation
This model reads an input sentence, understands the message and the
concepts, then translates it into a second language. Google Translate is built
upon an encoder decoder structure, for more detail follow this paper.

Image by the author

What is an encoder decoder model?

The best way to understand the concept of an encoder-decoder model is by
playing Pictionary. The rules of the game are very simple, player 1 randomly
picks a word from a list and needs to sketch the meaning in a drawing. The
role of the second player in the team is to analyse the drawing and identify
the word which it describes. In this example we have three important
elements player 1(the person that converts the word into a drawing), the
drawing (rabbit) and the person that guesses the word the drawing
represents (player 2). This is all we need to understand an encoder decoder
model, below we will build a comparative of the Pictionary game and an
encoder decoder model for translating Spanish to English.

Pictionary Game, Image by the author

If we translate the above graph into machine learning concepts, we would

see the below one. In the following sections we will go through each
component.

Encoder Decoder Model, Image by the author

1-Encoder (Picturist)
Encoding means to convert data into a required format. In the Pictionary
example we convert a word (text) into a drawing (image). In the machine
learning context, we convert a sequence of words in Spanish into a two-
dimensional vector, this two-dimensional vector is also known as hidden
state. The encoder is built by stacking recurrent neural network (RNN). We
use this type of layer because its structure allows the model to understand
context and temporal dependencies of the sequences. The output of the
encoder, the hidden state, is the state of the last RNN timestep.

Encoder, Image by the author

2-Hidden State (Sketch)

The output of the encoder, a two-dimensional vector that encapsulates the
whole meaning of the input sequence. The length of the vector depends on
the number of cells in the RNN.

Encoder and hidden state, Image by the author

3-Decoder
To decode means to convert a coded message into intelligible language. The
second person in the team playing Pictionary will convert the drawing into a
word. In the machine learning model, the role of the decoder will be to
convert the two-dimensional vector into the output sequence, the English
sentence. It is also built with RNN layers and a dense layer to predict the
English word.
 Open in app

Search Write

Decoder, Image by the author

Conclusion
One of the major advantages of this model is that the length of the input and
output sequences may differ. This opens the door for very interesting
applications such as video captioning or question and answer.

The major limit of this simple encoder decoder model is that all the
information needs to be summarized in one dimensional vector, for long
input sequences that can be extremely difficult to achieve. Having said that,
understanding encoder decoder models is key for the latest advances in NLP
because it is the seed for attention models and transformers. In the next
article, we will follow the process of building a translation model with an
encoder decoder structure. Then we will continue by exploring the attention
mechanism in order to achieve higher accuracy.

How to build an encoder decoder translation model using LSTM

with Python and Keras.
Follow this step by step guide to build an encoder decoder model
and create your own translation model.
towardsdatascience.com

Machine Learning Artificial Intelligence Deep Learning AI Neural Networks

Written by Nechu BM Follow

183 Followers · Writer for Towards Data Science

Data Scientist & Entrepreneur! Learn Artificial Intelligence and Machine Learning to
become a Linchpin ➜ https://bit.ly/36vajnu

More from Nechu BM and Towards Data Science

Nechu BM in Towards Data Science Damian Gil in Towards Data Science

How to build an encoder decoder Mastering Customer Segmentation

translation model using LSTM wit… with LLM
Follow this step by step guide to build an Unlock advanced customer segmentation
encoder decoder model and create your ow… techniques using LLMs, and improve your…

8 min read · Oct 20, 2020 24 min read · Sep 27

205 3 3.4K 25

Adrian H. Raudaschl in Towards Data Science Nechu BM in The Startup

Forget RAG, the Future is RAG- Introduction to Recurrent Neural

Fusion Networks (RNN)
The Next Frontier of Search: Retrieval How are computers able to understand
Augmented Generation meets Reciprocal… context? What is a recurrent neural network…

· 10 min read · Oct 6 · 6 min read · Jun 10, 2020

1.7K 23 220

See all from Nechu BM See all from Towards Data Science

Recommended from Medium

 AL Anany Unbecoming

The ChatGPT Hype Is Over — Now 10 Seconds That Ended My 20 Year

Watch How Google Will Kill… Marriage
It never happens instantly. The business It’s August in Northern Virginia, hot and
game is longer than you know. humid. I still haven’t showered from my…

· 6 min read · Sep 1 · 4 min read · Feb 16, 2022

16.8K 492 68K 981

Lists

Natural Language Processing Predictive Modeling w/

755 stories · 338 saves Python
20 stories · 529 saves

AI Regulation Generative AI Recommended

6 stories · 162 saves Reading
52 stories · 341 saves

Jonte Dancker in Towards Data Science Minhajul Hoque

A Brief Introduction to Recurrent A Comprehensive Overview of

Neural Networks Transformer-Based Models:…
An introduction to RNN, LSTM, and GRU and Transformers are a type of deep learning
their implementation architecture that have revolutionized the fiel…

12 min read · Dec 26, 2022 5 min read · Apr 30

413 5 9

Amirhossein Abaskohi Tomas Vykruta

Navigating Transformers: A Understanding Causal LLM’s,
Comprehensive Exploration of… Masked LLM’s, and Seq2Seq: A…
Introduction In the world of natural language processing
(NLP), choosing the right training approach i…

9 min read · Aug 16 7 min read · Apr 30

19 30

See more recommendations