[go: up one dir, main page]
Scribd
Upload
125 views14 pages

Intro to Information Theory

This document discusses information theory and source coding techniques: 1) Information theory defines information as being inversely related to the probability of an event - the less likely an event, the more information it contains. 2) Source coding aims to efficiently describe information sources using techniques like variable length coding to represent frequent items with fewer bits. 3) Examples show how a fixed length code and two variable length codes can represent letters, with the variable codes using fewer total bits.

Uploaded by

Matrix Bot
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
125 views14 pages

Intro to Information Theory

This document discusses information theory and source coding techniques: 1) Information theory defines information as being inversely related to the probability of an event - the less likely an event, the more information it contains. 2) Source coding aims to efficiently describe information sources using techniques like variable length coding to represent frequent items with fewer bits. 3) Examples show how a fixed length code and two variable length codes can represent letters, with the variable codes using fewer total bits.

Uploaded by

Matrix Bot
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Information Theory

What is information

 The information content of any message is closely related to the past


knowledge of the occurrence of event and the level of uncertainty it contains
with respect to the recipient of the message •

 The amount of information received from the knowledge of occurrence of an


event is inversely related to the probability or the likelihood of occurrence of
the event.

 The message related to an event least likely to occur contains more


information. •
What is information

Information I(mk) = log(1/Pk) = -log(Pk)

Example: If two binary digits 1 and 0 occur with equal


probability and are correctly detected at the receiving
end, then the information content in each digit is 1 bit.
I(0 or 1) = -log2(1/2) = 1 bit
Source Coding: Goal is to find an efficient description of
information sources – Reduce required bandwidth – Reduce
memory to store
Source Coding
In a fixed-length code (FLC) each codeword has the same length.

In a variable-length code (VLC) codewords may have different


lengths.
We should represent the more frequently occurring letters by fewer
number of bits and less frequently occurring letters by larger number
of bits.

Prefix Code: It is a variable length code in which no codeword is


a prefix of another codeword.
Source Coding (Example)
Represent first eight letters of English alphabet (A to
H) using FLC and VLC.
The fixed length code for this set of letters would be :

Letter Codeword Letter Codeword


A 000 E 100
B 001 F 101
C 010 G 110
D 011 H 111
Source Coding (Example)
VLC1
Letter Codeword Letter Codeword
A 00 E 101
B 010 F 110
C 011 G 1110
D 100 H 1111

VLC2
Letter Codeword Letter Codeword
A 0 E 10
B 1 F 11
C 00 G 000
D 01 H 111
Source Coding (Example)
“A BAD CAB”
FLC: 000 001 000 011 010 000 001 (bits=21)
VLC 1: 00 010 00 100 011 00 010 (bits=18)
VLC 2: 0 1001 0001 (bits=9)
Huffman Coding (VLC with symbols not
equally probable)
Steps for Huffman Coding
1)Arrange the source symbols in a decreasing order of their probabilities.
2) Take the bottom two symbols and tie them together, add the probabilities of the
two symbols and write it on the combined node. Label the two branches with ‘1’ and
‘0’.
3) Treat this sum of probabilities as a new probability associated with a new symbol.
Again repeat step 2.
4) Continue the steps until only one probability left (and should be 1). This completes
the construction of the Huffman code tree.
5) To find out the prefix codeword for any symbol, follow the branches from the final
node- back to the symbol. This gives the codeword for the symbol.


CYCLIC CODES

You might also like