Chapter 6

Information Theory and Coding

• Shannon’s Capacity Theorem
• Information
• Entropy
• Shannon-Fano Coding
• Huffman Coding
• Error Control Coding
• Hamming Codes

1
 Shannon’s Capacity Theorem

• The theorem states that the capacity C of a channel which is

perturbed by AWGN (additive white Gaussian noise) is a
function of the average received signal power S, average
received noise power N, and channel bandwidth B :
C = B log 2 (1 + S / N ) bits per second …………….. [6.1]

• The theorem implies that we can transmit information at bit rate

Rb ≤ C with small bit error probability (almost error-free).
Hence, C is maximum error-free transmission rate.
• For Rb > C, it is not possible to find a code that can achieve
small error probability.

2
 Shannon’s Capacity Theorem

 S  Eb Rb 
C = B log 2 1 +  = B log 2 1 + 
 N  No B 

S = EbRb and N = NoB

S: signal power, N: noise power, Eb: signal energy per bit,
Rb: bit rate, No: noise power spectral density, B: bandwidth
Eb Rb
= 2C / B − 1
No B
Eb B C/B
=
N o Rb
(2 − 1) …………….. [6.2]

When Rb = C
Rb  Eb Rb 
= log2 1 +  …………….. [6.3]
B  No B 
3
 Shannon’s Capacity Theorem

Figure 6.1 – Bandwidth Efficiency Diagram

4
 Shannon’s Capacity Theorem

Example 6.1:

Consider a telephone circuit with a bandwidth of

3kHz. Assume that the circuit can be modelled as
an AWGN channel.

(a) What is the capacity of such a circuit if the

SNR is 30 dB?
(b) What is the SNR required for a maximum
error-free transmission rate of 4,800 bits/s?

5
 Information

 A message source produces n possible messages, each with a

probability pi :

message n
source message i with pi , i = 1 to n  pi = 1
i =1

 The message source is discrete memoryless.

 A message i is an information content conveyed by event
outcome i, and it is defined as:

1
message i = log 2 = − log 2 pi bits ………….. [6.4]
pi

 Bit (binary digit) is used as a measure of information content.

6
 Information

 Before an event occurs, if pi (probability of outcome i) is low,

we are not certain of the presence of outcome i. After the event
occurs, if outcome i is produced, then information gained is
large.
 Before an event occurs, if pi is high, we are quite certain of the
presence of outcome i. After the event occurs, if outcome i is
produced, then information gained is small.

message i
(information) message i = –log2 pi

pi (probability)
0 1
7
 Entropy

 Average amount of information over all n possible outcomes is

defined as entropy H:
n 1 n
H =  p i log 2 = −  pi log 2 pi bits per outcome ………….. [6.5]
i =1 pi i =1

 For example, consider an event of flipping a coin. There are two

possible outcomes: face A or B is displayed. Let pA (probability
of face A) = 0.1 and pB = 0.9.
Message A = – log2 pA = – log2 0.1 = 3.32 bits.
Message B = – log2 pB = – log2 0.9 = 0.15 bit.
n
H = − pi log2 pi = − pA log2 pA − pB log2 pB = 0.47 bit per outcome
i =1

8
 Entropy

Example 6.2:

A given source alphabet consists of 300 words, of

which 15 occur with probability 0.06 each and the
remaining 285 words occur with probability
0.00035 each. If 1,000 words are transmitted each
second, what is the average rate of information
transmission?

9
 Shannon-Fano Coding

 A suboptimal technique of coding based on a set of symbols

and their probabilities.
 For a given list of symbols, sort according to probability, with
decreasing probability from left to right.
 Divide the list into two parts, with the total probability of the
left half being as close as possible to that of the right half.
 The left half of the list is assigned the binary digit 0, and the
right half is assigned the digit 1.
 Continue until each symbol has become a corresponding code
leaf on the tree.

10
 Shannon-Fano Coding

0 S1 S2 S3 S4 1

S1 S2 S3 S4
0 1

S2 S3 S4

0 1
Symbol Prob. S-F Word Length
Si pi Codeword Li S3 S4
S1 0.4 0 1 bit
S2 0.3 10 2 bits
S3 0.2 110 3 bits
S4 0.1 111 3 bits
11
 Shannon-Fano Coding

Average word length Lav

n
=  pi Li
i =1

= 0.4(1) + 0.3( 2) + 0.2(3) + 0.1(3)

= 1.9 bits / codeword
Source entropy H
n
1
= 
i =1
p i log 2
pi
1 1 1 1
= 0.4 log 2 + 0.3 log 2 + 0.2 log 2 + 0.1 log 2
0 .4 0 .3 0 .2 0 .1
= 1.85 bits per symbol
H 1.85
Efficiency of the Shannon - Fano code = = = 0.97
Lav 1.9
12
 Shannon-Fano Coding

Example 6.3:

Find the Shannon-Fano codewords for a set of

symbols with probabilities as shown below:

Symbol Si Prob. pi
S1 0.4
S2 0.2
S3 0.2
S4 0.1
S5 0.1

13
 Huffman Coding

 Consider a source emitting n possible symbols Si, each with

probability pi, i = 1 to n.
 In normal coding, k bits required to encode a symbol since n =
2k. In Huffman coding, a symbol can be encoded with < k bits.
 First step in Huffman encoding is to list down the source
symbols in order of decreasing pi. The lowest pn is assigned a
‘1’ and the second lowest pn-1 is assigned a ‘0’.
 pn and pn-1 are added to form a new probability, which is placed
highest possible with remaining probabilities.
 This reduction process continues until two probabilities are left.
The resulting structure is called Huffman tree.
 We then trace back the sequences of ‘0’ and ‘1’ to construct
Huffman codewords for each symbol.
14
 Huffman Coding

Symbol Si Prob pi
S1 0.4 0.4 0.4 0.6 0
S2 0.2 0.2 0.4 0 0.4 1
S3 0.2 0.2 0 0.2 1
S4 0.1 0 0.2 1
S5 0.1 1
Symbol Si Prob. pi Huffman Codeword Word Length Li
S1 0.4 00 2 bits
S2 0.2 10 2 bits
S3 0.2 11 2 bits
S4 0.1 010 3 bits
S5 0.1 011 3 bits 15
 Huffman Coding

Average word length Lav

n
=  pi Li
i =1
= 0.4(2) + 0.2(2) + 0.2(2) + 0.1(3) + 0.1(3)
= 2.2 bits / codeword
Source entropy H
n
1
= 
i =1
pi log 2
pi
1 1 1 1 1
= 0.4 log 2 + 0.2 log 2 + 0.2 log 2 + 0.1 log 2 + 0.1 log 2
0 .4 0 .2 0 .2 0 .1 0 .1
= 2.12 bits per symbol
H 2.12
Efficiency of the Huffman code = = = 0.96
Lav 2.2
16
 Huffman Coding

 Property of Huffman codes is that they encode more frequent

(high pi) symbols with fewer bits, and less frequent (low pi)
symbols with more bits.
 Drawback of Huffman codes is that knowledge of symbols’
probabilities is required.
 More optimal than Shannon-Fano codes.

17
 Huffman Coding

Example 6.4:

Find the Huffman codewords for a set of symbols

with probabilities as shown below:
Symbol Prob.
Si pi
S1 0.4
S2 0.3
S3 0.2
S4 0.1

Determine the efficiency of the code.

18
 Error Control Coding

 When data is sent over a transmission medium, the

characteristics of the medium normally will alter the transmitted
data.
 Adverse characteristics of a medium consists of noise,
distortion, and attenuation.
 In the case of digital data transmission, these impairments may
transform binary 0s into binary 1s and vice versa.
 Error control coding is required.

19
 Error Control Coding

There are two types of error control:

1. Error detection codes

- Use redundant parity bits to detect an error. The receiving
terminal does not attempt to correct the error, but simply alerts
the transmitter to retransmit the data.
- Advantages: Decoding equipment is simpler, fewer check bits.

2. Forward error control

- Encoding process is achieved by incorporating redundant bits
to the information bits prior to transmission over a noisy
channel. The redundancy is then exploited to detect and correct
errors at the decoder end.
- Advantage: Needs only one-way link.
20
 Block Codes

 Efficiency (code rate) = k / n.

 Redundancy = (n - k) / k.
 e.g., given a code rate of 3/4,
the redundancy is 33%.

ETN3046
21
 Linear Block Codes

 A code is said to be linear if any two codewords in the code can

be added in modulo-2 arithmetic (XOR logic operation) to
produce a third codeword in the code.
 If we take two information sequences and add them together,
symbol by symbol, before they go into a linear encoder, the
resulting codeword should be the sum of the two codewords
that correspond to the individual information sequence.
Information Codeword The all-zero word is
Example of a always a codeword.
linear 00 00000
This is because if we
systematic code 01 01011 add a codeword to
10 10101 itself we obtain a
11 11110 string of zeros.

The information 11 could be produced by addition of 10 and 01. Adding the

corresponding codewords gives 10101 + 01011 = 11110, which is the correct
codeword for 11. 22
 Linear Systematic Codes

 The block codes studied above have the information bits appear
in the codeword unchanged with some parity bits added.
 In this case, the code is said to be systematic.
Information Codeword
00 00000
01 01011
10 10101
11 11110

 Indeed, the information can appears at the start/end of the

codeword.
 The use of systematic codes simplifies design of the decoder.
 Most practical block codes used are in systematic form.
ETN3046
23
 Error Control Coding

 A code is said to be linear if any two codewords in the code can

be added in modulo-2 arithmetic (XOR logic operation) to
produce a third codeword in the code.

k information bits m = (n – k) parity check bits

n-bit codeword
Figure 6.2 – An (n, k) linear block code

 Efficiency (code rate) = k / n. ………….. [6.6]

 Redundancy = (n - k) / k. ………….. [6.7]

 For example, given a code rate of 3/4, the redundancy is 33%.

24
 Hamming Codes

 This is an early linear block code pioneered by Richard

Hamming at the Bell Telephone Laboratories in 1948.
 Codeword length is constrained to n = 2m – 1.

 The systematic form of the generator matrix is given by

G = [I k QT ] ………….. [6.8]
Ik = k × k identity matrix
QT = k × m matrix which rows are the k m-tuples of weight 2 or
larger. QT is the transpose of Q.

 The corresponding parity-check matrix is given by

H = [Q I m ] ………….. [6.9]
Q = m × k matrix
Im = m × m identity matrix
25
 Hamming Codes

 Consider a Hamming (7, 4) code, in which three check bits (m1,

m2, and m3) are combined with four information bits (k1, k2,
k3, and k4) to produce a codeword of length n = 7.

1 1 0
1 0 1
QT =  
0 1 1
 
1 1 1
1 0 0 0 1 1 0
0  1 1 0 1 1 0 0
1 0 0 1 0 1
G= , H = 1 0 1 1 0 1 0
0 0 1 0 0 1 1  
  0 1 1 1 0 0 1
0 0 0 1 1 1 1

26
 Hamming Codes

 Hamming distance - the number of positions by which any

two codewords in a code differ.
 If the minimum distance of a code is d, then (d – 1) errors can
be detected and int[(d – 1) / 2] error can be corrected.
 For the Hamming (7, 4) code, d = 3. Hence, 2 errors can be
detected, and 1 error can be corrected.

Encoding Algorithm

 For encoding,
uG = v ………….. [6.10]
where v is the vector of the encoded bits, u is the vector of k
information bits. The summation is performed using modulo-
2 addition.
27
 Hamming Codes

 For example, to determine the codeword for the message bits

1 0 1 0 using a Hamming (7, 4) code and the previous generator
matrix

1 0 0 0 1 1 0
0 1 0 0 1 0 1 
uG = [1 0 1 0] 
0 0 1 0 0 1 1 
 
 0 0 0 1 1 1 1 
= (1 ⊕ 0 ⊕ 0 ⊕ 0)(0 ⊕ 0 ⊕ 0 ⊕ 0)(0 ⊕ 0 ⊕ 1 ⊕ 0)(0 ⊕ 0 ⊕ 0 ⊕ 0)
(1 ⊕ 0 ⊕ 0 ⊕ 0)(1 ⊕ 0 ⊕ 1 ⊕ 0)(0 ⊕ 0 ⊕ 1 ⊕ 0)
=1 0 1 0 1 0 1

28
 Hamming Codes

Table 6.1 – Complete Set of Codewords for Hamming (7, 4) Code

No Info Bits Check Bits Weight

0 0000 000 0
1 0001 111 4
2 0010 011 3
3 0011 100 3
4 0100 101 3
5 0101 010 3
6 0110 110 4
7 0111 001 4
8 1000 110 3
9 1001 001 3
10 1010 101 4
11 1011 010 4
12 1100 011 4
13 1101 100 4
14 1110 000 3
15 1111 111 7

29
 Hamming Codes

Error Correction Strategy

 Suppose a codeword t is transmitted and an erroneous bit
stream r is received.
r=t+e ………….. [6.11]

where r, t, and e (error) are column vectors.

Hr = Ht + He = He = s (syndrome) ……….. [6.12]

 Syndrome s is a vector having (n – k) components.

 If s = 0, assume that no errors have occurred.
 If s ≠ 0, use the error pattern to locate and correct the error.

30
 Hamming Codes

 For example, for Hamming (7, 4) code, a codeword

0 1 0 1 1 1 0 is received. Locate and correct the bit in error.
Hr = s
0
1
 
1 1 0 1 1 0 0 0  0 ⊕ 1 ⊕ 0 ⊕ 1 ⊕ 1 ⊕ 0 ⊕ 0  1
1 0 1 1 0 1 0 1 = 0 ⊕ 0 ⊕ 0 ⊕ 1 ⊕ 0 ⊕ 1 ⊕ 0 = 0
      
0 1 1 1 0 0 1 1 0 ⊕ 1 ⊕ 0 ⊕ 1 ⊕ 0 ⊕ 0 ⊕ 0 0
 
1
0

Error is in bit number 5.

Corrected codeword: 0 1 0 1 0 1 0.
31