Chapter 7: Channel Coding

Contents
Error control
Redundancy
Hamming distance & error correction
Error correction
Block codes, Hamming Codes

2
Channel Coding – Error Control

Source Channel Bandpass

Information Baseband
encode encode mod.
Source mod.

Digital modulation

Channel
Digital demodulation

Source Channel Demod. &

Information Detect
decode decode Sample
Sink

3
Error Control …
Digital transmission systems introduce errors
Applications require certain reliability level
Data applications require error-free transfer
Voice & video applications tolerate some
errors
Error control (or channel) coding ensures a data
stream is transmitted to a certain level of
accuracy despite errors
Usually build on top of frames (block of bits or
codeword)

4
Error Control …
Two basic approaches:
Error detection – are there incorrect bits?
Error correction – repair any mistakes that have happened?
Forward error correction (FEC)
Invest effort before error happened
Backward error correction
Invest effort after error happened; try to repair it

Error control

Error detection Error correction

Forward error Backward error

correction correction

5
Error Control – Redundancy
Any form of error control requires redundancy in the
frames/codewords
Without redundancy
A frame of length k can represent 2k differentframes
All of them are legal!
How could a receiver possibly decide that one legal frame
is not the one that had originally been transmitted?
Not possible!
000….001
000….000
111….111
Set of all 111….110
possible
frames Set of all legal
frames

6
 Error Control – Redundancy …
Core idea: Declare some of the possible messages illegal!
Still need to be able to express 2k legal frames
! More than 2k possible frames are required
! More than k bits are required in a frame
Use frames with n > k total length
r =n-k are the redundant bits (typically, as header or trailer)
Having more possible than legal frames allows receiver to
detect illegal frames
Set of all legal
frames
000….001010
000….000000
111….111011
111….110110
Set of all 000….000010 111….111111
possible 000….000011
111….111101

7
How do illegal messages help with detecting bit
errors?

Transmitter only sends legal frame

Physical medium/receiver might corrupt some bits
Hope: A legal frame is only corrupted into an illegal
message
But one legal frame is never turned into another legal frame
Necessary to realize this hope:
Physical medium only alters up to a certain number of bits (by
assumption) – say, k bits per frame
This is only an assumption!
How does it relate to the BER or the SNR?
Legal messages are sufficiently different so that it is not possible to
change one legal frame into another by altering at most b bits

8
 Altering frames by changing bits
Suppose the following frames are the only legal bit
patterns: 0000, 0011, 1100, 1111

0001
0000 0011
Lines connect frames
0010
that only differ in a single
0101, 0110,
bit = that can be
1000 0100 1001, 1010 0111 1011 converted into each
other by flipping one bit
1101

1100 1110 1111 Here: No single bit error

can convert one legal
frame into another one!
uvxy – legal frame abcd – illegal frame

9
Simple Redundancy Example: Parity
A simple rule to construct 1 redundant bit (i.e., n=k+1):
Parity
Odd parity: Add one bit, choose its value such that the number of
1’s in the entire message is odd
Even parity: Add one bit, choose its value such that the number of
1’s in the entire message is even

Example:
Original message without redundancy: 01101011001
Odd parity: 011010110011
Even parity: 011010110010

Parity bit used in ASCII code

10
How good is the single parity check code?
Redundancy: single parity check code adds 1 redundant bit
per k information bits
Overhead = 1/(k + 1)

Coverage: For even parity all error patterns with odd # of

errors can be detected
An error pattern is a binary (k + 1)-tuple with 1s where errors occur
and 0’s elsewhere
Of 2k+1 binary (k + 1)-tuples, ½ areodd
So 50% of error patterns can be detected

Is it possible to detect more errors if we add more check

bits?
Yes, with the right codes

11
Two-Dimensional Parity Check
More parity bits to improve coverage
Arrange information as columns
Add single parity bit to each column
Add a final “parity” column
Used in early error control systems
1 0 0 1 0 0
0 1 0 0 0 1
Last column consists
1 0 0 1 0 0 of check bits for each
row
1 1 0 1 1 0
1 0 0 1 1 1
Bottom row consists of
check bit for each column

12
Error-detecting capability
1 0 0 1 0 0 1 0 0 1 0 0
0 0 0 0 0 1 0 0 0 0 0 1
Two
1 0 0 1 0 0 One 1 0 0 1 0 0
errors
error
1 1 0 1 1 0 1 0 0 1 1 0
1, 2, or 3 errors can
1 0 0 1 1 1 1 0 0 1 1 1 always be detected;
Not all patterns >4
errors can be detected
1 0 0 1 0 0 1 0 0 1 0 0
0 0 0 1 0 1 0 0 0 1 0 1
1 0 0 1 0 0 Three 1 0 0 1 0 0
Four
errors 1 0 0 0 1 0
1 0 0 1 1 0 errors
(undetectable)
1 0 0 1 1 1 1 0 0 1 1 1
Arrows indicate failed check bits

13
 Hamming distance & error correction
Error patterns are o o
o x x o
introduced by channels Poor
o x x x o o distance
These error patterns map x x properties
transmitted codeword to o o
o o o
nearby n-tuple
If codewords close to
x = codewords
each other then detection o = non-codewords
failures will occur
Good codes should o x
x o
maximize separation o Good
o o
between codewords x o x o x distance
o o properties
o o
x o x

14
 Altering frames by changing bits
Example: Suppose the following frames are the only legal
bit patterns: 0000, 0011, 1100, 1111
Lines connect frames
0001 that only differ in a single
0000 0011 bit = that can be
0010 converted into each
other by flipping one bit
0101, 0110,
1000 0100 1001, 1010 0111 1011 Here: No single bit error
can convert one legal
frame into another one!
1101

1100 1110 1111 Two bit changes

necessary to go from
one legal frame to
uvxy – legal frame abcd – illegal frame another

15
Distance between frames
In previous example: Two bit changes necessary to go
from one legal frame to another
Formally: Hamming distance
Let x=x1,…, xn and y=y1,…, yn be frames
d(x,y) = number of 1 bits in x XOR y (x⊕y )
Intuitively: the number of bit positions where x and y are different

Example: x=0011010111
y=0110100101
x ⊕ y=0101110010

d(x,y) = 5

16
 Hamming distance of a set of frames
The Hamming distance of a set of frames S:

The smallest distance between any two frames in the set

Examples: 0000 0011 3

001011 011101

1 4
101011
1100 1111
All distances are 2 One distance is 1!

17
Hamming distance and error detection/correction
What happens if d(S) = 0?
This does not make sense, by definition

What happens if d(S) = 1?

There exist x,y S such that d(x,y) = 1; no other pair is closer
1 bit difference
x y
A single bit error converts from one legal frame x to another legal
frame y
Cannot detect or correct anything

18
Hamming distance and detection/correction
What happens if d(S) = 2?
There exist x,y S such that d(x,y) = 2; no other pair is closer
In particular: any u with d(x,u) = 1 is illegal,
As is any u with d(y,u)=1

1 bit difference 1 bit difference

x u y

I.e., errors which modify a single bit always lead to an illegal frame
! Can be detected!
Generalizes to all legal frames, because Hamming distance
describes the “critical cases”
But not corrected – upon receiving u, no way to decide whether x
or y had been sent (symmetry!)

19
Hamming distance and detection/correction
What happens if d(S) = 3?
There exist x,y S such that d(x,y) = 3; no other pair is closer
Every s with d(x,s)=1 is illegal AND d(y,s) > 1!

1 bit difference 1 bit difference 1 bit difference

x s u y

Hence: the receipt of s could have the following causes:

Originally, x had been sent, but 1 bit error occurred
Originally, y had been sent, but 2 bit errors occurred
(Originally, some other frame had been sent, but at least 2 bit errors
occurred)
Assuming that fewer errors have happened, a received frame s
can be mapped to a frame x!
Hence, the error has been “corrected” – hopefully, correctly!

20
Generalization – Required Hamming distances
The examples above can be generalized

To detect d bit errors, a Hamming distance of d+1 in the

set of legal frames is required
So that it is not possible to re-write a legal frame into another one
using at most d bits

To correct d bit errors, a Hamming distance of 2d+1 in the

set of legal frames is required
So that all illegal frames at most d bits away from legal frame are
more than d bits away from any other legal frame

21
Frame sets – code books, codes
A terminology aspect:
The set of legal frames S {0,1}n is also called a code book or
simply a code

The rate R of a code S is defined as:

Rate characterizes the efficiency

The distance ofa code S is defined as:

Distance characterizes error correction/detection capabilities

A good code should have large distance and large rate –

but arbitrary combinations are not possible

22
Linear Block Code
The information bit stream is chopped into blocks of k bits
Each block is encoded to a larger block of n bits
Called codeword
The coded bits are modulated and sent over channel
The reverse procedure is done at the receiver

Channel
Data block Codeword
encoder
k bits n bits

n-k Redundant bits

k Information bits per codeword
Rc = Code rate symbol
n

23
Linear Block Code …
In linear block code (LBC), the extra n-k bits are linear
functions (combinations) of the original k bits
These extra bits are called parity-check bits

LBCs use a larger number of parity bits to either

Detect more than one error or
Correct one or more errors

24
Linear Block Codes …
Consider an (n, k) binary block code
There are 2n possible codewords
Select 2k codewords from the 2n possibilities to code
Each k bit information block is uniquely mapped to one of these 2k
codewords
Code rate Rc = k/n information bits per codewordsymbol

For 2k << 2n the distance between codewords can be

increased
Thus provides for better error correction and detection

25
Linear Block Codes …
Define a generator matrix G of the form

Encoding in (n, k) block code

G describes how codewords are generated from information bits

26
Linear Block Codes …
Example: Block code (6,3)

27
Systematic Block Codes
For a systematic block code (n, k), the first (or last) k
elements in the codeword are information bits

28
Systematic Block Codes
For any linear code we can find a matrix H(n k ) n which its
rows are orthogonal to rows of G
T
GH = 0
H checks the parity of the received word
i.e. maps the n-bit word to a m-bit syndrome
Codewords (=mG) should have parity of 0

H is called the parity check matrix and its rows are linearly
independent
For systematic linear block codes:

H = [In - k PT]

29
Systematic Block Codes
For the (6,3) block code example see earlier, the parity
check matrix is given as

Thus, multiplication of any valid codeword with the parity-

check matrix results in an all-zero vector
This property is used to determine whether the received
vector is a valid codeword or has been corrupted

30
Systematic Block Codes

Data source
Source m Channel U Modulation
coder encoding

channel

Source Channel Demodulation

Data sink
decoder decoding Detection
m̂ r
r = U+e
r = (r1, r2 ,....,rn ) received codeword or vector
e = (e1,e2 ,...., en ) error pattern or vector

Syndrome testing
S is syndrome of r, corresponding to the error pattern e
S = rHT = eHT

31
Syndrome Testing
If r is a valid codeword then S = 0
Thus, the syndrome equals the all-zero vector if
The transmitted codeword is not corrupted or
Is corrupted in a manner such that the received codeword is a valid
codeword in thxe code but is different from the transmitted
codeword
If the received codeword r contains detectable errors, then
S ≠0
If the received codeword contains correctable errors, the
syndrome identifies the error pattern corrupting the
transmitted codeword
These errors can then be corrected

32
Syndrome Testing
Note that the syndrome is a function only of the error
pattern e and not the transmitted codeword U

Because S = eHT corresponds to n−k equations in n

unknowns
Hence, not easier to solve mathematically
However, since the probability of bit error is typically small
and independent for each bit, the most likely error pattern
is the one with minimal weight
This corresponds to the least number of errors introduced
in the channel

33
Systematic Block Codes
For the (6,3) block code example above

There is a unique mapping from Syndrome ↔ Error Pattern

34
Generator Matrix
Example: The generator matrix for (7, 4) systematic code is

1. Compute the code generated for 1000, 0100 and 0010

2. Let 1101000 be transmitted and 1100000 be received.
Show how the error is detected.

35
Hamming codes
Developed by Richard Hamming in the late 1940’s
He recognized that the further evolution of computers
required greater reliability
In particular the ability detect errors and correct errors
His search for error-correcting codes led to the Hamming
Codes
Hamming Codes are still widely used in computing,
telecommunication, and other applications

36
 Hamming Codes

Class of error-correcting codes

Can detect single and double-bit errors
Can correct single-bit errors
For each m > 2, there is a Hamming code of length
n = 2m – 1 with n – k = m parity check bits

37
Hamming (7,4) Code
m = 3 Hamming Code
Information bits are b1, b2, b3, b4
Equations for parity checks b5, b6, b7

There are 24 = 16 codewords

(0,0,0,0,0,0,0) is a codeword

38
Hamming (7,4) Code …
Hamming code really refers to a specific (7,4) code
Hamming introduced in 1950
Hamming code adds 3 additional check bits to every 4
data bits of the message for a total of 7
Hamming's (7,4) code can correct any single-bit error, and
detect all two-bit errors
Since the medium would have to be uselessly noisy for 2
out of 7 bits (about 30%) to be lost, Hamming's (7,4) is
effectively lossless

39
Hamming (7,4) Code …
Information Codeword Weight
b1 b2 b3 b4 b1 b2 b3 b4 b5 b6 b7 w(b)
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 1 1 1 1 4
0 0 1 0 0 0 1 0 1 0 1 3
0 0 1 1 0 0 1 1 0 1 0 3
0 1 0 0 0 1 0 0 0 1 1 3
0 1 0 1 0 1 0 1 1 0 0 3
0 1 1 0 0 1 1 0 1 1 0 4
0 1 1 1 0 1 1 1 0 0 1 4
1 0 0 0 1 0 0 0 1 1 0 3
1 0 0 1 1 0 0 1 0 0 1 3
1 0 1 0 1 0 1 0 0 1 1 4
1 0 1 1 1 0 1 1 1 0 0 4
1 1 0 0 1 1 0 0 1 0 1 4
1 1 0 1 1 1 0 1 0 1 0 4
1 1 1 0 1 1 1 0 0 0 0 3
1 1 1 1 1 1 1 1 1 1 1 7

40
Parity Check Equations
Rearrange parity check equations:

In matrix form

All codewords must satisfy these equations

41
Error Detection with Hamming Code

0
0
1 0 1 1 1 00 1
1 Single error detected
s = H e= 1 1 0 1 0 10 0 = 0
0 1 1 1 0 01 0
1
0
0

0
1
1 0 1 1 1 00 0 0 1 1
s = H e= 1 1 0 1 0 10 0 = 1 + 0 = 1 Double error detected
0 1 1 1 0 01 1 1 0 1
0
0

42
Error Detection with Hamming Code

1
1
1 01 11 00 1 1 0 1
s=H e= 1 10 10 10 0 = 1 + 1 + 0 = 0 Triple error not
0 11 10 01 0 0 1 1 detected
0
0

43
Thank You