ECE 154C Homework Assignment #3

(Due: Friday, May 31, 2019)

Reading assignment: Jack Wolf lecture notes pp. 69–151.

Recommended: Solve HW# 4, 5 from spring 2017 for practice (appended at the end)

Programming assignment: You can use any programming language you prefer (MATLAB,
Python, C/C++, or Julia, for example). Write down your code as clearly as possible and
add suitable comments. You should write your program from scratch. However, it is okay to
use some common functions such as sort or random generators in your code. Submit your
code and the output plots to AFAZELIC@UCSD.EDU or turn in the hard copies during the office
hours on Friday afternoon.

1. In this homework, we examine the performance of (7,4) Hamming code compared to

other (7,4) codes over BSC(p), the binary symmetric channel with bit error probability
p. Throughout this problem, we consider the systematic generator matrix G of the
following form: G = [Ik |A] where Ik is the k × k identity matrix and A is a k × (n − k)
matrix. For example, for the (7,4) Hamming code, we consider
 
1 0 0 0 1 1 0
0 1 0 0 1 0 1
G= 0 0 1 0 0 1 1 .


0 0 0 1 1 1 1

In this case, recall that the parity check matrix H is obtained easily by H = [AT |In−k ].

(a) Recall that the minimum distance decoding is maximum likelihood decoding for
BSC(p), where p < 1/2. Write a minimum distance decoder function
mindistdec(y,C). This function takes a received bits y and a codebook C as
inputs. (To implement this, you may need to write a function for generating the
codebook for a given generator matrix G.)
(b) Write a syndrome decoder function syndromedec(y,H), which takes a received
bits y and a parity check matrix H as inputs.
(c) For the (7, 4) Hamming code, verify that any one bit error in code bits can be
corrected by both mindistdec and syndromedec. Hence, for simplicity, we now
use mindistdec as our decoder for the rest of this problem.
(d) (NOT a programming) For the (7, 4) Hamming code, find the formula of the
probability of decoding to a wrong codeword when the code is used for BSC(p).
(e) We want to run a simulation that verifies the result found in Problem (c). For
p = 0.05 : 0.05 : 0.5, do the following steps 1,000 times to get a good estimate of
the decoded word error rate (WER).

1
 i. Generate 4 random binary digits.
ii. Encode them by calculating the 3 check digits, or using generator matrix G.
iii. Put the codeword through BSC(p).
iv. Decode the received 7-bit vector to a 4-bit word.
v. Compare the decoded word with the transmitted word and count word errors
to find word error rate (WER). Also count the errors in the information
positions of the code words to obtain an estimate of the decoded bit error
probability (BER).
Plot your simulation results p vs. BER(p) and p vs. WER(p), along with the
analytical expression found in part (d). Is the bit error probability larger or
smaller than the word error probability?
(f) (10% bonus) Now we consider another systematic generator matrix
 
1 0 0 0 1 0 0
0 1 0 0 0 1 0
G= 0 0 1 0 0 0
.
1
0 0 0 1 1 1 1

Repeat the same experiment as in (e), and compare the results with BER and
WER of the (7,4) Hamming code.
(g) (10% bonus) Finally we examine the performance of a randomly generated (7, 4)
systematic generator matrices. Generate a random 4 × 3 binary matrix A and use
G = [I4 , A] as our generator matrix. For each G, repeat the same experiment in
(e), and take averages of BERs and WERs over the random matrices. Compare
the results with BER and WER of the (7,4) Hamming code. Is there any p where a
randomly generated G performs better than the Hamming (7,4) code on average?

(Note: you do not have to compute the theoretical expression of WER for parts (f)
and (g).)

2
UCSD ECE154C Handout #14
Prof. Young-Han Kim Thursday, May 11, 2017

Homework Set #4
(Due: Thursday, May 18, 2017)

1. Suppose that a pair of dice are thrown, each independently taking on the values
{1, 2, 3, 4, 5, 6} with equal probabilities. Let X denote the output of the first die and
Y denote the output of the second die. Let Z = X + Y and Q = Z mod 2. Compute
the following entropies.

(a) H(X), H(Y ), H(Z).

(b) H(X, Z), H(Z|X), H(X|Z).
(c) H(Q, X), H(Q|X), H(X|Q).
(d) H(Q, Z), H(Q|Z), H(Z|Q).
(e) H(X, Y, Z), H(Z|X, Y ), H(X, Y |Z).
(f) H(X, Y, Z, Q), H(X, Y, Z|Q), H(Q|X, Y, Z).

2. Consider the cascade of two binary channels shown below.

1−a 1−c
1 1 1 1
a c
X1 Y 1 = X2 Y2
b d
0 0 0 0
1−b 1−d
(a) Draw a single channel with binary input and binary output that is equivalent to
this cascade.
(b) Repeat part (a) when the order of the two channels is reversed.
(c) Show that a cascade of two binary symmetric channels is also a binary symmetric
channel.

1
UCSD ECE154C Handout #15
Prof. Young-Han Kim Thursday, May 18, 2017

Solutions to Exercise Set #4

1. Suppose that a pair of dice are thrown, each independently taking on the values
{1, 2, 3, 4, 5, 6} with equal probabilities. Let X denote the output of the first die and
Y denote the output of the second die. Let Z = X + Y and Q = Z mod 2. Compute
the following entropies.

(a) H(X), H(Y ), H(Z).

(b) H(X, Z), H(Z|X), H(X|Z).
(c) H(Q, X), H(Q|X), H(X|Q).
(d) H(Q, Z), H(Q|Z), H(Z|Q).
(e) H(X, Y, Z), H(Z|X, Y ), H(X, Y |Z).
(f) H(X, Y, Z, Q), H(X, Y, Z|Q), H(Q|X, Y, Z).

Solution:

(a) We have H(X) = H(Y ) = log 6 = 2.585. By considering all the combinations of
X and Y , we have
 
6 1 2 3 4 5 36
H(Z) = log 6+2 log 36+ log 18+ log 12+ log 9+ log = 3.27.
36 36 36 36 36 36 5

(b) We have

H(X|Z) = H(X, Z) − H(Z)

= H(Z|X) + H(X) − H(Z)
= H(X + Y |X) + H(X) − H(Z)
= H(Y ) + H(X) − H(Z)
= 1.9.

Hence
H(X, Z) = H(X|Z) + H(Z) = 5.17
and
H(Z|X) = H(X, Z) − H(X) = 2.585.

1
 (c) Since given X is even, the binary random variable Q = 0 if Y is even and Q = 1
if Y is odd. Similarly for the case when X is odd. Thus H(Q|X) = 1. Note that
since H(Q) = 1, X and Q are independent of each other. Thus

H(Q, X) = H(Q|X) + H(X) = 3.585

and
H(X|Q) = H(Q, X) − H(Q) = 2.585.
(d) Since Q is a function of Z, we have H(Q|Z) = 0. Thus

H(Z, Q) = H(Q|Z) + H(Z) = H(Z) = 3.274

and
H(Z|Q) = H(Z, Q) − H(Q) = 2.274.
(e) Since Z = X + Y , we have H(Z|X, Y ) = 0 and H(Y |X, Z) = 0. Note that
H(X, Y ) = H(X) + H(Y ) since X and Y are independent, hence

H(X, Y, Z) = H(Z|X, Y ) + H(X, Y ) = H(X, Y ) = H(X) + H(Y ) = 5.17

and
H(X, Y |Z) = H(Y |X, Z) + H(X|Z) = H(X|Z) = 1.896.
(f) Since H(Q|X, Y, Z) = 0, we have

H(X, Y, Z, Q) = H(Q|X, Y, Z) + H(X, Y, Z) = H(X, Y, Z) = 5.17.

Thus
H(X, Y, Z|Q) = H(X, Y, Z, Q) − H(Q) = 5.17 − 1 = 4.17.

2. Consider the cascade of two binary channels shown below.

1−a 1−c
1 1 1 1
a c
X1 Y 1 = X2 Y2
b d
0 0 0 0
1−b 1−d
(a) Draw a single channel with binary input and binary output that is equivalent to
this cascade.
(b) Repeat part (a) when the order of the two channels is reversed.
(c) Show that a cascade of two binary symmetric channels is also a binary symmetric
channel.

2
Solution:

(a) The cascaded channel is the following

1 − e = p(Y2 = 1|X1 = 1) = (1 − a)(1 − c) + ad,

e = p(Y2 = 0|X1 = 1) = (1 − a)c + a(1 − d),
f = p(Y2 = 1|X1 = 0) = (1 − b)d + b(1 − c),
1 − f = p(Y2 = 0|X1 = 0) = (1 − b)(1 − d) + bc.
1−e
1 1
e
X1 Y2
f
0 0
1−f
(b) The cascaded channel is the following

1 − e′ = p(Y1 = 1|X2 = 1) = (1 − c)(1 − a) + cb,

e′ = p(Y1 = 0|X2 = 1) = c(1 − b) + (1 − c)a,
f′ = p(Y1 = 1|X2 = 0) = d(1 − a) + (1 − d)b,
1 − f′ = p(Y1 = 0|X2 = 0) = (1 − d)(1 − b) + da.
1 − e′
1 1
e′
X2 Y1
f′
0 0
1 − f′
(c) Let a = b and c = d in part (a), it is easy to see that the cascaded channel is
a binary symmetric channel. Similarly, by letting a = b and c = d in part (b),
we get a binary symmetric channel as well. Thus, the order of cascade doesn’t
matter.

3
UCSD ECE154C Handout #16
Prof. Young-Han Kim Thursday, May 18, 2017

Homework Set #5
(Due: Tuesday, May 30, 2017)

1. Suppose the Hamming code is used to detect errors.

(a) For i = 0, 1, 2, . . . , 7, find the number of error patterns containing exactly i errors
that the decoder will fail to detect?
(b) If the code is used for a binary symmetric channel with bit error probability p,
find an analytic expression for the probability of an undetected word error as a
function of p.

2. Suppose the Hamming code is used to correct erasures.

(a) For i = 0, 1, 2, . . . , 7, find the number of erasure patterns containing exactly i
erasures that the decoder will fail to correct?
(b) If the code is used for a binary erasure channel with bit erasure probability p, find
an analytic expression for the probability of failing to correct the erasures.

3. For the (23, 12) Golay code given in the notes, calculate the probability of decoding to
the wrong codeword if the code is used to correct errors caused by a binary symmetric
channel with bit error probability p.

4. Consider the (15, 7) parity check code with parity check matrix given below:
 
1 0 0 0 1 0 0 1 1 0 1 0 1 1 1
 0 1 0 0 1 1 0 1 0 1 1 1 1 0 0 
 
 0 0 1 0 0 1 1 0 1 0 1 1 1 1 0 
 
 0 0 0 1 0 0 1 1 0 1 0 1 1 1 1 
H =  1 0 0 0 1

 1 0 0 0 1 1 0 0 0 1  
 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 
 
 0 0 1 0 1 0 0 1 0 1 0 0 1 0 1 
0 1 1 1 1 0 1 1 1 1 0 1 1 1 1
(a) Without changing the order of the digits put the parity check matrix in a form
with a 8-by-8 identity matrix on the right.
(b) Give the generator matrix of the code with a 7 by 7 unit matrix on the left.
(c) What is the minimum Hamming distance of the code?
(d) Show the form of an encoder using a table look-up. What is the size of the table?
(e) Show a decoder for the code using two tables. What are the size of the tables?

1
UCSD ECE154C Handout #17
Prof. Young-Han Kim Tuesday, May 30, 2017

Solutions to Exercise Set #5

1. Suppose the Hamming code is used to detect errors.

(a) For i = 0, 1, 2, . . . , 7, find the number of error patterns containing exactly i errors
that the decoder will fail to detect?
(b) If the code is used for a binary symmetric channel with bit error probability p,
find an analytic expression for the probability of an undetected word error as a
function of p.
Solution: Let Ē be an error pattern. The decoder will not be able to detect error
patterns that match non-zero codewords, since
H(Ē + C̄) = 0̄
for any codeword C̄. By inspecting the list of codewords for the Hamming (7, 4) code,
we find 7 codewords with WH = 3, 7 codewords with WH = 4, and 1 codeword with
WH = 7. Hence
P(undetected error) = 7p3 (1 − p)4 + 7p4 (1 − p)3 + p7 .

2. Suppose the Hamming code is used to correct erasures.

(a) For i = 0, 1, 2, . . . , 7, find the number of erasure patterns containing exactly i
erasures that the decoder will fail to correct?
(b) If the code is used for a binary erasure channel with bit erasure probability p, find
an analytic expression for the probability of failing to correct the erasures.
Solution: Since the (7, 4) Hamming code has dmin = 3, it can correct all erasure
patterns with up to 2 erasures. The decoder cannot fill in any pattern with 4 or more
erasures since H only has 3 rows. Hence we cannot find 4 or more linearly independent
columns in H. However, the decoder can correct some patterns with 3 erasures, but
not all of them. We are able to fill in patterns that occur in positions that correspond
to linearly independent columns in H. If we select 3 columns from H, there are
7


2
3

=7
2
3


combinations that are linearly dependent. The reason we divide by 2
is to remove
redundant sets of linearly dependent columns.

1
 For 4, 5, 6, and 7 erasures, there are 74 = 35, 75 = 21, 76 = 7, and 77 = 1 patterns

respectively. Hence the analytic expression for the probability of uncorrectable erasures
is
X 7
P(uncorrectable erasures) = ei pi (1 − p)7−i
i=3

where ei is the number of patterns with i erasures that cannot be corrected calculated
above, i.e., e3 = 7, e4 = 35, e5 = 21, e6 = 7, e7 = 1.

3. For the (23, 12) Golay code given in the notes, calculate the probability of decoding to
the wrong codeword if the code is used to correct errors caused by a binary symmetric
channel with bit error probability p.

Solution: From the lecture, we know that the Hamming (23, 12) Golay code has
dmin = 7. Hence it can correct all error patterns of ⌊ dmin2 −1 ⌋ errors or less. The
probability of decoding to a correct codeword Pc can be given as
3  
X 23
Pc = pi (1 − p)23−i
i=0
i

and the probability of decoding to a wrong codeword Pw is

Pw = 1 − Pc .

4. Consider the (15, 7) parity check code with parity check matrix given below:
 
1 0 0 0 1 0 0 1 1 0 1 0 1 1 1
 0 1 0 0 1 1 0 1 0 1 1 1 1 0 0 
 
 0 0 1 0 0 1 1 0 1 0 1 1 1 1 0 
 
 0 0 0 1 0 0 1 1 0 1 0 1 1 1 1 
H =  1 0 0 0 1

 1 0 0 0 1 1 0 0 0 1  
 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 
 
 0 0 1 0 1 0 0 1 0 1 0 0 1 0 1 
0 1 1 1 1 0 1 1 1 1 0 1 1 1 1

(a) Without changing the order of the digits put the parity check matrix in a form
with a 8-by-8 identity matrix on the right.
(b) Give the generator matrix of the code with a 7 by 7 unit matrix on the left.
(c) What is the minimum Hamming distance of the code?
(d) Show the form of an encoder using a table look-up. What is the size of the table?
(e) Show a decoder for the code using two tables. What are the size of the tables?

2
Solution: Let H = [A|B] where A is a 8 × 7 matrix and B is a 8 × 8 matrix. Then

H ′ = B −1 [A|B] = [B −1 A|I8×8 ],

where B −1 is the modulo 2 inverse of B. Using Gaussian elimination (or Matlab), we

can find that B −1 A is
1 1 0 1 0 0 0
0 1 1 0 1 0 0
0 0 1 1 0 1 0
0 0 0 1 1 0 1
1 1 0 1 1 1 0
0 1 1 0 1 1 1
1 1 1 0 0 1 1
1 0 1 0 0 0 1.
Then the generator matrix is G = [I7×7 |(B −1 A)T ]. By observing the codeword with
the minimum Hamming weight, we can find that dmin = 5. If a table look-up was
used for encoding, the size of the table would be 27 × 8 where each row corresponds
to each message sequence of length 7 and the each column corresponds to each bit of
the parity sequence of length 8. The size of a decoding table would be 28 × 15 where
each row corresponds to each syndrome of length 8 and the columns correspond to the
error patterns ē. We would decode a received codeword r̄ by adding it to ē.