EE 121 Digital Communication Systems University of California, Berkeley: Spring 2005

Gastpar March 3, 2005

First Midterm Exam

Last name First name SID

• You have two hours to complete this exam.

• There are 100 points for this exam. Points for the individual problems and subproblems
are marked in the problem statement.
• The exam is closed-book and closed-notes; calculators, computing and communication
devices are not permitted.
• However, one handwritten and not photocopied single-sided sheet of notes is allowed.
• No form of collaboration between the students is allowed. If you are caught cheating, you
may fail the course and face disciplinary consequences.
• If we can’t read it, we can’t grade it.
• We can only give partial credit if you write out your derivations and reasoning in detail.
• You may use the back of the pages of the exam if you need more space.

*** Good Luck! ***

Problem Points earned out of

Problem 1 20

Problem 2 30

Problem 3 25

Problem 4 25

Total 100

1
Problem 1 (Short questions.) 20 Points
For each of the following statements, decide whether they are true of false. If you believe a
statement is true, give a proof. If you believe a statement is false, give a counterexample.
(a) (5 Pts) Consider a communication channel as in Figure 1 with input X , output Y , and
conditional probability mass function p(y|x) . Then it is true that H(Y ) ≥ H(X) .

X Y
p(y|x)

Figure 1: A communication channel.

(b) (5 Pts) D -ary Huffman codes always satisfy the Kraft inequality with equality ( D ≥ 2 ).

2
(c) (5 Pts) Let X and Y be two zero-mean jointly Gaussian random variables. Let Z =
X − E[XY
E[Y 2 ]
]
Y. Then, Z and Y are independent.

(d) (5 Pts) Suppose {ui (t)}N N

i=1 and {vj (t)}j=1 are two orthonormal bases for the same space
S . Then if
N
X N
X
s(t) = αi ui (t) = βj vj (t) (1)
i=1 i=1
PN 2
PN 2
we have i=1 |αi | = j=1 |βj | .

3
Problem 2 (Source Coding with Side Information.) 30 Points

X bits X
Encoder Decoder

Figure 2: Source coding with side information.

A discrete memoryless source produces, in each time slot, a vector (X, Y ) with joint distribution
pX,Y (x, y) as follows:
p(x, y) y=α y=β
x=a 1/6 1/6
x=b 1/12 1/6
x=c 1/24 1/6
x=d 1/24 1/6
As illustrated in Figure 2, Y is given both to the encoder and to the decoder, but X is only
observed by the encoder. Your task is to devise an algorithm to be used by the encoder. The
output of the encoder is a sequence of bits (i.e., of zeros and ones) such as to enable the decoder
to get to know X .
Remark. First, read all the subproblems (a)-(d). Then, start solving them.
(a) (10 Pts) Determine H(X|Y = α), H(Y |X = b), and H(X|Y ) .

4
(b) (8 Pts) Develop an efficient prefix-free source encoding/decoding algorithm that maps each
(x, y) pair independently into a uniquely decodable short sequence of bits. Carefully describe
each step taken by the encoder and the decoder.
p(x, y) y=α y=β
x=a 1/6 1/6
x=b 1/12 1/6
x=c 1/24 1/6
x=d 1/24 1/6

(c) (5 Pts) Suppose your encoder from Part (b) outputs the bit string: 0010100101001010001011101...
At the decoder, you observe the following sequence of Y values: β, β, α, β, α, α, β . Decode the
first 7 X symbols, using the coding scheme you devised in Part (b). Note: You may not need
to decode all the bits in the string.

(d) (7 Pts) Determine the average number of bits, L , that your encoder from Part (b) produces
for each source output symbol. Show your derivation. It is not sufficient to merely give a
number.

5
Problem 3 (Quantization.) 25 Points

quantization
X index (bits) X̂
Quantizer Reconstruction

Figure 3: Quantization.

Consider a memoryless source whose outputs X are uniformly distributed over the interval
[0, 2] . In this problem, you will design quantizers for this source. The overall system looks like
in Figure 3.
(a) (5 Pts) Find the best 1 -bit scalar quantizer (quantization cell boundary and reconstruction
points), determine the resulting mean-squared error distortion, and draw the corresponding
point into the rate vs. distortion plot in Figure 4.

(b) (5 Pts) Find the best 2 -bit scalar quantizer (quantization cell boundaries and reconstruction
points), determine the resulting mean-squared error distortion, and draw the corresponding point
into the rate vs. distortion plot in Figure 4.

6
 Distortion

Rate (Bits per source symbol)

Figure 4: Draw the solutions to Parts (a)-(c) here. Select an appropriate scale.

(c) (5 Pts) Find the best R -bit scalar quantizer (quantization cell boundaries and reconstruction
points), where R is a positive integer, and determine the resulting mean-squared error distortion.
Sketch the resulting behavior into the rate vs. distortion plot in Figure 4. Hint: Do all your
calculations for a scalar quantizer with M cells. What is the relationship between R and M ?
At the very end, use this relationship to plot the distortion vs. R.

7
 f Y(y)

3/7

1/7

y
1 2 3

Figure 5: The source distribution for Parts (d) and (e).

(d) (5 Pts) Consider the memoryless source whose outputs Y are distributed according to the
probability density function fY (y) illustrated in Figure 5. We want to use a two-cell quantizer
that assigns quantization indices as follows:

0, if 0 ≤ y ≤ α
Q(y) = (2)
1, if α < y ≤ 3,

where α is chosen such as to minimize the mean-squared error of the source reconstruction
based on the quantization index. Will the best α lie between 0 and 2 , or between 2 and 3 ?
Justify your answer with a mathematical argument. Hint: Start with the initial choice α0 = 2
and continue from there.

(e) (5 Pts) (Solve this problem after solving all other problems.) For the same setup as in Part
(d), determine explicitly the optimal value of α . Remark: Start by giving an outline of your
derivation of the optimal α , and carry out the calculations as far as you can.

8
 s1(t) s2(t) s3(t) s4(t)

b b b
T T/2 T
T/2 T t T/2 T t T/2 t t

−b −b −b

p
Figure 6: The waveforms for Problem 4. Here, b = 6/T .

Problem 4 (Signal Space.) 25 Points

The following set of four waveforms is to be used for transmission across a channel that adds
white Gaussian noise of power spectral density N0 /2 (that’s
p exactly the noise process that we
considered in class). Assume that N0 = 1 and that b = 6/T .
(a) (6 Pts) Draw an orthonormal basis for this set of waveforms. How many dimensions are
there? Briefly show that your basis functions are orthonormal. Hint: Gram-Schmidt may not
be the simplest solution.

(b) (6 Pts) Sketch the signal space characterization of this set of waveforms, including the signal
points s1 , s2 , s3 , s4 .

(c) (3 Pts) Determine the average energy of the waveforms.

9
(d) (10 Pts) Now suppose that the signals of Figure 6 are used for transmission across a channel
that adds white Gaussian noise of power spectral density N0 /2 (that’s exactly the noise process
that we considered in class). Denote the received signal vector by r .
For the special case when P rob(s1 ) = 1/3, P rob(s2 ) = 2/3, and P rob(s3 ) = P rob(s4 ) = 0,
determine the MAP decoding rule, using formulas.p Then, sketch the MAP decoding rule into
the figure you have drawn in Part (b). Use b = 6/T and N0 = 1 as before. You may make
the approximation ln 2 ≈ 0.7 .

10