Turbo Codes For PCS Applications: Abstract
Turbo Codes For PCS Applications: Abstract
ABSTRACT:Turbo codes are the most exciting and potentially important correspond to 9.6 Kbps and 13 Kbps with roughly 20ms frames). Higher
development in coding theory in many years. They were introduced in capacity can be obtained with larger interleavers. Note that low rate codes
1993 by Berrou, Glavieux and Thitimajshima [l], and claimed to achieve can be used for CDMA since an integer number of chips per coded symbol
near Shannon-limit error correction performance with relatively simple is used and bandwidth is defined mainly by chip rate.
component codes and large interleavers. A required Eb/N, of 0.7 dB was Three new contributions are reported in this paper: a new simple
reported for BER of and code rate of 1/2 [l]. However, some im- method for trellis termination, the use of unequal rate component codes
portant details that are necessary to reproduce these results were omitted. which results in better performance, and the development of decoders for
This paper confirms the accuracy of these claims, and presents a complete multiple-code encoders -the original turbo decoder scheme operates in
description of an encodeddecoder pair that could be suitable for PCS ap- serial mode, while for multiple-code encoders we found that the decoder
plications. We describe a new simple method for trellis termination, we for the whole turbo code based on the optimum MAP rule must operate
analyze the effect of interleaver choice on the weight distribution of the in parallel mode, and we derived the appropriate metric as illustrated in
code, and we introduce the use of unequal rate component codes which Sec. 111.
yields better performance. Turbo codes are extended to encoders with
11. PARALLEL
CONCATENATION
OF CONVOLUTIONAL
multiple codes and a suitable decoder structure is developed, which is
CODES
substantially different from the decoder for two-code based encoders.
The codes considered in this paper consist of the parallel concatenation
I. INTRODUCTION of multiple convolutional codes with random interleavers (permutations)
Coding theorists have traditionally attacked the problem of designing at the input of each encoder. Fig. 1 illustrates a particular example that
good codes by developing codes with a lot of structure which lends to will be used in this paper to verify the performance of these codes. The
feasible decoders, although coding theory suggests that codes chosen “at
random” should perform well if their block size is large enough. The
challenge to find practical decoders for “almost” random, large codes has
not been seriously considered until recently. Perhaps the most exciting
and potentially important development in coding theory in recent years
has been the dramatic announcement of “Turbo-codes” by Berrou et al.
in 1993 [l]. The announced performance of these codes was so good that
the initial reaction of the coding establishment was deep skepticism, but
recently researchers around the world have been able to reproduce those
results [3]- [4]. The introduction of turbo-codes has opened a whole new
way of looking at the problem of constructing good codes and decoding
them with low complexity.
These codes are claimed to achieve near Shannon-limit error correc-
tion performance with relatively simple component codes and large inter-
leavers. A required &,/No of 0.7 dB was reported for BER of [l].
However, some important details that are necessary to reproduce these
results were omitted. The purpose of this paper is to shed some light on
the accuracy of these claims, and to present a complete description of an
encodeddecoder pair that could be suitable for PCS applications, where
lower rate codes can be used.
For example, in multiple-access schemes like CDMA the capacity Figure 1: Example of encoder with three codes
(maximum number of users per cell) can be expressed as C = &+ 1,
where 17 is the processing gain and E,,/N, is the required signal-to-noise
encoder contains three recursive binary convolutional encoders, with M I ,
ratio to achieve a desired bit error rate (BER) performance. For a given
M2 and M3 memory cells respectively. In general, the three component
BER, a smaller required E b / N o implies a larger capacity or cell size.
encoders may not be identical. The first component encoder operates
Unfortunately, to reduce E h / No it is necessary to use very complex codes
directly on the information bit sequence U = (u1. . . . , U N ) of length N ,
(e.g. large constraint length convolutional codes). In this paper, we design
producing the two output sequences y l rand y l p .The second component
turbo codes suitable for CDMA and PCS applications that can achieve
encoder operates on a reordered sequence of information bits u2 produced
superior performance with limited complexity. For example, if a (7,1/2)
by an interleaver n2 of length N , and outputs the sequence y2,,.Similarly,
convolutional code is used at BER=10-3, the capacity is C = 0.5q.
subsequent component encoders operate on a reordered sequence of in-
However, if two (5,1/3) punctured convolutional codes or three (4,1/3)
formation bits u j produced by interleaver nj and output the sequence y j p .
punctured codes are used in a turbo encoder structure, the capacity can
The interleaver is a pseudo-random block scrambler defined by a permu-
be increased to C = 0.817 (with 192-bits and 256-bits interleavers which
tation of N elements with no repetitions: a complete block is read into
‘The research described in this paper was carried out at the Jet Propulsion the the interleaver and read out in a specified (fixed) random order. The
Laboratory, California Institute of Technology, under contract with the National same interleaver is used repeatedly for all subsequent blocks. Figure 1
Aeronautics and Space Administration. shows an example where a rate r = 1/ n = 1/4 code is generated by three
55
us consider a weight-3 data sequence such as (. . . 001 1100. . ,) which erating random integers i, 1 5 i 5 N , without replacement. We define
for our example corresponds to the minimum distance of the code (using a “S-random” permutation as follows: each randomly selected integer
no permutations). The probability that this sequence is reproduced with is compared to S previously selected integers. If the current selection
one random interleaver is roughly 6 / N 2 , and the probability that some is equal to any S previous selections within a distance of fS,then the
sequence of the form (. . ,0011100. . .) is paired with another of the same current selection is rejected. This process is repeated until all N integers
form is 1 - (1 - 6 / N 2 ) Nx 6 / N . Thus for large block sizes, the bad are selected. While the searching time increases with S, we observed
weight-3 data sequences have a small probability of being matched with that choosing S < usually produces a solution in reasonable time.
bad weight-3 permuted data sequences, even in a two-code system. (For S = 1 we have a purely random interleaver).
For a turbo code using q codes and q - 1 random interleavers this In the simulations we used S = 11 for N = 256 and S = 31 for
probability is even smaller, 1 - [l - (6/N2)9- ‘1” :(+)q-*. This N = 4096.
The advantage of using three or more constituent codes is that the
implies that the minimum distance codeword of the turbo code in Fig. 1
is more likely to result from a weight-2 data sequence of the form corresponding two or more interleavers have a better chance to break se-
(, . .001001000 . . .) than from the weight-3 sequence (. . ,0011100. . .) quences that were not taken care by another interleaver. The disadvantage
that produces the minimum distance in the unpermuted version of the is that, for an overall desired code rate, each code must be punctured more,
same code. Higher weight sequences have even smaller probability of resulting in weaker constituent codes. In our experiments, we have used
reproducing themselves after being passed through a random interleaver. randomly selected interleavers and interleavers based on the row-column
permutation described above. In general, randomly selected permuta-
For a turbo code using q codes and q - 1 interleavers, the probability
that a weight-n data sequence will be reproduced somewhere within the r o o . . . 0 0 0 1
block by all q - 1 permutations is of the form 1 - [ 1 - (/?/N“-’)q-‘]N,
0
WRITE
where fi is a number that depends on the weight-n data sequence but does +
not increase with block size N . For large N , this probability is propor-
tional to (l/N)“q-“-q, which falls off rapidly with N , when n and q are 0 0
L o o 0 . . . 0 0 1
greater than two. Furthermore, the symmetry of this expression indicates
that increasing either the weight of the data sequence n or the number of
codes q has roughly the same effect on lowering this probability. Figure 3: Example where a block interleaver fails to “break” the
In summary, from the above arguments we conclude that weight-2 data input sequence.
sequences are an important factor in the design of the component codes,
and that higher weight have decreasing importance. Also, increasing the tions are good for low SNR operation (e.g., PCS applications requiring
number of codes may result in better turbo codes. More accurate results f$ = where the overall weight distribution of the code is more
and derivations are discussed in [6]. important than the minimum distance
The minimum distance is not the most important quantity of the turbo 111. TURBO
DECODINGCONFIGURATION
code, except for its asymptotic performance, at very high E!,/N,. At mod- The turbo decoding configuration proposed in [1] for two codes is shown
erate SNRs, the weight distribution for the first several possible weights schematically in Fig. 4. This configuration operates in serial mode, i.e.
is necessary to compute the code performance. Estimating the complete “Dec1” processes data before “Dec2” starts its operation, and so on.
weight distribution of these codes for large N and fixed interleavers is still An obvious extension of this configuration to three codes is shown in
an open problem. However, it is possible to estimate the weight distribu-
tion for large N for random interleavers by using probabilistic arguments.
(See [4] for further considerations on the weight distribution).
I n t e r l e a v e r Design - Interleavers should be capable of spread-
ing low-weight input sequences so that the resulting codeword has high
Figure 4: Decoding structure for two codes.
weight. Block interleavers, defined by a matrix with U, rows and U,
columns such that N = ur x U,, may fail to spread certain sequences. For
Fig. 5(a), which also operates in serial mode. But, with more than two
example, the weight 4 sequence shown in Fig. 3 cannot be broken by a
codes, there are other possible configurations, as that shown in Fig. 5(b)
block interleaver. In order to break such sequences random interleavers
where “Decl” communicates with the other decoders, but these decoders
are desirable. (A method for the design of interleavers is discussed in [ 3 ] ) .
do not exchange information among each other. This “Master & Slave’’
Block interleavers are effective if the low-weight sequence is confined to
configuration operates in a mixed serial-parallel mode, since all other
a row. If low-weight sequences (which can be regarded as the combina-
decoders except the first operate in parallel. Another possibility, shown in
tion of lower weight sequences) are confined to several consecutive rows,
then the U, columns of the interleaver should be sent in a specified order Fig. 5(c) is that all decoders operate in parallel at any given time. Note that
self loops are not allowed in these structures since they cause degradation
to spread as much as possible the low-weight sequence. A method for re-
or divergence in the decoding process (positive feedback). We are not
ordering the columns is given in [8]. This method guarantees that for any
considering other possible hybrid configurations. Which configuration
number of columns U, = aq + r , ( r 5 a - 1), the minimum separation
performs better? Our selection of the best configuration and its associated
between data entries is q - 1, where a is the number of columns affected
decoding rule is based on a detailed analysis of the minimum bit error
by a burst. However, as can be observed in the example in Fig. 3, the se-
decoding rule (MAP algorithm) as described below.
quence 1001 will still appear at the input of the encoders for any possible
Turbo D e c o d i n g for M u l t i p l e Codes - Let uk be a binary
column permutation. Only if we permute the rows of the interleaver in ad-
random variable taking values in (0, l}, representing the sequence of
dition to its columns it is possible to break the low-weight sequences. The
information bits U = ( U , , . . . , u N ) ,The MAP algorithm [7] provides the
method in [SI can be used again for the permutation of rows. Appropriate
log likelihood ratio Lk given the received symbols y:
selection of of a , and q for rows and columns depends on the particular
set of codes used and on the specific low-weight sequences that we would
like to break. We designed random permutations (interleavers) by gen-
56
Note that P(ulyi)is not separable in general A reasonable criterion for
(a) SERIAL
this approximation is to choose n,”=,
pi( u k ) such that it minimizes the
Kullback distance or free energy [9, lo]. Define
eukilk
pi (uk) = -
e.. +
1 eick
9
c U , ~ = ‘(Ylu) = 1)
= log
l n,fk ‘(‘1)
+ log p(uk (2) We can use (4) and (5) again, but this time for i = 1 , 3 , to express (3) as
c U , u k = O‘(ylu) n,#k ‘(’1) p(uk = 0)
are non-uniform, the modified MAP algorithm in [2] is simpler to use and similarly
than the version considered in [l]. Therefore, in this paper we use the Lk = f ( Y 3 , Ll, L2, k) f i l k fL 2 k (10)
modified MAP algorithm of [2] as we did in [4]. A solution to eqs. 7,9, and 10 is
The channel model is shown in Fig. 6 where the nz,k’s and the n p k ’ S
are i.i.d. zero mean Gaussian random variables with unit variance, and i l k f ( y l , E 2 1 L3, k ) ; L2k = f(Y2, fJ1, k); k)
L3k = f(Y3, fJI, 6 2 ,
p = ,/- is the signal-lo-noise ratio. (The same model is used (11)
for each encoder). To explain the basic decoding concept we restrict provided that a solution does indeed exist. fork = 1 , 2 , , . , , N.The final
decision is then based on
Lk = i l k + +
i2k i,k (12)
where, for each iteration, a?’ and a?’ can be optimized (simulated an-
nealing) or set to 1 for simplicity. The decoding configuration for two
codes, according to the previous section, is shown in Fig. 8. In this spe-
X X
cia1 case, since the two paths in Fig. 8 are disjoint, the decoder structure
reduces to that of Fig. 4, i.e. to the serial mode.
If we optimize a y ) and a?), our method for two codes is similar to -1.5 -1.0 -0.5 0.0 0.5 1.o
the decoding method proposed in [l], which requires estimates of the EdN,, dB
variances of i l k and i2k for each iteration in presence of errors. In the
method proposed in [2] the received “systematic” observation was sub- Figure 9: Turbo codes performance, r = 1/4
tracted from i l kwhich
, results in performance degradation. In [3] the
method proposed in [2] was used hut the received “systematic” observa- Unequal Rate Encoders -We now extend the results to encoders
tion was interleaved and provided to decoder 2. In [4],we argued that with unequal rates with two K = 5 constituent codes (1, gb/ga, g,/ga)
there is no need to interleave the received “systematic” Observation and
and (gb/ga), where ga = (Wocrar, 8 6 = (33)octar and gc = (25)octar.
provide it to decoder 2, since E l k does this job. It seems that our proposed This structure improves the performance of the overall, rate 1/4, code, as
method with ay’ and a?) equal to 1 is the simplest and achieves the same
shown in Fig.9. This improvement is due to the fact that we can avoid
performance reported in [3] for rate 1/2 codes.
using the interleaved information data at the second encoder and that the
Terminated Parallel Convolutional Codes as Block
rate of the first code is lower than that of the second code. For PCS ap-
Codes. - Consider the combination of permuter and encoder as a
plications, short interleavers should be used, since the vocoder frame is
linear block code. Define P, as the parity matrix of the terminated convo-
usually 20ms. Therefore we selected 192 and 256 bits interleavers as an
lutional code i . Then the overall generator matrix for three parallel codes
example, corresponding to 9.6 and 13 Kbps. (Note that this small dif-
is ference of interleaver size does not affect significantly the performance).
G = [ I Pi ~ 2 P 2n3P3] The performance of codes with short interleaver is shown in Fig. 10 for the
where n,are the permutations (interleavers). In order to maximize the K = S codes described above for random permutation and row-column
minimum distance of the code given by G we should maximize the number permutation with a = 2 for rows and a = 4 for columns.
of linearly independent columns of the corresponding parity check matrix Three Codes - The performance of a three-code turbo code with
H . This suggests that the design of P, (code) and n,(permutation) are random interleavers is shown in Fig. 1 1 for N = 4096. The three recursive
closely related and it does not necessarily follow,that optimum component codes shown in Fig. 1 where used for K = 3. Three recursive codes with
codes (maximum d,,,) yield optimum parallel concatenated codes. For g, = (13)ocror and gb = (ll)octalwere used for K = 4. Note that the
very small N we used this concept to design jointly the permuter and the non-systematic version of this encoder is catastrophic, hut the recursive
component convolutional codes. systematic version is non-catastrophic. We found that this K = 4 code
IV. PERFORMANCE has better performance than several others.
Two Codes - The performance obtained by turbo decoding the Although it was suggested [SI that g, be a primitive polynomial, we
code with two constituent codes (1, go/g,), where g, = (37),,r,l and found several counterexamples that show better performance, e.g. g, for
g b = (21),,ral, and with random permutations of lengths N = 4096 and
K = 5 proposed in [l] is not primitive.
In Fig. 11 , the performance of the K = 4 code was improved by
2Note that the components of the E’s corresponding to the tail bits are set to going to 30 iterations and using a S-random interleaver with S = 3 1 . For
zero for all iterations. shorter blocks (192 and 256), the results are shown in Fig. 10 where it
58
can be observed that approximately 1 dB SNR is required for BER=10-3,
which implies a CDMA capacity C = 0 . 8 ~We have noticed that the Random Interleaving N=4096
Code Rate=1/4
slope of the BER curve changes around BER=10-5 (flattening effect) if 1 Y
the interleaver is not designed properly to maximize d,,, or is chosen at
random. Galileo Code
r=l I4 10-
m=20
K
W
m N=l9:
1
Three &State Codes
t \
Figure 10: Performance with short block sizes. [4] D. Divsalar and E Pollara, “Turbo Codes for Deep-Space Communications”,
JPL TDA Progress Report 42-120, Feb. 15, 1995.
V. CONCLUSIONS [5] G. Battail, C. Berrou and A. Glavieux, “Pseudo-random recursive con-
We have shown how turbo codes and decoders can be used to improve volutional coding for near-capacity performance”, Comm. Theory Mini-
the coding gain for PCS applications. These are just preliminary results conference, GLOBECOM ’93, Dec. 1993.
that require extensive further analysis. In particular, we need to improve [6] D. Divsalar, S. Dolinar and E Pollara, “Weight distribution of multiple turbo
our understanding of the influence of the interleaver choice on the code codes”, JPL TDA Progress Report, (In preparation).
performance, to explore the sensitivity of the decoder performance to the [7] L. R. Bahl, J. Cocke, E Jelinek, and J. Raviv, “Optimal Decoding of Lin-
precision with which we can estimate Ebf No. ear Codes for Minimizing Symbol Error Rate,” IEEE Trans. Znfurm. Theory,
An linteresting theoretical qwstion is to determine “how random” these vol. IT-20 (1974), pp. 284-287.
codes can be so as to draw conclusions on their performance based on [8] E. Dunscombe and EC. Piper, “ Optimal interleaving scheme for convolutional
comparison with random coding bounds. In [4] we obtained the complete codes”, Electronic Letters, 26 Oct. 1989, Vol. 25, No. 22, pp. 1517-1518.
weight distribution of a turbo code, calculated the upper bound on BER [9] M. Moher, “Decoding via Cross-entropy Minimization”, Proceedings
and compared it with maximum-.likelihood(ML) decoding. Those results GLOBECOM ’93, Dec. 1993, p.809-813.
showed that the performance of turbo decoding is close to ML decoding [lo] G. Battail and R. Sfez, “Suboptimum Decoding using the Kullback Princi-
and to optimum MAP decoding. However, the approximation used in ple”, Lecture Notes in Computer Science, Vol. 313, pp. 93-101, 1988.
eq.(4) implies that turbo decoding is only close to but not equal to MAP
decoding.
VI. ACKNOWLEDGMENTS
The authors are grateful to S. Dolinar and R.J. McEliece for their helpful
comments.
RE~FERENCES
[1] C. Berrou, A. Glavieux, and P. Thitimajshima, “Near Shannon Limit Error-
Correcting Coding: Turbo Codes.” Proc. 1993 IEEE International Conference
on Communications, pp. 1064--1070.
59