Illuminating the Structure of Code and Decoder of Parallel Concatenated Recursive Systematic (Turbo) Codes

Patrick Robertson * Institute for Communications Technology German Aerospace Research Establishment (DLR) P.O. Box 1116, D-82230 Wessling, Germany. Tel.: ++49 8153 282808; FAX: ++49 8153 281442 e-mail: pat rick0 holmes. nt .op.dlr.de

Abstract

We will try t o shed some light on the basic understanding of the decoder of the recently proposed 'turbo-codes' - a Recently, a coding scheme (turbo-codes) was proposed, that parallel concatenation of recursive systematic punctured conachieves almost reliable data communication at signal-to-noise ra- volutional codes with random interleaving between the entios very close to the Shannon-limit. In this paper we show that coders, and a n associated iterative decoding scheme [3]. The the associated iterative decoder can be formulated in a simpler proposeld scheme is claimed, under certain circumstances, to fashion by passing information from one decoder to the next us- achieve near Shannon-limit error correction capabilities [3]. ing log-likelihood ratios as opposed to channel values that need to However the interface between the output of one decoder, be normalized. Also no heuristically determined correction pa- and the input of the next decoder in the iterative decodrameters are necessary for stable decoding. In addition, we can ing scheme was not described sufficiently. Furthermore, the reduce the average number of iterations needed for the same BER non-trivial problem of trellis termination -important for small performance by determining when further iterations achieve no block sizes- had not been explicitly mentioned nor a solution more benefit. Furthermore, it seems that the trellis-termination given; we shall see t h a t this causes no severe problems even problem appears non-trivial and we give a pragmatic suboptimal for small1 block sizes. solution. We investigate different block sizes and also a hybrid We will not repeat the derivation of the modified Bahl et scheme that performs extremely well with less computations. A al. algorithm (MAP) [4][3], b u t will only briefly state the redrawback of the codes has been discovered: the BER curves show sults, in particular the modification to be made to the branch a flattening at higher signal-to-noise ratios, this is due to the small transition probability in order t o allow one decoder t o make minimum distance of the whole code. By analyzing the interleaver use of the output of the other decoder. By optionally estimatused in the encoder we can calculate approximations to the BER ing the variance of the decoder output we are able to reduce at high SNRs. Finally, by careful interleaver manipulation the the average number of iterations needed for the same BER minimum distance of the code can be increased and the error- performance by determining when further iterations achieve coefficient for the remaining small distance events can be further no more benefit. Furthermore, we will look at simulation rereduced. Furthermore, we have investigated the influence of the sults for a range of block sizes. To conclude, we shall present interleaver length on the SNR needed to achieve a certain BER. a method for estimating a bound on the BER at high SNR Simulations confirm both the analytical approximation to the BER a n d use it as a n optimization criterion for interleaver design. as well as the method for interleaver design which yields a marked improvement at higher SNR.

The Encoder and Channel Model

The encoder contains of two recursive systematic binary convolutional encoders (henceforth denoted as component enMultidimensional array codes and the associated decoding coders) [3]. In the following let the component encoders be techniques have received attention recently, as they are identical (extensions t o different component codes are trivpromising candidates for communications over very noisy ial), they have A memory elements. The first encoder op4 channels. To mention are product codes constructed with erates directly-on the d a t a t o be transmitted, let this be simple parity check codes [l],Hamming codes or convolu- the sequence d = ( d l , ..., d N ) of N bits. The first compotional codes [2]. nent encoder has two outputs, one is ;he sequence of information bits 2 = (2i5, ' ' ..., = d , since the encoder 'This work was partly carried out at the Communications Research is systematic. T h e other is the 'parity information' seCenter, Ottawa, Canada during an exchange program financed in part by GKSS/IB project KAN INF 20. : g , f f a i P i ,where quence &' = ( x k p , ..., x$), with x" =

I Introduction

xG)

0-7803-1820-x/94 0 1994 IEEE $4.00

1298

(sif,...,g z ) is the feed-forward generator of the first encoder 3

and U: = dk gilbakWi;similarly, ( g i b , ..., g z ) is the feed-back generator of the first encoder. The d a t a input sequence t o the second component encoder is now interleaved. Let it be represented- by the sequence d = (d:, ...,d;), with d i = d r k , where I = (11, ..., IN) 12, specifies the interleaver that reorders the information bits feeding the second component encode_r. Similarly, the associated de-interleaver is specified by I and dk = d i , k . The systematic sequence 2 is not transmitted, only the parity information sequence Z z P l generated in a n identical fashion to the first component encoder. Both encoders are assumed t o start in the all zero state, i.e. all U: and U: are zero initially. To achieve a higher global rate than R = 1/3, the outputs and s2Pare punctured according t o a puncturing matrix P . To achieve R = l / 2 , P is chosen as . The first row ? determines when x is sent and the bottom row when x p is sent. The channel investigated here is the AWGN channel, where after modulation each transmitted symbol is added t o an independent noise sample with zero mean and variance No/2. The receiver inserts those symbols t h a t were punctured with the value zer? into the sequence R. The k-th element of the sequence R is defined as
SIP

The Decoder
T h e MAP Algorithm

3.1

[Ay]

We will not repeat the derivation of the MAP algorithm here, but only state the deviations necessary for a more structured approach t o pass information between the decoders. The notation applies to the first decoder in the concatenated scheme, the second decoder can be treated is the same way. In [3], an additional variable 2 was introduced which was an indepen, dent observation of xi that was made available by the other decoder. The first decoder had three channel inputs, the second decoder had two (it did not use the systematic information directly). I t was stated in the paper that normalization is necessary to convert the LLR (the MAP output) t o a form suitable for use as an input t o the MAP. In the following we will present the MAP algorithm in such as way that estimation of variances is not necessary -we simply convert the LLR output by one decoder t o a-priori probabilities t o be used by the other decoder. Let the state of the encoder a t time IC be Sk, can take on it values between 0 and 2M - 1. The bit dk is associated with the transition from step le - 1 t o step le. Goal of the MAP algorithm is t o provide us with the logarithm of the ratio of the a posteriori probability (APP) of each information bit dk being 1 t o the A P P of it being 0. In a derivation similar t o [3] we obtain:

(2.2;

The forward recursion of the MAP can be expressed in a simpler fashion (no splitting of a ) , as in [ ] 4:

2.1

Trellis Termination

ak(m)

We have mentioned in the introduction, that a solution t o the problem of terminating the trellises of both component encoders has not been stated in [3]. Since the encoders are recursive, it does not suffice to set the last A4 information bits t o zero in order t o drive the encoder t o the all zero state. The termination sequence necessary, is a function of which state the encoder is in a t the point when termination is t o begin. If we force the upper encoder t o the all zero state, However, if we then we can still evaluate the sequence .d now wish t o terminate the second encode; also, this would mean changing the information sequence d due t o the interleaving between the component encoders. This would affect the state of the first encoder before termination begins, requiring a different termination sequence for the first encoder, resulting in a different d and so forth. There appears no simple solution t o terminate both encoders a t the same time using only M bits. In our investigations so far, we terminated the first encoder only, and left the second trellis open. This meant a slight modification t o the way the decoder associated with the second encoder is initialized. Due t o terminating the upper encoder, we only actually transmit N - M information bits.

c cYi(Yk, m ) . = m, m) . cc
1

@k-l()

i=O

(3)
%-I()

Yi(Yk,

m mi=O

A similar equation is obtained for the backward recursion & ( m ) . The branch transition probabilities are given by
Y~(Y;, y i p , m, m ) = p(y:ldk = i, S = m , S k - 1 = m). k

q ( d k = i1Sk = m , S k - 1 = m) is either zero or one depending on whether bit i is associated with the transition from state m to m. It is in the last component t h a t we use the information of the previous decoder: the probability P T { S ~ mlSk-1 = m} depends directly on the a-priori = probability of the information bit dk [5]. We use the a-priori probability of bit dk given t o us by the previous decoder in

(5)
if q ( d k = 1lSk = m, S k - 1 = m) = 1; and
oL(dk)

1299

with the variance of the meta-channel in the evaluation of the branch transition probability. This is unnecessary since we can use the LLR directly. But we can still calculate the signalto-noise ratio of the meta-channel (at the output of each MAP decoder) in order t o decide whether or not we should carry on iterating. The hope is that the variance of the meta-channel should be low, if and only if further iterations are without effect on the number of bit errors in that block. -(.is -bS m) To do this, let us first remind ourselves what connects p(yisldk = i, S = m , Sk-1 = m) = H,e k No 1 (7) channel values and LLRs. Assume a channel model where -(ykP-bp (~,m,m)~ x c { - l , + l } is transmitted across an AWGN channel with p(y?ldk = i , Sk = m , S - = m) = k,e k1 No 1 (8) noise variance u2,the received value is y. It is easy to show where k, and IC, are constants that can be set to 1 for the that if rt 1 are equally likely that actual calculations, and bsl,(i, m, m) is the modulator output 2 . x i s - 1 resp. 2 . z: - 1 associated with the branch from state m at step IC- 1 to state m at step IC if the corresponding where h ( z ) is the conditional LLR of z given observation y. d k was equal to i. In [3] the concept of extrinsic and intrinsic information So the point to note is that there exists a linear relation bewas introduced in order t o prevent say, the first decoder from tween the LLR A(.) and the associated channel observation passing on information to the second decoder which had been y: y = c..XI. 22 A( generated by the second decoder in the first place. In other We want to evaluate the value of u. words, we must ensure that the a-priori information is indeThe P D F of z i s p x ( x ) = ~ . 6 ( ~ - - 1 ) + ~ . 6 ( z + l ) , assuming - ,2 pendent of the other information (observations) being used equally likely *1s; the PDF of the noise is p ( n ) = e1 . e 3 , in the decoder. We can write the MAP output for bit d k as: so the PDF of R ( z ) becomes
(ti,

if p(ds = OlSk = m,Sk-1 = m) = 1; otherwise we can assign P r { S k = mlSk-1 = m} arbitrarily. The term L(dk) = Lze(dk) = L z e ( d i I k is the extrinsic component of the LLR ) that the other decoder has output for the information bit d f I k = d k . It is used as a-priori information in the current decoder. The other probabilities in (4)are calculated according to

m m

Let m 2 ~ represent the second moment of the RV A ( x ) , it can (9) easily be shown that 2 m2A = 1. + + 5 + (3)] . Rearranging, we arrive with y l ( y y , m , m ) = p(ypldk = i,Sk = m,Sk-1 = m) q ( d k = ilSk = m, Sk-1 = m). The second component in (9) at a quadratic equation for U: g;m2A = 4a2 4,with the m 2 So we must estimate the is the a-priori term, generated by the previous decoder, and single sollution a2 = 2t2d1tmzA. ~ the last components is the systematic term. The first compo- second moment of A(x) at every decoding stage, in order to nent is the extrinsic component L l e ( d k ) ,and is independent evaluate an estimate of u2. of the a-priori and systematic information for the bit dk. We now have an estimate of the overall quality of the decoded bits for that iteration. For example, a variance of 0.055 is equivalent to an AWGN channel with an SNR of 9.6 dB 3.1.1 Initialization and should have an average BER of The number of a0 and PN must be initialized as follows: bit errors found in a block at each iteration was found to ao(0) = 1,ao(m # 0 ) = 0; PN(O) = 1,P N ( # 0) = 0. (10) be closely linked to the predicted average value. In our sim~ ulations, we have set the threshold t o stop iterations to an Except in the case for the decoder associated with the second equivalent channel SNR of either 10 dB or 12.5 dB depending encoder, where the trellis is simply left open: on the BER range we are operating in. In practice we have observed the following: blocks that were hard to decode (i.e. ao(0) = 1,ao(m # 0 ) = O ; p N ( m ) = aAr(m). (11) needed many iterations) have a high variance associated with In the second case the backward recursion uses the value of each iteration until they are error-free. No situation has been the state probabilities generated by the last forward recursion encountered so far where the variance threshold was reached, step [6]. yet where more iterations could still have improved performance.

[5 (3)

3.2

Reducing Computational Requirements by Observing the M A P Output Second 3.3 The Decoder Structure Moment We are now able to describe how the individual components

Earlier we have said that one method of making the interface of the decoder are linked together. Fig. 1 shows the involved between the two MAP decoders is to use channel values along variables. The two central units are the (symmetrical) MAP
1300

decoders associated with the two encoders, the extrinsic in- size. Observe the reduction in the steepness of the curves, formation passed between them must be interleaved or de- particularly for N = 100, compared t o the large block size interleaved accordingly. Decoder delays are not taken into presented in 131. Fig. 6 shows the SNR needed t o achieve a BER of account in the figure. and 10W5 as a function of the block size N for memory L2e(d:,l). JZe(d:,J) 2 and 4 component codes, for R = l / 2 . The number of De-Inteilesver iterations was the same for memory 2 and 4: 18 for N 2 4000; (Lze(d:), 10 for 1024 5 N 5 2000 and 6 for N 5 400. From these No LZe(df,)) (Lie(drl)> + curves we notice the following:
I

...I

JlP

.+MAP

1-4
le ( d N ))

Lie(drN)) Interleaver
B2P-

*MAP 2

The steepness of the BER curves increases dramatically ponent codes.

( A ( d i 1,

..., *

r
Decision a n d De-inicrleavei

Estimate m 2 L

Evaluate
T

Threshold
( U '

Data out

<

0.03)

At BER the penalty for using memory 2 is only about 0.25 dB; small when seen in the light of a com0

Stop iterations

Figure 1: Structure of the Turbo-Decoder.

The loss from using memory two codes becomes more pronounced for block sizes ranging from approximately 400 t o 4000 and only if a BER of is required.

Simulation Results
5

A saturation seems t o have been reached for very large N . For very short blocks the turbo-code construction
does not seem worthwhile.

In order t o produce more reliable simulation results in a relatively short time, we have chosen t o restrict our main investigations t o block sizes N 5 10000 for memory 4 codes. In Fig. 4 we see the performance in terms of BER for N = 10000 and M1 = M 2 = 4, R = 112, generators 37 and 21 as given in [3]. A random interleaver was used. The parameter is the maximum number of iterations allowed (not the average, which is shown additionally). Note the sharp drop a t around 0.75 dB -the BER drops by two orders of magnitude in only 0.1 dB at, 18 iterations. Interesting is the drop in the steepness of the curve at 0.9 dB, this is an effect we will also see for smaller block sizes. The dotted line shows the evaluation of the approximate bound (14) introduced in the next section. Never up t o the present time have error events been observed that lie above the raw bit error rate of the channel, (for block sizes N 2 1000). This appears t o be due t o the fact that the decoder makes use of channel inputs a t each decoding step. If the outputs of the MAP decoders become unreliable, then the decoder will rely more heavily on the channel values. Finally, we see that the average number of iterations needed is smaller for larger SNR. Since it was observed that a less powerful memory two codes performed better in the first iteration, we tried a hybrid construction where M = 2 for the first code and A4 = 4 for the second code. For N = 10000 and 18 iterations, we found that this combination is only 0.1 dB worse (at a BER of l o v 4 ) than the original A4 = 4 construction, a t a lower overall decoder complexity. In Fig. 5 we see the BER performance for N = 100, 400 and 1024;again R = l/2. Also shown for N = 100 and 1024 is the degradation due t o not terminating the second trellis (done by transmitting the all zero sequence -both coders are always in the all zero state so we can use (10) t o initialize the second decoder), it is more pronounced for the smaller block

Minimum Distance of Whole Code and Interleaver Choice

Let us assume that we are transmitting the all zero d a t a sequence except that two bits are ones (bearing in mind that the code is linear). For a random interleaver, there is a chance that the interleaver maps these two non-zero bits from d to 'd in such a way t h a t the resulting distance from the all zero code word is small. This is because the recursive codes can be driven out of the all zero state and back t o it with a minimum of two ones that are suitably spaced. The above described event corresponds t o this being the case for both encoders, due t o an 'unlucky' mapping of the interleaver shown in Fig. 2. The rate 2/3 memory 4 component code with octal generators 37 and 21, has an error event where the component code word corresponding t o the information sequence ...00100001000 ... is Hamming distance 4 away from the all zero component code word. If the mapping described above really occurs, then this will result in two 'turbo'-code words (the all zero code word and the code word corresponding to ...00100001000 ...) t o be 4 2 = 6 apart, assuming the worst case puncturing of ZIP and Z2P and bearing in mind that the systematic information is only sent once. This is just one example of a set of two code words that are a small distance apart, there are many different such mappings, analysis of which is quite involved. If one knows the number, u ( w , d), of such code words of information bit weight w, that are Hamming distance d away from the all zero code word then one can apply the union bound [7] for the bit error rate:
4

z=

1301

The problem lies in finding the distance spectrum u(w,d). For distances d less than 50, and information bit weights w not more than about 6 this can be done by computer search that takes the puncturing, and of course the component code, into account.

0000001000~010000.

.I

1. .

00/010

Figure 3: Resolving the event of Fig. 2. Ik and Ii are swapped, (IC chosen at random.) Dashed: before swapping. Recent investigations have shown that the component code with the generators 23 and 35 lead to better distance properFigure 2: Illustration of a mapping leading t o a small distance ties of the turbo-code. The above described weight two and event. Shown are the sequences d (top) and d. Also the cor- weight four events have a much higher distance. Through our responding paths through the trellises of the two component interleaver search, we have found a code that has 7 weight four code words at Hamming distance 21 from the zero code encoders. word. However, information bit weight five and higher code It has been observed that the event leading to the minimum words still need to be analyzed. distance of 6 is quite rare, it typically occurs just a few times for any particular interleaver. And assuming that it does Conclusions occur , the information sequence weight difference between 6 the competing code words is also small. The results of such We have shown how the iterative turbo decoder can be fora search and evaluation of (14) has been compared t o our mulated without needing an estimate of the variance of any simulations, and is shown as dotted lines in the figures of the of the decoder outputs, furthermore, we have not found any next section. The approximate bound is reasonably accurate heuristically determined correction factors that help stability for higher SNR where neglecting higher weight (and hence to be necessary as suggested in [3]. The results claimed in [3] higher distance) events is less detrimental t o the approximate (for N I= 65536) can be reproduced except for a flattening of bound. the BE11 curves -especially for smaller block sizes. We now present a method of interleaver construction that It is possible t o analyze the performance of the turbo-codes works as follows: at high SNR by giving an approximate union bound for the bit error rate. This involves partial analysis of the inter1. Begin with any randomly chosen interleaver. leaver, component codes and puncturing. Extensive search 2. Look for the pattern in the information bit sequence re- for higher information weight code words remains a chalsulting in the smallest distance from the all zero code- lenging task. An interleaver modification procedure was presented that minimizes this approximate bound and has reword. sulted in an improvement where the BER curve flattens out. 3. In this pattern weJocate the first position in the infor- Finally, we have given a simple suboptimal solution to the mation sequence d that is involved, in the example of trellis-termination problem that remains unmentioned in [3] Fig. 3 the position i. We then randomly select another and show that the degradation due t o not terminating the position E (1, ...N } but not equal t o i, in our example k. second trellis is only significant for smaller block sizes.
all zero state

\&.J

~~

all zero state

4. Then we simply swap

5 . Continue with step 2.

I i

and I k .

Acknowledgements

Note that the swapping can result in an even worse pattern The author would especially like to thank Dr. J . Lodge, Dr. being created, but after many such operations the interleaver J . Hagenauer and M. Moher for many invaluable discussions will become better. To reduce the computational burden of and comments. steb 2, we can initially limit the search to code words of information bit weight 1 (edge effects) or 2, then when we have References exceeded a certain minimum distance, continue t o include higher weights in the search. After each swapping operation, [l] G. Battail, Building long codes by combination of simple ones, thanks to weighted- output decoding, in Proc. URSI we need to start our search again at the smallest weight.
1302

'89, pp. 634-637, 1989

[2] J. Lodge, R. Young, P. Hoeher, and J. Hagenauer, "Separable MAP 'filters' for the decoding of product and concatenated codes," in Proc. ICC'9.9, pp. 1740-1745, May 1993.
[3] C. Berrou, A. Glavieux, and P. Thitimajshima, "Near Shannon limit error-correcting coding and decoding: Turbo-codes (l)," in Proc. ICC '93, May 1993.

[4] L. Bahl, J. Cocke, F. Jelinek, and J. Raviv, "Optimal decoding of linear codes for minimizing symbol error rate," IEEE Trans. IT., vol. 20, pp. 284-287, March 1974.

m D

[5] G. D. Forney, "The Viterbi algorithm," IEEE Trans. Commun., vol. 61, pp. 268-278, March 1973.
[SI J. Lodge. Personal Communication, June 1993.

\
W

[7] A. J. Viterbi and J. K. Omura, Principles of Digital Communication and Coding. New York: McGraw Hill Book Co., 1979.

,0
ma

Figure 5: BER of turbo-codes for smaller N a n d different interleavers. M1 = M 2 = 4. Generators 37, 21. Dotted lines show a p p r o x i m a t e bound.

83 18

Figure 4: BER of Turbo codes for N = 10000, M1 = M 2 = 4 Generators 37, 21. D o t t e d line shows a p p r o x i m a t e b o u n d .

Figure 6: Influence of t h e block size N on the SNR needed to achieve a BER of loT3, and for component code memory 2 (Generators 5, 7) a n d m e m o r y 4.

1303