Adaptive Filters:

Introduction
C
W
C The back ground of adaptive filters is introduced. The key issues to be
covered during the course and some applications of adaptive filters are
considered.

Contents:
■ Filters
■ Adaptive filters
■ Linear filter structures
■ Adaptive algorithms
■ Applications
■ Historical notes

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 1

Filters
■ Dictionary entry for filter
● any substance, as cloth, paper, porous porcelain, or a layer of charcoal or sand
through which a liquid is passed to remove suspended impurities or to recover
C solids
W
C
● a device that selectively damps oscillations of certain frequencies while not
affecting oscillations at other frequencies.
■ In signal processing: to remove undesired signal components while not
affecting the desired ones.
● Often specified in frequency domain: lowpass, highpass etc. filters to remove
out of band signals.
■ Often the desired signals and undesired signals are at the same frequency
band. In such a case statistical characterization for filter design is needed.
■ A filter is linear if its output is a linear function of the input, i.e., if
Input Output
x1 → y1
x2 → y2
a1 x1 + a2 x2 → a1 y1 + a2 y2

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 2

1
 Prediction, Filtering and Smoothing

■ A predictor uses only the past samples up to discrete time n-1, where n is
C the time index of interest.
W
C ■ A filter uses also the current sample, i.e., samples up to time n.
■ A smoother uses also future samples, i.e., also samples n+1 and so on.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 3

Adaptive Filters
■ The optimal linear filters
C ● Wiener filters (stationary environment)
W
C ● Kalman filters (non-stationary environment with Markov model)
depend on the statistical parameters.
■ Change in parameters ➾ the filter must be adapted, i.e., to be adaptive.
■ Two problems:
● estimation of the statistical parameters
– usually second order statistics (covariance)
● computation of the filter impulse response.
■ For simplicity, joint solution by adaptive algorithm.
■ Adaptive algorithm self-tunes the impulse response of the filter
● adaptation based on filter inputs
➾ impulse response depends on data
➾ nonlinear filter (strictly speaking).

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 4

2
 Performance Measures of Adaptive Filters

■ Rate of convergence
C
W
■ Misadjustment
C ■ Tracking
● performance in nonstationary environment
■ Robustness
● impact of small disturbances
■ Computational complexity
■ Structure
● modularity, parallelism
■ Numerical properties
● numerical stability
● numerical accuracy
● impact of quantization.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 5

Linear Filter Structures

■ Conventional adaptive filters are linear.
C
W
■ Nonlinear adaptive filters:
C ● Volterra-based
● neural networks.
■ Structure of the filter
● finite impulse response (FIR)
– transversal
– lattice
– systolic array
● infinite impulse response (IIR).
■ Adaptive algorithm.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 6

3
 Transversal Filter
■ Transversal filter or tapped-delay line filter consists of
● unit-delay elements
C ● multiplier(s)
W ● adder(s).
C M −1
■ Filter output y (n ) = ∑ w k∗u(n − k ) = w Hu(n ),
k =0

[ ]
where T
w = w 0 w1 L w M −1 ,
u(n ) = [u(n ) u(n − 1) L u (n − M + 1)] .
T

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 7

Multistage Lattice Filter (1)

C
W
C

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 8

4
 Multistage Lattice Filter (2)

■ Predictor consisting of several individual stages, lattices.

C
W
■ Forward prediction error: fm (n ) = fm −1( n ) + κ∗m bm −1(n − 1).
C ■ Backward prediction error: bm (n ) = bm −1(n − 1) + κ mfm −1(n ).
● Above κm is the mth reflection coefficient.
■ Correlated input sequence u(n) → uncorrelated backward prediction error
sequence b(n).

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 9

Systolic Array

■ Parallel computing network of

boundary cells (a) and internal
C
W cells (b).
C ■ Capable to implement matrix-
vector products
➾ applications
● matrix triangularization
● matrix equation solving
(backsubstitution)
■ Efficient VLSI implementation due
to
● modularity
● local interconnections
● pipelined and synchronized
processing.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 10

5
 IIR Filter

■ FIR filters are usually

C preferred within adaptive
W filtering due to better stability.
C

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 11

Adaptive Algorithms
■ Stochastic gradient algorithms
● cost function based on statistical model (mean squared error)
C ● iterative minimization in the direction of negative gradient
W
– solution to the filter computation problem
C
● stochastic approximation for the gradient
– solution to the statistical parameter estimation problem
– simplest approximations (LMS algorithm) based on reference signals
(training).
■ Least squares estimation
● minimize the error of the filter output with respect to a reference signal (training)
– no statistical model directly involved
● recursive computation to simplify implementation
➾ recursive least squares (RLS) algorithms
– standard RLS
❍ based on matrix inversion lemma ➾ numerically unstable

– square-root RLS ➾ numerically stable

❍ based on QR decomposition

– fast RLS
❍ less computation by exploiting the matrix structure.
M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 12

6
 Applications
■ Identification
● system identification
C ● layered earth modeling.
W
C
■ Inverse modeling
● deconvolution
● adaptive and blind equalization.

■ Prediction
● linear predictive coding
● adapt. differential PCM
● spectrum analysis
● signal detection
■ Interference cancellation
● noise canceling
● echo cancellation
● adaptive beamforming.
M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 13

System Identification

■ Steps:
C ● experimental planning
W
C ● model selection
● parameter estimation
● validation.
■ Example: linear filter
system model
● filter coefficients found
by an adaptive
algorithm.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 14

7
 Adaptive Equalization
■ Adaptive equalization to remove
intersymbol interference (ISI).
C ■ Adaptation based on reference
W
C signal obtained from
● training
– unimodal error surface
● decision-direction
– multimodal error surface
● blind equalization
– higher order statistics
– cyclostationarity.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 15

Linear Predictive Coding

■ Model the source (speech) by a linear model (transfer function of a linear

C filter).
W
C ■ Use an adaptive algorithm to find the model parameters (filter coefficients)
➾ data compression.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 16

8
 Adaptive Differential Pulse-Code Modulation

■ Predict the new value of signal

C waveform.
W
C ■ Quantize only the uncorrelated
information (innovation process)
➾ reduction in number of bits
➾ data compression.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 17

Adaptive Spectrum Estimation

■ Parametric model
C of a stochastic
W process.
C
■ Linear filter
(autoregressive)
model
● input: white
noise
● output: the
observed signal
● find the model
parameters
(filter
coefficients) by
an adaptive
algorithm.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 18

9
 Adaptive Signal Detection

■ Adaptive line enhancer (ALE) can compensate for unknown signal

C distortions before detection (classification, hypothesis testing).
W
C ■ Sufficient statistics from
● ALE output
● ALE parameters.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 19

Adaptive Noise Cancellation

■ Reference output uncorrelated to the desired signal, but correlated to noise

C ● correlation unknown ➾ adaptive algorithm needed.
W
C ■ Sensor separation in space, time or frequency.

Applications:
● electrocardiogaphy
(ECG)
● acoustic noise in
speech
● adaptive speech
enhancement.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 20

10
 Echo Cancellation (1)

C
■ Coupling due to imperfect
W balancing in hybrid
C transformer creates an
echo in analog telephone
lines.
■ Echo signal can be
estimated by an adaptive
filter and the subtracted
out.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 21

Echo Cancellation (2)

C
W
C

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 22

11
 Adaptive Beamforming

■ Multiple sensors in space

C
W
➾ spatial sampling.
C ■ By appropriate combining
gain selection, the beam can
be steered.
■ In changing or unknown
environment, adaptive
algorithms are needed.
■ Application areas:
● radar
● sonar
● radio communications
● geophysical exploration
● astrophysical exploration
● biomedical signal processing.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 23

Adaptive Beamforming in Beam Space

■ Orthogonal beamforming network ➾ multiple
sidelobe canceler.
C
W ■ Several look directions for the main beam.
C ■ Null in the look direction of the other main beams.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 24

12
 Historical Notes (1)
■ Linear estimation theory
C ● method of least squares by Gauss in 1795
W
C ● minimum mean squared error estimation in late 1930s and early 1940s
● discrete-time Wiener-Hopf equation by Levinson in 1947
● Kalman filter by Swerling in 1958 and by Kalman in 1960.
■ Stochastic gradient algorithms in late 1950s
● stochastic approximation by Robins and Monro in 1951
● LMS algorithm by Widrow and Hoff in 1959
● gradient adaptive lattice (GAL) algorithm by Griffiths in 1977, 1978.
■ Recursive least-squares algorithms
● standard RLS algorithm by Plackett in 1950
● Kalman filter ➾ Godard algorithm by Godard in 1974
● exact relationship between RLS and Kalman filter by Sayed & Kailath in 1994
● QR decomposition based systolic array by Gentleman and Kung in 1981
● fast RLS algorithms in 1970s, in particular by Morf in 1974

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 25

Historical Notes (2)

■ Neural networks
C ● logical calculus for neural networks by McCulloch & Pitts in 1943
W
C ● perceptron by Rosenblatt in 1958
● back-propagation algorithm to train multilayer perceptrons by Rumelhart, Hinton &
Williams in 1986
– actually already by Werbos’s PhD thesis in 1974
● radial basis function network by Broomhead & Lowe in 1988
– idea already by Bashkirov, Braverman & Muchnick in 1964.

■ Applications:
● adaptive equalization in 1960s
– zero-forcing equalizer by Lucky in 1965
– MMSE equalizer by Gersho in 1969 and Proakis & Miller in 1969
❍ LMS analysis by Ungerboeck in 1972

– Godard algorithm by Godard in 1974

– fractionally space equalizer (FSE) by Brady in 1970
– decision-feedback equalizer by Austin in 1967 and MMSE by Monsen in
1971.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 26

13
 Historical Notes (3)

● speech coding
C – maximum likelihood speech prediction by Saito & Itakura in 1966
W – linear predictive coding (LPC) by Atal in 1970 and Atal & Hanauer in 1971
C – adaptive lattice predictor by Nakhoul & Cossell in 1981
● spectrum analysis, basics in early 1900s
– maximum entropy method by Burg in 1967
– method of multiple windows by Thomson in 1982
● adaptive noise cancellation started around 1965
● adaptive beamforming
– intermediate frequency (IF) sidelobe canceler by Howells in late 1950s
– control law (maximum SINR) for adaptive array antenna by Applebaum in
1966
– application of LMS algorithm by Widrow et al. In 1967
– minimum variance distortionless response (MVDR) beamformer by Capon in
1969
– simplified Gentleman-Kung systolic array for RLS estimation by McWhirter in
1983.

M. Juntti: Adaptive Filters: Introduction © Telecommunication Laboratory, University of Oulu 27

14