EITN90 Radar and Remote Sensing

Lecture 10: Machine learning approaches to

radar signal analysis
Daniel Sjöberg

Department of Electrical and Information Technology

Spring 2020
 Learning outcomes of this lecture
In this lecture we will
I Give an introduction to machine learning and simple
implementations.
I See how supervised learning has been used in radar target
classification problems.
I Consider a case study and live demonstration for gesture
recognition.
Target
Transmit signal Antenna

Transmitter T/R

Receive
signal

Receiver
protector
switch Mixer/Preamplifier
Detection and
Signal
Detector A/D measurement
processor
results

Low noise Local

IF amplifier
amplifier oscillator
Receiver (Adapted from Fig. 1-1)

2 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

3 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

4 / 44
 Some radar data representations
RANGE-DOPPLER CONTOUR PLOT OF UNPROCESSED DATA
5.5

4.5

range (km)
4

3.5

2.5

-150 -100 -50 0 50 100 150

velocity (m/s)

Classical PPI radar screen Doppler-range plot

SAR-image
Interpretation of the data may require a skilled operator.
5 / 44
 Machine Learning

Machine learning
A computer program is said to learn from experience E with respect
to some class of tasks T and performance measure P if its perfor-
mance at tasks in T, as measured by P, improves with experience E.
(Tom Mitchell, 1997)
The ML approach focuses on the data rather than how the data is
generated. Four main problems are in focus:
I Classification
I Regression
I Clustering
I Dimensionality reduction

6 / 44
 Solving different ML-problems with scikit-learn
An easy-to-use package: https://scikit-learn.org

More possibilities with https://www.tensorflow.org/

7 / 44
 Different classifiers

Classification example: training data in 2D, blue or red dots

distributed in three different ways.

Different ML approaches provide different predictions (the red and

blue colored backgrounds).

In the following, we focus on support vector machines (SVM) for

simplicity of presentation, but there are several alternatives.
8 / 44
 Linear Support Vector Machine (SVM)

A linear support vector machine finds the widest linear separation

of labeled training data (a hyperplane in higher dimensions).
9 / 44
 SVM mathematics
Training data: {(x1 , y1 ), . . . , (xn , yn )}, where xi ∈ Rd is the
observation and yi ∈ {−1, 1} is the classification. The parameters
w ∈ Rd and b ∈ R are determined by a convex optimization
problem:

Hard margin Soft margin

(training data is separable) (training data not separable)

1 T n
minimize w w 1 T X
2 minimize w w+C ζi
2 i=1
subject to yi (wT xi − b) ≥ 1
subject to yi (wT xi − b) ≥ 1 − ζi
ζi ≥ 0, i = 1, 2, . . . , n
Classification of new data is made using the decision function:
f (x) = sign(wT x − b). SVMs are considered robust and able to
learn from small sets of data.
10 / 44
 Kernel trick

When data is not separable by a hyperplane, replace the scalar

product xT x0 with a product in a higher dimensional space, like
k(x, x0 ) = ϕ(x)T ϕ(x0 ), where ϕ(x) = (a, b, a2 + b2 )
This can allow for finding a separating hyperplane in the higher
dimension. A typical kernel is the radial basis function kernel,
!
kx − x 0 k2
k(x, x0 ) = exp −
2σ 2
11 / 44
 Neural networks
Another popular model is artificial neural networks (ANN),
connecting the inputs
P to the outputs via a sequence of functions
(neurons) yk = ϕ( i wki xi ), where ϕ is the activation function
and wki are the weights. The network may have many layers.

http://playground.tensorflow.org 12 / 44
 Deep learning

Short detour to slides 8 and 12 in Deep Learning Basics 2019 from

MIT, https://deeplearning.mit.edu/.

13 / 44
 Preprocessing

In order to make the data more amenable to ML techniques, some

preprocessing is usually required:
I Mean removal
I Scaling
I Normalization
I Binarization
I ...

14 / 44
Unified framework for automatic target recognition (ATR)

I Identify the target set: what will be observed?

I Select the feature set: what is important?

I Observe the feature set: measure accurately

I Test the feature set: train and use a classifier

15 / 44
Unified framework for automatic target recognition (ATR)

I Identify the target set: what will be observed?

I Broad classes of targets (humans, rabbits, cars, bikes. . . )
I Variations within classes (adults, children, minivans, trucks. . . )
I Select the feature set: what is important?

I Observe the feature set: measure accurately

I Test the feature set: train and use a classifier

15 / 44
Unified framework for automatic target recognition (ATR)

I Identify the target set: what will be observed?

I Broad classes of targets (humans, rabbits, cars, bikes. . . )
I Variations within classes (adults, children, minivans, trucks. . . )
I Select the feature set: what is important?
I Maximize the similarity of objects in the same class while
maximizing the dissimilarity of objects in different classes.
I Observe the feature set: measure accurately

I Test the feature set: train and use a classifier

15 / 44
Unified framework for automatic target recognition (ATR)

I Identify the target set: what will be observed?

I Broad classes of targets (humans, rabbits, cars, bikes. . . )
I Variations within classes (adults, children, minivans, trucks. . . )
I Select the feature set: what is important?
I Maximize the similarity of objects in the same class while
maximizing the dissimilarity of objects in different classes.
I Observe the feature set: measure accurately
I Processing to increase SNR: averaging, background removal,
pulse compression. . .
I Test the feature set: train and use a classifier

15 / 44
Unified framework for automatic target recognition (ATR)

I Identify the target set: what will be observed?

I Broad classes of targets (humans, rabbits, cars, bikes. . . )
I Variations within classes (adults, children, minivans, trucks. . . )
I Select the feature set: what is important?
I Maximize the similarity of objects in the same class while
maximizing the dissimilarity of objects in different classes.
I Observe the feature set: measure accurately
I Processing to increase SNR: averaging, background removal,
pulse compression. . .
I Test the feature set: train and use a classifier
I Supervised learning: each training observation is given a
ground truth from the operator.
I Unsupervised learning: the training data are clustered into
classes, which are then used as ground truth.
I Reinforced learning: learn from new data through feedback.

15 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

16 / 44
General methodology

17 / 44
 ML in remote sensing

Review paper summarizing many uses of ML in remote sensing.

Results from a few papers are shown in the following, using
multispectral data in each pixel to classify the terrain.

18 / 44
19 / 44
 Classification results

The training data consisted of 350 samples, 50 pixels per class.

The SVMs learn with few number of samples.
20 / 44
21 / 44
Data was aquired in 11 spectral
bands, only 3 used for processing.

22 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

23 / 44
Thorough review of target recognition techniques, not only ML.
Too much to go through here in detail.
24 / 44
25 / 44
26 / 44
27 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

28 / 44
 ML in micro-Doppler analysis

I Complex live targets like a waving hand, running horse, flying

helicopter etc, present many different velocity components in
a Doppler spectrum.
I Recording the Doppler spectrum as a function of time
provides a 2D spectrogram a(fd , t).
I The analysis of a(fd , t) is called micro-Doppler analysis.
I ML techniques can be used to identify features in the
spectrograms and do classification.

29 / 44
30 / 44
31 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

32 / 44
 Gesture recognition

In 2015, Google released a 60 GHz radar platform (Soli) for

gesture recognition. Here, we will see that similar functionality can
be achieved using the 24 GHz radar system from the lab.
33 / 44
 Demonstration

control signal amplifier

radar module raspberry pi with AD/DA card

The same radar equipment as used in the lab last Friday has been
augmented with a simple gesture recognition program.
34 / 44
 Principal operation of the radar system

transmitting
Vtune VCO antenna

90◦

I
receiving
antenna

FMCW operation, stepped frequency.

 Signal model
After down-conversion, we form the analytic signal from I and Q:
s(t) = I(t) + jQ(t) = (Aej(−4π(f0 +fd )R/c+φ) ) ej(2πfd t−4π∆f (t)R/c)
| {z }
=A0

Sample the up-chirp and down-chirp signal at tn = n∆t with

∆fn = nB/N , and take the Fourier transform:
(
A0 ejn2π(fd ∆t−2BR/(N c)) up-chirp
sn = =⇒ ŝn = FFT[sn ]
A0 ejn2π(fd ∆t+2BR/(N c)) down-chirp
Combine M signals into one data frame:
    
(0) (0) (0) (0) (0) (0)
 |ŝ0 | |ŝ1 | · · · |ŝN | |ŝ | |ŝ1 | ··· |ŝN |
 0(1)
 

(1) (1) (1)  (1) (1) 
 
 |ŝ0 | |ŝ1 | · · · |ŝN |   |ŝ0 | |ŝ1 | ··· |ŝN | 

 

one frame =  . .. .. ..  ,
 .. .. .. .. 
 ..
 . . .   . . . . 
  

 

 (M ) (M ) (M ) (M ) (M ) (M ) 
|ŝ0 | |ŝ1 | · · · |ŝN | |ŝ0 | |ŝ1 | ··· |ŝN |
 
up-chirp down-chirp

Only the amplitudes of the 2M × N complex numbers are used,

and each frame is normalized with its peak value.
36 / 44
 Choosing system parameters
I Carrier frequency f0 = 24 GHz.
I Bandwidth B = 425 MHz.
I Sampling time ∆t = 0.34 ms.
v 2·24·109
I Doppler shift fd = 2v
c f0 = 1 m/s 3·108 Hz =
v
1 m/s 160 Hz.
2BR
I Sampled received signal sn = A0 ejn2π(fd ∆t∓ N c ) (up- and
down-chirp).
I Phase shift relative to 2π at n = N , R = 0.5 m, v = 1 m/s:
2BR 2 · 425 · 106 · 0.5
Range: N = = 1.4
Nc 3 · 108
Doppler: N fd ∆t = N · 160 · 0.34 · 10−3 = 0.054N
The choice N = 32 makes range and Doppler almost equally
balanced with more than 2π phase change.
I Dwell time Td = N ∆t = 32 · 0.34 · 10−3 s = 0.011 s.
I Collecting M = 16 dwells in one frame gives aquisition time
Tf = 2M Td = 2 · 16 · 0.011 = 0.35 s, in the order of the time
of a typical gesture. 37 / 44
 Data processing

Each raw data frame goes through the following processing:

I Background subtraction.
I Estimation and subtraction of a ramp signal.
I Fourier transform (FFT).
I Taking the amplitude and normalizing by the maximum
amplitude in the frame.
A number of frames are used for training the SVM, which is
subsequently used to classify gestures performed.

38 / 44
 Gestures

0 0 0 0 0 0
f f f f f f

t t t

up-chirp down-chirp up-chirp down-chirp up-chirp down-chirp

39 / 44
 Additional gestures evaluated
I Static: a hand at different altitudes above the sensor
(separated roughly by the resolution c/(2B) = 35 cm).
I Circle: clockwise or counter-clockwise rotation of index finger
in non-radial motion (should be difficult to distinguish).
I Soli gestures:

Each gesture was repeated until 100 frames had been recorded.
For 1 ≤ n ≤ 29, n frames were chosen for training and 70 frames
for evaluation. The selection was random, and the outcome
averaged over 100 selections.
40 / 44
 Results for different sets of gestures
Solid = true, dotted = false Solid = true, dotted = false
1.00 1.00

0.75 0.75
0 0
1 1
Rate

Rate
0.50 2
0.50 2
3 3
0.25 0.25

0.00 0.00
0 10 20 30 0 10 20 30
Number of training frames Number of training frames

Thumb Static
Solid = true, dotted = false Solid = true, dotted = false
1.00 1.00

0.75 0.75
0
0
1
Rate

0.50 1 Rate 0.50 2

2
3
0.25 0.25

0.00 0.00
0 10 20 30 0 10 20 30
Number of training frames Number of training frames

Circle Soli
41 / 44
 Discussion of the gesture recognition system

I The system can distinguish between movements with

significantly different Doppler spectrograms.
I The system performs well for thumb up/mid/down, and static
poses separated by the resolution.
I The system performs less well for non-radial motion, and
micro-motions like the Soli gestures.
I With one single sensor, the gestures need to be adapted to
the sensor; with several sensors, more relaxed requirements for
the gestures are expected.
I Unsupervised learning can be implemented.

42 / 44
 Discussion of the gesture recognition system

I The system can distinguish between movements with

significantly different Doppler spectrograms.
I The system performs well for thumb up/mid/down, and static
poses separated by the resolution.
I The system performs less well for non-radial motion, and
micro-motions like the Soli gestures.
I With one single sensor, the gestures need to be adapted to
the sensor; with several sensors, more relaxed requirements for
the gestures are expected.
I Unsupervised learning can be implemented.

I Very short development time: a fully functioning system after

one weekend!

42 / 44
 Outline

1 An overview of machine learning problems

2 ML in remote sensing

3 ML in target recognition

4 ML in micro-Doppler analysis

5 Case study: gesture recognition with an FMCW radar

6 Conclusions

43 / 44
 Conclusions

I Some ML approaches to radar signal analysis have been

reviewed, with particular emphasis to supervised learning using
SVMs.
I Terrain classification in SAR images has been performed
based on multispectral data in each pixel.
I Target classification in SAR images may require an initial pose
estimation. Open data is scarce.
I Gesture classification in Doppler spectrogram is a field of
current research and implementation, expected to emerge in
consumer products.

44 / 44