Testing and Debugging of Real-

Time and Embedded Systems

Daniel Sundmark
Mälardalen University
 What will you learn today?
• Problems
– Timeliness, Observability,
Reproducibility, HW/SW Co-Design
• Possibilities
– Identical Hardware, Simple Hardware,
Strict Scheduling

2
 Introduction
REAL-TIME SYSTEMS EMBEDDED SYSTEMS TESTING DEBUGGING
Software Testing and
Debugging?
 We know about the facts
• ”Exhaustive testing is practically impossible”
• ”Testing cannot prove the absence of defects”
• ”Software development is hard”
• ”Testing is very time-consuming”
• ”Most software projects do not meet
deadlines”
• ”Finding bugs late is very expensive”
5
We know about complexity

Source: Richard Stiennon, zdnet.com: Bringing order to threat chaos through news, views and analysis
 Still…
• We want testing to be
– Efficient (detect many bugs per time unit)
– Effective (detect the important bugs)

7
 Still…
• We want testing to be
– Efficient (detect many bugs per time unit)
– Effective (detect the important bugs)
• Threats to effectiveness and efficiency
– Technical (e.g., low testability)
– Process-related (e.g., poor use of available
testing resources)
8
Real-Time and Embedded
Systems?
 Real-Time Systems
• Correctness depends a)

PRIORITY
on function and time B

– Tasks
A

TIME

– Deadlines b)

PRIORITY
– Scheduling B
A y=x

• Examples: Airbag,air TIME Deadline (A)

traffic control, etc…

10
 Embedded Systems
• Built into other devices/products
– Functionality, Flexibility, Communcation
• Examples: Airbag, dishwasher control
display
• Embedded systems ≠ Real-Time
Systems
11
 Embedded & Real-time
systems trends
• Increasing:
– Functionality
– Complexity
– Distribution
– Resource sharing
– Distribution of development

12
Real-Time System Testing:
Problems and Possibilities

Real-Time &
Embedded Software Testing
Systems
 Problems
TIMELINESS TESTABILITY HW/SW CODESIGN
 Timeliness
• Correctness in real-time systems depends on
timely delivery of correct results
• How to test timeliness?
– Adds another problem dimension to testing
• WCET/response time analysis (safe)
• WCET/response time measurement

15
 Testability
“Testability, a property applying to an empirical
hypothesis, involves two components:
(1) the logical property that is variously described as
contingency, defeasibility, or falsifiability, which means
that counterexamples to the hypothesis are logically
possible, and
(2) the practical feasibility of observing a reproducible
series of such counterexamples if they do exist.”
- Wikipedia

16
 Testability
• Falsifiability
Example: Our system
should be user-friendly!

17
 Testability
• Falsifiability To what extent the
internal state of a system
• Observability
can be determined by
observing its outputs

Richer output information = higher observability

Embedded Systems Probe Effect

18
 Testability
• Falsifiability Example I: My home
• Observability computer

• Reproducibility
Example II: Robot
manufacturing line

Non - determinism

19
 Testability
• Falsifiability
• Observability
• Reproducibility
• (isolatability, understandability and
lots of other ilities…)

20
 Observability and
Reproducibility Example
The Probe Effect and Non-
Determinism

21
 iter1 assign4 assign5

assign1 x = 4; iter4 assign6

iter2
cond2 assign7

assign2 cond3 assign8

iter3
y = 3x; assign9

cond1

assign3
task A task B task C

PRIO

B
A
TIME
 iter1 assign4 assign5

assign1 x = 4; iter4 assign6

iter2
cond2 assign7

assign2 cond3 assign8

iter3
y = 3x; assign9

cond1

assign3
task A task B task C

• Tasks PRIO

• ControlBflow graphs
• Shared Avariable: x
• Output variable: y TIME
 iter1 assign4 assign5

assign1 x = 4; iter4 assign6

• Gantt chart
iter2
cond2 assign7

• y-axis: Task
assign2
Priority cond3 assign8

• x-axis: Execution
iter3
Timey = 3x; assign9

cond1

assign3
task A task B task C

PRIO

B
A
TIME
 PRIO
x=0 y = 12
y = 3x;
B x = 4;
A
TIME

PRIO
Observability problem
x=0 y=0
y = 3x;
B x = 4;
A
TIME

PRIO Reproducibility problem

x=0 y=0
C y = 3x;
B x = 4;
A
TIME
 HW/SW Co-Design
• Example: Mobile phone development
– Hardware is developed in parallel with software
– How to get fast to first system integration?
• Hardware-in-the-loop (HIL)
– Electronic simulation of environment and attached
subsystems
• Software-in-the-loop (SIL)
– Software simulation of environment and attached
subsystems

26
 Summary:
Real-Time- and Embedded System-Specific Test and Debug Problems

• Timeliness
• Testability
– Observability
– Reproducibility
• HW/SW Co-Design

27
 Possibilities
IDENTICAL HARDWARE SIMPLE HARDWARE STRICT SCHEDULING
Simplicity and Predictability

In order to discuss the possibilities of real-

time system testing and debugging, I need to
explain how complexity affects software
(and system) testing

29
 How to Measure Test Adequacy
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];

for (i = 1; i <= n; ++i) {

d[i] = INFINITY;
visited[i] = 0;
}

d[s] = 0;
Test case: < input, output >
for (k = 1; k <= n; ++k) {
mini = -1;
for (i = 1; i <= n; ++i)
if (!visited[i] && ((mini == -1) || (d[i] < d[mini])))
mini = i;

visited[mini] = 1;
for (i = 1; i <= n; ++i)
if (dist[mini][i])
if (d[mini] + dist[mini][i] < d[i])
d[i] = d[mini] + dist[mini][i];
}
}
 Graph Representation
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
assign1
} assign1 assign2

assign2 iter2

for (iter2) { assign3

assign3
for (iter3) iter3
if (cond1)
assign4 cond1 assign5

assign5 assign4 iter4

for (iter4)
if (cond2) cond2
if (cond3)
assign6 cond3
}
}
assign6

CONTROL FLOW GRAPH

 Test Criteria and Coverage
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
assign1 assign1 assign2
}
iter2
assign2
assign3
for (iter2) {
assign3 iter3
for (iter3)
if (cond1) cond1 assign5
assign4
assign4 iter4
assign5
Test Criteria
for (iter4) cond2
if (cond2)
• Statement Coverage
if (cond3) cond3
assign6
}
assign6
}

CONTROL FLOW GRAPH

 Test Criteria and Coverage
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];

for (iter1) {
assign1
} iter1
assign2

for (iter2) { assign1 assign2

assign3
for (iter3)
if (cond1)
assign4 iter2
assign5
for (iter4)
if (cond2)
if (cond3)
assign3
assign6
}
}
iter3

cond1 assign5

assign4 iter4
Test Criteria cond2
z Statement Coverage cond3
z Path Coverage assign6

CONTROL FLOW GRAPH

 Creating a Data Flow Graph
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
assign1 assign1 assign2
}
iter2
assign2
assign3
for (iter2) {
assign3 iter3
for (iter3)
if (cond1) cond1 assign5
assign4
assign4 iter4
assign5
for (iter4) cond2
if (cond2)
if (cond3) cond3
assign6
}
assign6
}

CONTROL FLOW GRAPH

 Creating a Data Flow Graph
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
d[i] = INFINITY; assign1 assign2
visited[i] = 0;
} iter2

d[s] = 0; assign3

for (iter2) { iter3

mini = -1;
for (iter3) cond1 assign5
if (cond1)
mini = i; assign4 iter4

visited[mini] = 1; cond2
for (iter4)
if (cond2) cond3
if (cond3)
d[i] = d[mini] + assign6
dist[mini][i];
}
} CONTROL FLOW GRAPH
 DU-path Coverage
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
d[i] = INFINITY; d[i] = INFINITY; d[s] = 0;
visited[i] = 0; visited[i] = 0;
} iter2

d[s] = 0; mini = -1;

for (iter2) { iter3

mini = -1;
for (iter3) cond1 visited[mini] = 1;
if (cond1)
mini = i; mini = i; iter4

visited[mini] = 1; cond2
for (iter4)
if (cond2) cond3
if (cond3)
d[i] = d[mini] +
d[i] = d[mini] +
dist[mini][i];
dist[mini][i];
}
} DATA FLOW GRAPH
 DU-path Coverage
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
d[i] = INFINITY; d[i] = INFINITY; d[s] = 0;
visited[i] = 0; visited[i] = 0;
} iter2

d[s] = 0; mini = -1;

for (iter2) { iter3

mini = -1;
for (iter3) cond1 visited[mini] = 1;
if (cond1)
mini = i; mini = i; iter4

visited[mini] = 1; cond2
for (iter4)
if (cond2) cond3
if (cond3)
d[i] = d[mini] +
d[i] = d[mini] +
dist[mini][i];
dist[mini][i];
}
} DATA FLOW GRAPH
 DU-path Coverage
void dijkstra(int s) {
int i, k, mini;
int visited[GRAPHSIZE];
iter1
for (iter1) {
d[i] = INFINITY; d[i] = INFINITY; d[s] = 0;
visited[i] = 0; visited[i] = 0;
} iter2

d[s] = 0; mini = -1;

for (iter2) { iter3

mini = -1;
for (iter3) cond1 visited[mini] = 1;
if (cond1)
mini = i; mini = i; iter4

visited[mini] = 1; cond2
for (iter4)
if (cond2) cond3
if (cond3)
d[i] = d[mini] +
d[i] = d[mini] +
dist[mini][i];
dist[mini][i];
}
} DATA FLOW GRAPH
 Multi-Tasking, Time, and Real-Time
iter1 iter5

assign1 assign2 assign7 assign8

iter2 iter6

assign3 assign9

iter3 iter7

cond1 assign5 cond4 assign11

assign4 iter4 assign10 iter8

cond2 cond5

cond3 cond6

assign6 assign12

Task A Task B

processor
 Shared Variables
iter1 iter5

assign1 assign2 assign7 assign8

iter2 iter6

x = 0; assign9

iter3 iter7

cond1 x = 1; cond4 assign11

assign4 iter4 assign10 if (x>0)

cond2 cond5

cond3 cond6

assign6 assign12

Task A Task B
 Complexity and preemption
• Question: what happens to the number of
potential system behaviours (e.g.,
represented by paths or DU-paths) as timing
variability increases?
• Answer: it explodes!
• This might be ok if the tasks and the systems
look like in the above examples, but…
41
 ModelDU(MS): How Many DU-Paths Exist?

Source: Richard Stiennon, zdnet.com: Bringing order to threat chaos through news, views and analysis
 Real-time systems typically
strive to limit timing variability
• Identical hardware
• Simple hardware
• Strict scheduling

43
 Identical Hardware
• Desktop system
– Many configurations
• Embedded real-time system
– Often (not always) a single or few
configurations

44
 Simple Hardware
• 8-, 16-, and 32-bit microcontrollers
• Low Jitter (=WCET-BCET is small)
• Temporal predictability
• Caches? Pipelines? Hard drive?

45
 Strict Scheduling
• Static number of tasks
– Time triggered
– Event-triggered (sporadic and (aperiodic))
• No dynamic memory allocation
• Interrupt limitations

46
 root

[0,8)
A:4
[0,0]
Usage example: EOGs
[7,8)
(Thane, Hansson)
[90,98)
A:4

[97,98) • Displays all execution

[100,137)
B:2
orderings in a FPS-
[107,137) scheduled system
[148,170)
[135,184)
C:1 [170,170] • Each path is treated
as a sequential
program
[175,180) [170,211) [175,180) [170,211)
D:3 D:3

[180,180] [180,180) [180,180]

[180,188) [180,188) [175,194) [180,188) [180,188)

• Large jitter -> large
A:4

[187,188)
A:4

[187,188) [175,180)
C:1

[180,180]
A:4

[187,188)
A:4

[187,188)
number of paths
I [187,219) [180,188) [180,188) [187,202) [187,219)
D:3 A:4 A:4 C:1 D:3

[187,219) [187,188) [187,188) [187,202) [187,219)

II III [187,202) V [187,233)

C:1 C:1
[187,202) [187,233)
47
IV VI
 root

[0,8)
A:4
[0,0]
Assignment: EOGs
[7,8)
(description found in course webpage)
[90,98)
A:4

[97,98) • Assignment 1:
[100,137)
B:2
based on a schedule
[107,137) where tasks have no
[148,170)
[135,184)
C:1 [170,170]
jitter, create an EOG
• Assignment 2:
based on same
[175,180) [170,211) [175,180) [170,211)
D:3 D:3

[180,188) [180,188)
[180,180]

[175,194)
[180,180)

[180,188)
[180,180]

[180,188)
schedule, but with
A:4 A:4 C:1 A:4 A:4
task jitter, create a
new EOG
[187,188) [187,188) [175,180) [180,180] [187,188) [187,188)

I [187,219) [180,188) [180,188) [187,202) [187,219)

D:3 A:4 A:4 C:1 D:3

[187,219) [187,188) [187,188) [187,202) [187,219)

II III [187,202) V [187,233)

C:1 C:1
[187,202) [187,233)
48
IV VI
 My Research

Example: Reproducibility (and

observability) by Execution
Replay
 PRIO
x=0 y = 12
y = 3x;
B x = 4;
A
TIME

PRIO
x=0 y=0
y = 3x;
B x = 4;
A
TIME

PRIO
x=0 y=0
C y = 3x;
B x = 4;
A
TIME
 iter1 assign4 assign5

assign1 x = 4; iter4 assign6

iter2
cond2 assign7

assign2 cond3 assign8

iter3
y = 3x; assign9

cond1

assign3
task A task B task C

PRIO
x=0 y = 12
y = 3x;
B x = 4;
A
TIME
 iter1 assign4 assign5

assign1 x = 4; iter4 assign6

iter2
cond2 assign7

assign2 cond3 assign8

iter3
y = 3x; assign9

cond1

assign3
task A task B task C

PRIO
x=0 y = 12
y = 3x;
B x = 4;
A
TIME
Summary
 Summary
Problems Possibilities
• How to test timeliness • Predictability and
• Low testability Simplicity acheived by
– Low reproducibility – Identical hardware
– Low observability – Simple hardware
– Strict scheduling
• HW/SW Co-Design
delays system-level
testing
Thank you!

Questions?