ADA PROGRAMS USING

Software fault-tree
analysis provides extra
assurance that
systems meet safety
requirements. The
analysis templates
provided here apply to a
range of languages,
not just Ada.

NANCY G. LEVESON
STEPHEN S. CHA
Universiiy of California at lrvine

TIMOTHY J. SHIMEAU
US Naval Postgraduate School

w en computers are used to control

or partially control safetycritical
processes - those that can a w e or con-
neers are finding themselves faced with
potential liability and with government
agencies requiring certification of soft-
tribute to death, injury, loss of equipment ware safety. New standards are appearing
or property, or environmental hami - (such as Mil-Std-882B Notice 1 in the US
they themselves become safetycritical, and draft standard MOD-0055in Britain)
and there is a need to verify that the soft- that require extensive software-safety
ware will not cause or contribute to an analysis and verification procedures.
accident. These requirements are pushing the state
Until relatively recently, a natural re- of the art in both system-safety and soft-
luctance to add unknown and complex ware engineering.
factors to these systetns kept computers Software fault-tree analysis is a tech-
out of most safety-critical control loops. nique derived from a conunonly used sys-
But the potential advantages of using tem-safety analysis technique that is useful
computers are now outweighing appre- in partially verifymg the safety aspects of
hension (some might say good sense), and software. An earlier article' described how
both computer scientists and system engi- to apply the techtuque to the simple struc-

__
_ __
_

JULY 1991
tured statements of a Pascal-like language. Another way of loohng at this is that a lead to t h ~ hazard.
s 'The fault tree itself is a
Since then, it has been used on many in- 1 forward analysis tries to ensure that all graphical model of these event combina-
tions. T h e events may involve component
dustrial programming projects, including possible reachable states of the system are
avionics systems for military aircraft, a nu- ~ correct and safe, while the goal of safety hardware failures, human errors, or m y
clear power-plant shutdown system, med- analysis is to ensure that particular unsafe other events that can lead to the undesired
ical systems, and various military and aero- 1 states (whether correct or incorrect) are state. A fault mee thus depicts the logical
space software systems. Although until not reachable or are reachable onlywith a interrelationships of basic events that lead
recently most of t h ~software
s was written in I very low probability. to the hazard.
languages with simple control structures, , This type of safety analysis is practical
more complexityis now being introduced. i only under the assumptions that there are Basit procedure. The basic procedure in
This article extends the software fault- relatively few states that are unacceptable fault-tree analysis is to assume that the
tree analysis procedure to allow its use on , (hazardous) and that these can be deter- hazard has occurred and then to work
a more complex language involving such mined. In practice, this is usually true even backward to determine its set of possible
features as concurrency and exception in complex systems.
~ causes. The root of the fault tree is the
handling. We use Ada as the example lan- There have been several such system- hazard, and the necessary preconditions
p a g e because many safety-critical proj- ' safety analysis techniques developed, but are described at the next level of the tree
ects are using or planning to use Ada. It the most successful of these, fault-tree with either a logical And or a logical Or
also contains complex, real-time pro- analysis, was developed in the early 1960s relationship. Each subnode is expanded
gramming facilities found in other lan- ,
to analyze the safety of electromechanical similarly until all leaves describe events of
p a g e s used in these types of projects. 1 systems.' h fault-tree analysis, a hazard is calculable probability or cannot be ana-
Using the templates provided in this arti- specified and the system is then analyzed lyzed for some reason. Figure l shows part
cle, it should not be difficult to define the in the context of its environment and op- of a simplified fmlt tree for a traffic-light
procedures for applying the techruque to
programs written in most other declara- '
eration to find aedible parallel and se-
quential combinations of events that can
controller.
Once you have built the system-level
'
tive languages.
Safetv analvsis is not a substitute for the
i ,

usual functional-verification techniques.

'
'There will still need to be some type of
verification that the requirements are cor- Collision in
rect and complete and that the code is con- interredion
sistent with these requirements. T h e
safety-analysis techniques described here
help provide emz assurance that t h ~diffi-
s
cult process has been done correctly by
starting the verification from the system-
lights foil to control sohore Driver runs
safety requirements rather than from the turn green permits collision red light
software requirements and providing rig-
orous analysisthat the most aitical parts of
the problem have been handled ade-
quately. However, they provide no more
guarantee of perfection than any other
technique: The safety analysisis as likely to Sohwore turns Two drivers
be in error as any other verification proce- both lights to
oreen interredion
dure. T h e box on p. 50 explains basic
safety-analysis techruques.

SOFTWARE FAULT-TREE ANALYSIS

Most safety-analysis t e c h q u e s ensure
or assess safety using a backward approach
that starts with determining what are the
unacceptable or hazardous states and then
showing that these states cannot occur or
that their probability of occurrence is low. Figure 1. '4 simpli$ed .ystert-lrz.elfnultweef m a trffir-light ronrrollw.

IEEE SOFTWARE
WHAT IS SAFETY VERIFICATION?
System safety is defined in volves demonstratingthat the compensate for some software Alternatively, the software
terms of hazards and risk. A executionor nonexecution of failures. could be designedso that many
hazard is a set of condirions(a the software will not result in a Even in this second type of of these failure modes are non-
state)that can lead to an acci- ha~ardousstate. system, verification of safety critical. For example, one basic
dent, given certain environ- In most of these systems, need not necessarily be equiva- safety-design rule is to maintain
mental conditions. System- verification that the software lent to verifymg that the soft- the software (and the system) in
safety analysis involves does not do anydung harmful is ware willnever fail. It is neces- a safe state as much as possible.’
identiljmg the hazards of a sys- usually not equivalentto dem- sary only to demonstrate that In h s case, a safe state for the
tem and then either verifymg onstrating that the software the executionor nonexecution plant is to be shut down and a
that hazardousstates cannot be does anydung useful or that it of the softwarewiU not result in safe state for the softwareis to
reached or that the risk is ac- satisfies its functional-require- a hazardous state. It is possible have all variables with the value
ceptable. Risk is defined as a ments specification. In fact, to design such software so Trip. To implement this, in-
functionof the probabilityof a temporarily transferring to a some failures will not result in stead of initializingthe vari-
hazard occurring,the probabil- safe state that involves partial or hazards. ables to Untrip, they could in-
ity that the hazard will lead to total suspensionof normal soft- For example, consider soft- stead be initialized to Trip.
an accident,and the worst po- ware functionaloperation may ware to monitor and issue com- Procedures that check the sen-
tential loss associated with such be the optimal recoveryproce- mands to shut down a power sor values would then reset the
an accident You can diminish dure in some situations. plant when it reaches a poten- value to Untrip if the values are
risk by reducing any or all of Software can also contrib- tially dangerous state. Assume withm safe limits.This protects
these factors, and there are sys- ute to system hazardsif it fails that this must be accomphhed against softwareor hardware
tem-safetytechniques that to detect and take correctiveac- within a designated amount of problems that result in part of
focus on each. tion to recover fi-om a hazard- time after the sensors detect en- the code not being executed.
Software can thus increase ous system state or ifit does not vironmental parameters out- In addition, at the start of
risk in three ways: limit damage in the event of an side the safe ranges. the main loop, a watchdog
+ The softwarecan behave accident.This assumes, of One straightforwardway to timer could be set to the dead-
in a way that causes or contrib- course, that the softwarehas design such software is to have tine value. At the end of the
utes to the system reaching a been assigned these responsibil- a loop in which procedures are main loop, the watchdog is
hazardousstate. ities and a n take appropriate called to check the sensor vd- reset. If the loop does not com-
+ The software can fail to correctiveaction. The hazard- ues and, if they are outside the plete within the deadline,the
detect and take corrective ac- ous state may be reached inde- safe range,intemal variables watchdog will time out and a
tion to recover fi-oma hazard- pendently of previous software are set to denote that the plant signal to shut down the plant
ous state -this state may or behavior: For example, a hard- should be shut down, or will be issued by hardware.The
may not be related to software ware component such as a tripped. At the end of the main software is now protected
behavior. radar unit fails or the nuclear loop, the software checks to see againsttiming problems and
+ The software can fail to reactor reaches a critical state. if any of the intemal variables runtime errors resulting in
mitigate the damage once an In this case, the softwareacts as have the value Trip. In this de- aborts.
accident occu~s -it fails to re- a safety device (or an interlock) sign, almost any failure of the
The remaining hazard is
duce the probabilitythat the within the larger system, and its software could be critical: For
that the routines that check
worst-caseloss will occur. primary safety-relatedfunction example, the main loop may
plant conditions incorrectlyun-
In the first case, the soft- is to provide safety. not execute within the time
mp the variables when the sen-
ware causes or contributesto Any fadure of this type of limit,the procedures to check sor inputs indicate unsafe con-
the system reaching a hazard- softwareis potentiallysafety- the sensorvalues may incor-
ditions. This is the hazard that
ous state. This can occur either critical unlessthe larger system rectly decide they are within
must be analyzed with safety-
because the software does not or the software or both have limitswhen they are not, the
analysis techniques.
satisfyits specification or be- been designedto be safe de- code that checks to see if any of
cause the specified behavior is spite wme types of software the variables have the value
unsafe with respect to the over- failure. For example, electro- Trip may be incorrect,or a run- REFERENCE
all system operation.To protect mechanicalbackups to the soft- 1. N.G. Leveson, ‘‘Software Safety:
time error could result in no
What, Why, and How,”AGV
against this typeof unsafe soft- ware may sometimesbe de- output when the plant is in a Computing Sumty,June 1986, pp
ware behavior, verification in- signed to detect and critical state. 125.164.
~~
fault tree down to the software interface, can identifirerrors in the latter aid identify tics of the progra~nming-language state-
you have delineated the high-level re- functional requirements that conflict with ments. Because the technique depends on
quirements for software safety in terms of safety requirements. the semantics ofthe laqpage in which the
software behavior (usually involwng out- Because the p a l is to prove that the algorithm is specified, the templates may
puts or lack of outputs) that could ad- software will not do something, it is conve- be different for each language.
versely affect the system's safety for the nient to use the type ofreasoning involved
hazard being considered. in mathematical proofhy contradiction. In Template design. The templates and trees
After you have identified this hazard- software fault-tree analvsis, it is hypothe- in this article use three standard fault-tree
ous software behavior in the system fault sized that the software has symbols. A rectangle indi-
tree, you can apply software fault-tree produced an unsafe out- cates an event that must
analysis at the design or code level to iden- put, and it is shown that be analyzed further. A di-
tify safety-critical items and components, t h ~ could
s not happen be- aiiioiid indicates an event
to detect software logic errors, to deter- cause the hypothesis leads Fault-tree analysis tries that is not further ana-
mine the conditions under which fault- to a contradiction. T h e lyzed, either because it is
tolerance and fail-safe procedures should generation of any branch to verify that the inapplicable to the state-
be initiated, to guide the placement and ofa tree can be stopped a s program will never let ment being considered or
content of runtime checks to detect haz- soon as a contradiction is liecause a contradiction is
ardous software states, and to facilitate ef- located. If a path is found a particular unsafe found. An oval indicates a
fective safety testing by pinpointing criti- through the software and condition noniial to the
cal functions and test cases. If used with a out into the controlled
state be reached, but operation of the system
system simulator, you can examine the in- system or its environment it determines nothing that contributes to the
terfaces of the fault tree to determine ap- that does not contain a fiailure.
propriate siinulation states and events. 1o g i ca 1 contra d i c ti on , about incorrect, I n each template, it is
Software hilt-tree analysis has been used
on real software projects for each of these
software fiault-tree analy-
sis reveals the input con-
safe states. assumed that the state-
ment causes the critical
purposes. ditions necessary for this event, and the tree is con-
hazard to occur. structed by considering
Analysis goals. Software fault-tree analy- As an example, software fault-tree anal- how this might occur. ' l h s approach is th;
sis works backward from the cTitical con- ysis on a scientific-satellite control pro- same as that used in foniial axiomatic ver-
trol faults determined by the system fault gram' uncovered the fact that the satellite ification,j wherc you derive the weakest
tree through the p r o g " code or the de- could be destroyed if the input sensors de- preconditions necessary to satisfy the
sign to the software inputs. In otherwords, tected two solar pulses within (A 113s of given postcondidoils.
it starts froin hazardous outputs (or lack of each other. The designers aid progran- In faa, software fault-tree analysis is a
them) and waces backward inem had been entirely unaware that the graphical application of axiomatic verifi-
+ to find paths through the code honi correa operation of the software interrupt cation where the postconditions describe
particular inputs to these outputs or handler was based on an assumption about the hazardous conditions rather than the
+ to demonstrate that such paths do the minimum timing intend of the in- correcmess conditions. If the weakest pre-
not exist. coining solar pulses. condition is false, the postcondition can
The goal is slightly different and much An appropriateaction in h s case is to use never be satisfied. In software fault-tree
inore limited than that ofproving correct- runtitlie assertionsto detect such conditions analysis, this corresponds to a contradic-
ness (where correctness is defined as con- and siniply to reject incorrect or unsafe tion, and the analysis along the branch can
sistency of the code with a software re- input. In other cases,it might he most appro- be stopped. If the weakest precondition is
quirements specification). Software fault- priate to redesign the program, to initiate true, the program will always cause the
tree analysis tries to verify that the pro- software recovery routines, or to redesign hazard to occur - the program is mher-
gram will never let a particular unsafe state noncomputer pats of the systemto avoid or ently unsafe and must be changed. Under
be reached, although it detennines noth- mitigate h e effects of the hazard. all other conditions, the analysis must be
ing about the reachability of incorrect but followed back through the code until ei-
safe states. APPLYING SOFTWARE FAULT-TREE ANALYSIS ther true or false is obtained or an input
It also verifies that the program will not TO ADA statement has been reached (in which
allow cmvect but unsafe states to be case, again, the hazardous output is possi-
reached. This is possible because fault-tree Software fault-tree analysis uses fail- ble and the code needs to be changed).
analysis starts from a system-safety specifi- ure-mode templates to generate the fault
cation rather than the software's func- tree. T h e process of defining the templates Using the tempbtes far Ada. Consider the
tional-requirements specification and thus is equivalent to defining the failure seman- assignment-statement template in Figure

IEEE SOFTWARE 51
 Assignment
causer failure

i t h condition
Else part evaluation i th Then port
tourer failure tourer failure

-~ ~~~

Figure 2. Aaigiinient teniplflte Figure 4. Procedure-cull template.

2 . The assignment statement can cause of a compound statement, the analysis caused by a prior statement, and you must
failure because either an unsafe value was starts frotn the last statement. consider each prior statement in tum.)
assigned or the statement could not be ex- Analysis of procedures and function There are three ways a rendezvous
ecuted successfully. The possibility of the bodies (Figure 4) is straightforward. Be- could not occur:
former is captured in the leftmost branch cause a procedure body is a sequence of + the calling task may not be able to
whde that of the latter is explored in the sitnple or compound statements, you can execute the entry call,
righttnost two branches. analyze it with the existing templates. + the called task may not be able to
T h e software fault-tree analysis teni- Analysis of function calls is similar and is execute the Accept, and
plate for the If-Then-Else statetnent is part of the analysis of failures caused by + the called taskmay be able to execute
shown in Figure 3. Ada lets progratntners expression evaluation. The templates of the Accept, but some task other than the
include an arbitrary nutnber of Elsif state- Ada Case and Loop statenients are equally calling task may have made an entry call on
ments as desired. To simplifythe template, straightfoward, as Figures 5 and 6 show. that Accept and the called task proceeds
we have labeled this collectivelyas the “ith T h e templates for the Selecz and Ren- with a rendezvous with t h ~ sthird task.
T
.
h e n. part.” You can use additional dezvous statements are shown in Figures 7 If none of these conditions holds, the
branches for Elsif statenients as needed. ~ and 8. In the rendezvous, the event being rendezvous will occur, and you must con-
T h e body of Then and Else parts consists
of either a simple or a compound state-
tnent. T h e node “ith Then body causes
failure” or “Else body causes failure” are
’ analyzed (or “failure”)is caused by the ren-
dezvous if (1) either the rendezvous not
occurring causes the failure or (2) the ren-
dezvous does occur and a fault during the
sider whether a fault in the rendezvous
could cause the failure.
To determine if a fault in the rendez-
vous could cause the failure, you must
replaced by other statetnent templates as 1 rendezvous causes the failure. (If neither consider three cases. In the first case, the
the analysis proceeds. If the body consists case holds, the failure must have been values passed as parameters to the rendez-

5’
tourer failure

5‘tourer foilure

Figure F. Case template Figure 6. Loop tentplate.

 1 Accept body
rouses failure
1I '0s' obo'ed
I1 Parameters
rouses failure
1
U
+- Rendezvous
I I doesn't occur

Statements
I I I
touse failure
Colling tosk Colled task Called task in
not ot col1 not at Accept other rendezvous

or Iseebeiowl
causes failure

Called task not

rouser foilure at Accept

I 1Ond I 1 Ond

causes failure
conditiontrue
E1
lHu.*
n
t
i/To
h
ilioa(
F l rolled tosk above arcept rendezvous

Figure 7. Select template. Figure 8. Reizdezoozis tenrplnte.

vous are inappropriate and thus cause the

failure. In the second case, the body of the
Accept statement contains a fault, which
causes the failure. In the last case, the Ada Exreption
Lunpage RefieferemeManuul notes that if a causes
called task is aborted while it is in a rendez-
vous, an exception will be raised in the
calling task. You need to consider whether
this exception could lead to the event
I I I
under consideration (see the exception
template in Figure 9). Exception Exception
causes handler
Packages and generics in Ada consist of foilure roised
subunit specifications and corresponding
bodies. You can analyze the bodies by ex-
amining the statements in each of the con- I
stituent packages, procedures and func-
tions, as appropriate. It is necessary to
include the initialization bodyofa package
and the effect of instantiation on a generic
in the analysis.
T h e templates for the rest of the Ada
StatementSare described in the Figures 10
through 17.
 I I

Abort couses
failure

A I

Tasks not ye1 Tasks updole Tasks in Exception Block body

act iv aIed variables rendezvous couses failure causes failure

I
Figure 10. Abm-t tentplate.
I I 1 li
Figure 1 1 . Blork tmrplatr.

Code causes
failure
1
- l il

Incorrect
Task delay Expression
machine code
causes failure ev aI u at ion
causes failure
I
Figure 12. Code template.

1 Entry call causes

I
Exit causes
failure

I
 .-
TRAFFIC-CONTROL LIGHT EXAMPLE
1 procedure traffic is
2 type direction is (east, west, south, north);
T h e simplest way to explain the use of
3 type color is (red, yellow, green);
the templates is to analyze an example Ada 4 type light-type is array (direction) of color,
program. The program must satisfy the 5 lights : light-type := (green, green, red, red);
following requirements: 6 task type sensor-task is
A traffic-light control system at an in- 7 entry initialize (mydir : in direction);
tersection consists of four (identical) sen- 8 entry car-comes;
sors and a central controller. T h e sensors 9 end sensor-task;
in each direction detect cars approaching 10 sensor : array (direction) of sensor-task;
the intersection. If the traffic light is not 11 task controller is
green, the sensor notifies the controller so 12 entry notify (dir : in direction);
the light will be changed. A car is expected 13 end controller;
14 task body sensor-task is
to stop and wait for a green light. If the 15 dir : direction;
light is green, the car may pass the inter- 16 begin
section without stopping. T h e controller 17 accept initialize(my&
. . : in direction) do
accepts change requests from the four sen- 18 dir := mydir,
sors and arbitrates the traffic-light 19 end initiahze;
changes. Once the controller changes the 20 loop
light in one direction (east-west or south- 21 accept car-comes;
north) to green, it maintains the green sig- 22 if (lights(dir) /= green) then
nal for five seconds so other cars in the 23 controller.notify (dir);
24 end if,
same direction may pass the intersection
25 end loop;
without stopping. Before the green light in 26 end sensor-task;
any direction becomes red, it should re- 27 task body controlleris
tnain in yellow for one second so any car in 28 begin
the intersection may clear. T h e light then 29 loop
turns to red while the light in the opposite 30 accept notify (dir : in direction) do
direction turns green. 31 case du is
Figure 18 shows a sample Ada imple- 32 wheneast I west=>
mentation of these requirements. Because 33 lights := (green,green, red, red); delay 5.0;
of the asymmetric nature of the Ada ren- 34 lights := (yellow,yellow, red, red);delay 1.O;
35 lights := (red, red, green, green);
dezvous (the called task does not know the whensouth I north=>
36
identity of the calling task), an initializa- 37 lights := (red, red, green, green);delay 5.0;
tion (lines 17 through 19 and 45 through 38 lights := (red, red, yellow, yellow);delay 1.O;
47) is needed to assign a direction to each 39 lights := (green, green, red, red);
sensor.This direction is passed to the con- 40 end case;
troller when each sensor requests the con- 41 end notify;
troller to change the lights. When a car 42 end loop;
approaching the intersection is detected, 43 end controller;
the sensor for the corresponding direction 44 begin
executes line 2 1. T h e actual passing of the 45 for dir in -..north loop
46 sensor(dir).initialize(dir);
car through the intersection is assumed to
47 end loop;
begin when the program execution passes 48 end traffic,
line 24 of the sensor task.

analyzing the event where two cars travel-

ing from the north and east of the intersec- 19 shows. We have chosen to examine the In this example, we have decided to
tion are in the intersection simultaneously. case where the north car enters the inter- explore (again, among several possibili-
T h e analysis proceeds by finding the section as the east car enters. After select- ties) the case where the Sensodeast) task is
causes of the event and their relationships. ing the event to be analyzed, the next step in rendezvous w t h the controller task and
There could be many ways that two in the analysis is to find recursively the the Sensor(north) task bypassed the ren-
cars traveling north and east could be in possible subevents leadmg to the failure dezvous point (a null Else). Thls implies
the intersection simultaneously, as Figure event. that the signal was green when the car

IEEE SOFTWARE
 I

cor enters intersection

~

os tontroller
thonges light

from the north approached the intersec-

tion, and it therefore entered the intersec-
Sensor (E) tion without stopping. The tree indicates
in rendezvous that the top event could happen if and only
if both subevents occur.

A d y z i q the sensor tasks. To determine

Body of When E 1 W how the Sensor(north) task bypasses the
touses foilure ~ ~ rendezvous point, it is necessary to trace
the program backward from line 24, as
Figure 20 shows. Because the immediately
preceding statement is the If statement
Previous (lines 22 through 24), the If template is
statements
attached to the fault tree. The statement
touses touse failure
failure template offers the analyst suggestions on

02 stotements
how the specificstatement might cause the
fault. Because you know that the Sen-
sor(north) task bypasses the rendezvous
with the controller (the Then part) and
that the Else part does not have any state-
ments, you can ilnmediately terminate the
analysis along those two branches. T h e
diamond symhol (“undeveloped event”)
indicates this.
The refinement of the leftmost branch
is straighttoward. For the task to hypass
the rendezvous, Lights(north) must be
green. If the north sensor task checks this
as the controller task changes Lights at the
request of the east sensor m k , a race con-
dition will occur. % determine if this race
condition is possible, it is necessary to ex-
amine the behavior of the east sensor task
, in the rendezvous with the controller.
We analyzed the subevent “Sen-
sor(east) task in rendezvous” in a similar
way, as Figure 2 1 shows, using the rendez-
vous template. Arnong the three possible
~ causes, two can hc discarded iinniediately,
since examining the code indicates that
there is no task abortion and that parame-
ter evaluation (line 2 3 ) is not a direct cause
of the event. Whde the former decision is
obvious, the latter one represents a deci-
sion made by the analyst.
The only branch left to es-plore further
is the branch where the rendezvous body
(line 33 through 35) causes the top event.
T h e specification allows one second, dur-
ing which the lights are yellow, for any car
remaining in the intersection to clear.
Therefore, line 34 is excluded as a cause of
the top event. By similar logic, the Delay
statement in line 3 3 is also excluded.
Another failure mode potentially caus-
ing the hazard is that one second is insuffi-
cient for the cars to clear the intersection.
Body of When
Although this is a specification error, not a /=green Then
programming error, the analysis makes of direction causes failure
causes failure
aborted
t h ~ stype of critical specification assump-
tion apparent so you can check its reason-
ableness through other means. w I

This leaves the statement

lights := (green, grcen, red, rcd);
on line 3 3 to be analyzed, using the assign-
ment template (Figure 2 ) to analyze the
statement. Two branches i n the template
are not applicable here. The third branch,
“change in value causes failure,” tnust be
expanded further. According. to the speci-
fication, no light change is allowed froin
green to red without yellow occurring in
between. However, if the preceding ren-
dezvous was with the east or west sensor
tasks. line 35 would have set the state ofthe
lights to (red, red, green, green). In this
case, the assignment on line 33 would
cause a change from green to red with no
intervening yellow light.
This software fault-tree analysis den-
onstrates that the Ada code could contrib-
ute to the hazard (two cars, traveling at
right angles to one another, are in the in-
tersection at the same time) if two succes-
sive rendezvous occur with the east or west
sensor tasks and the north sensor task
checks the state of the lights immediatelv .1 1

before the second rendezvous. Figure 12 ~i~~~ 22. ncJ

fi12,!/,rmnp/ete fj!z,lf wee.
shows the complete fault tree.

COST AND PRACTICALITY

The traffic-light example is, of course,

very simple. Other strategies, such aevent
and state analyws like pem-net dnaly515,
would most likely do as M ell as software
1 fault-tree ,inalLsi? nt finding errors in t h ~ s
case. But doing a complete sute analysis
(for example, generaunga complete state- I
I
____ ~
- ~ ~~
~~~~
~ ~ ~ ~~

IEEE SOFTWARE 57
reachability graph) might be very costly at that point to detect the particular unsafe about its environment. Apath through the
and impractical for many real system, al- conditions that the analysis has shown coniponent in a software fault tree can

-
though it has the advantage of finding all could lead to the hazard. show the conditions in the environment
erroneous states, not just hazardous ones. This is particularly useful in those ap- under whch that component will e h b i t a
A general cost com- plications for which run- certain hazardous behavior. If there is no
parison between software bine software-initiated or such path through the software, that be-
fault-tree analysis and software-controlled fault- havior cannot occur.
other methods is not pos- We cannot stress tolerance or fail-safe pro- Even though reusable packages may
sible because it depends cedures are feasible: T h e themselves be hghly reliable, h s alone
very much on the applica- enough that si' software fault tree pro- does not preclude the possibility of acci-
tion. Although software
fault-tree analysis has
technique should be vides the information
necessary to determine
dents occurring when those components
are used in a particular system context, es-
been applied to relatively used COlllDk?fTlent
OS 0 which assertions and ex- pecially one for which the software was
small, industrial software ception conditions are not originally designed. Therefore, safety
at a reasonable cost, the to -nota s&,stitute most critical, what their analysis is necessary even when the reused
practicality of the use of
the technique on large-
for -other v&v content should be, and components have been successfully used
where they should be in other software systems and when they
scale software has not procedures. placed. Because runtime are considered to be highly reliable with
been demonstrated. checking is expensive in respect to their specifications.
Basically, the tech- I terms of time -and other A striking example of what can go
nique involves a structured walkthrough i resources, this infonnation is extremely wrong when reusing highly reliable soft-
with emphasis on hazardous behavior and useful &om a practical standpoint. ware involved the reuse of air-traffic con-
should have similar practicality to other
structured-walkthrough techniques. ~
The infonnation provided about criti-
cal assertions may be useful even if the
1 trol software in Britain that was originally
written and designed for air-traffic control
I

However, because the entire code is not fault tree is completed, as you can use t h ~ s centers in the US. It was not discovered
usually safety-critical, only a relatively information to catch runtinie errors in the until after the software was installed that
small percentage of the code may need to implementation or the underlymg sup- the American designers had not taken zero
be examined. port software or hardware. The analysis longitude into account, whch caused the
For example, on the scientific-satellite described in this article is concerned only computer to fold its map of Britain in two
software mentioned earlier. onlv 12 Der- I
. I " and assunies that thi 1
with the code's lopic at the Greenwich meridian.'
cent of the code needed to be examined, implementation of the underlying virtual
and the analysis was completed in two machme (the compiler) is perfect. &in, USE FOR REQUIREMENTSAND DESIGN
days. A software fault-tree analysis per- checking can be directed at those runtime
formed during the development of a nu- conditions that are the most critical. This article focuses on the application
clear power-plant shutdown system 'The information provided by the anal- of safety analysis procedures to program-
(which has about 6,000 lines of code writ- ysis may also help guide you in designing ming-language code, but you can also
ten in Fortran and Pascal) took about changes that augment safety. Software apply the same type of safety analysis to
three man-months, including becoming fault trees have been used this way to gen- requirements-modeling and specification
familiar with the technique. A full verifica- erate runtime assertions for the shutdown languages',' as well as to design represen-
tion of the same software using functional software for a nuclear power plant that has ta tions.
abstraction took 30 man-years. T h e e n p a fail-safe hardware b a c h p system and to At the requirements-specification
neers involved in tlus projea concluded reorganize the code to augment safety. stage, you can use safety analysis to iden-
that software fault-tree analysis was useful tify safety ,and fail-safe requirements, to
and easy to leam and that they would use ~MPLICAT~ONS
FOR REUSE determine conflicts between functional
it spin? This might not, however, be the and safety requirements, and to analyze a
result on a larger project. Software h l t - t r e e analysis also has set of software requirements for safety in
Another factor that must be considered inmortant iindications in the reuse of the context of the Svstem in which the
in determining practicality is that the soft- cokponents a i d packages. computer is embedded.
ware fault tree need not be completed en- Accidents often arise &om problems in Used at the highest level of design,
tirely to be useful. For example, in many the interfaces between system compo- safety analysis pinpoints the safety-critical
cases, it is possible to stop generating nents. hi interface is comprised of the as- components of the design and aids in the
branches ofthe tree before reachinga co~i- surnptions the components make about incorporation of special safety features
 minimizes the later code-verification pro- , anlined, the niorc likely t h a t errors will be
cess. At lower levels ofdesigi, it lets some found.
of the verification be accomplished 1)efore Software fault-tree analysis also starts
the detailed code is actually produced and, ti-om a separate specification (the systeni
again, may result in simplifylng the final fault tree) and therefore can find errors in
certification process. the so~are-require in en^ specification.
Furthermore, the proccss of working

B ecause fault-tree analysis was origi-

nally used to a n a l y i safety
~ in electro- ,
mechanical systems, the technique has the
backward through the software and out
into the environment allows identification
ofthe most critical assuinptions about the
advantage of being able to link the soft- environment.
ware system and the controlled system at But do not expect too much from the
their interfaces, letting the system be ‘ma- technique: T h e analysis is very human-
lyzed as a whole. You can use it at various oriented, and its success will depend on the
levels and stages of software development. analyst’s abilities. \?‘e arc developing tools
T h e basic technique of software fault-tree to aid in the analysis, hut indusmal users of
analysis has been applied to assembly-Ian- ~ the technique have commented that the
guage programs” and other simple se- , actual process of having people examine
quential programs written in Fortran and the code thoroughly was crucial to its ef-
Pascal-like languages, and it is now being fectiveness. Therefore, completely auto-
used by engineers on real industrial proj- i mating the generation ofthe trees may not
ects. This article extends the analysis tech- 1 be a worthwhile goal.
nique to include iiiore complex, multi- ~ Finally, we cannot stress enough that
tasking language constructs and deiiion- the technique should he used as a comple-
strates its use. ment to - not a substitute for - other
T h e success of software fault-tree anal- , types of verification and validation proce-
ysis appears to be related to the fact that ’ dures.
the analyst is forced toview the program in Software fiault-tree analysis provides
a different way than is common during exnx assurance (when this is required) by
development, and this increases the focusing on hazards, by forcing a different
chance for finding errors. Programmers view ofthe software, and by starting from
during development tend to concentrate a different specification. Also, it usually
on whatthey want the program to do; soft- costs much less than a complete static
ware fault-tree analysis requires consider- analysis ofthe code. But it is no less error-
ation of what the program must not do. prone than any other static-analysis tech-
T h e more different ways a prograi-n is ex- nique. +

Stephen S. C h a is a m e n - TimothyJ. S h e d is w
ber ofthe technical staffin assistant professor of Coni-
1 Iughcs .hrcraft‘s Ground puter Science at the Naval
Systems Group. His re- Postgraduate School. His re-
search mterests are soli- search interests arc s o h a r e
ware wfeg and fault toler- testing and safety.
ance. Shimeall received a
Cha receiwd a PhD in PhD in information and
inforinanon and computer computer science korn the
science from dit. Uiiivmity University of California a t
1 computer science from the Unijersity i1fCalifomi.i at of(:alifoniici at Ininc. Ininc. IIc is a member of
~1 Los Angeles. the IEEE Cmiputer Society.

md (iiniputcr Science I k p t . , University of(:alihirn~a. Inine, C A 927 1; Internet leveson@iin.uci.edu

Address questions aliout this amcle to Lcvewii ‘it Infiirin.in~~n