Static Analysis Lesson Learnned
Static Analysis Lesson Learnned
Lessons Learned©
Andy German
QinetiQ Ltd.
The United Kingdom Ministry of Defense has been in the forefront of the use of software static code analysis methodologies,
including some of the tools and their application. This article1 discusses what is meant by static analysis, reviews some of the
tools, and considers some of the lessons learned from the practical application of software static code analysis when used to
evaluate military avionics software.
Civil Airworthiness Authorities sug- It Costs Too Much This may result in diagrammatic and
gest that the use of formal methods be Based on project experience, an average graphical representations of the code
considered [8]. 10 percent of a military aircraft’s software being produced.
The application of static code analysis – or approximately 500,000 software lines
techniques in retrospect is not ideal; the of code – are found safety critical. The Data Flow Analysis
process is best suited and cheapest when average cost of retrospective independent The objective of data flow analysis is to
applied during software development. analysis for an aircraft is less than $13 mil- show that no execution paths in the soft-
Following the UK as low as reasonably practi- lion, and on average less than 0.4 percent ware exist that would access a variable not
cal approach [9] to risk, the retrospective of the total development cost for an air- set to a value. Tools use the results of
evaluation of safety-critical code is the craft. These costs can be further reduced control flow analysis in conjunction with
only reasonable method available at pres- if the semantic analysis element is direct- read/write access to variables. It can be a
ent to reduce safety-critical anomalies to a ed. It has been shown that the most cost- complex activity, as global variables can
minimum – after all other mitigations have ly element of static analysis is the seman- be accessed from anywhere. This analysis
been considered. tic element when comparing costs of the can also detect other code anomalies such
activity to total percentage of anomalies as multiple writes without intervening
Static Analysis Myths found. One important area for future reads.
But We Test It research is the justifiable targeting of
All software contains errors, and comput- techniques. Information Flow Analysis
er programs rarely work the first time [10]. If, however, the software is designed Information flow analysis identifies how
Usually, several rounds of rewriting, test- and analyzed as part of the development execution of a unit of code creates
ing, and modifying are required before a process, then the cost savings are likely dependencies between the inputs to and
working solution is produced [11]. Testing when compared to normal industry costs. outputs from that code. These dependen-
usually involves running and evaluating There are also considerable through-life cies can then be verified against the
the software across its expected range of cost savings and system reliability benefits. dependencies in the specification. This
operation. This process is limited by the analysis is often particularly appropriate
tester’s ability to predict this range of Dissimilar Systems for a critical output that can be traced all
operation, or rather, the range of inputs Although Defense Standard 00-55 [7] the way back to the inputs of the hard-
that the program will receive. This is how allows the use of dissimilar systems to be ware/software interface.
it is possible for large well tested software combined to create a safety-critical sys- Information flow analysis may be aug-
packages to still fail periodically: The user tem, this becomes a very difficult mented in some tools by using annotations.
has done something not anticipated by the approach to argue as being safe. The fol- These are stylized comments that docu-
tester. It would be nice to test every state lowing issues need to be addressed: ment certain assumptions about func-
of a program, but such exhaustive testing • The comparison software or liveware tions, variables, parameters, and types.
is impractical as it would take far too much (pilots) becomes safety critical (70 per- They enable an analysis to proceed more
time and expense. cent of aircraft accidents are due to efficiently because they give it more infor-
An argument often used against static aircrew error). mation relevant to a particular block of
analysis is that “our software has been • How do you prove dissimilarity? code.
extensively tested.” This argument does • Reliability goes down, as the lower
not stand up. Radio Technical Commis- integrity systems are likely to disagree Path Function Analysis (Also Called
sion for Aeronautics, Inc. (RTCA) and fail more often. Semantic Analysis or Symbolic Execution)
Defense Order (DO)-178B [8] Level A • The warning system becomes more Path function analysis is used to verify
requires extensive modified condition/ critical. properties of a program by algebraic
decision coverage testing while RTCA • The cost of ownership goes up (sup- manipulation of the source text, without
DO-178B Level B does not require this porting multiple equipment, increase requiring a formal specification. It
level of testing. When Level A was com- aircraft weight, etc.). involves checking the semantics of each
pared to Level B, no significant difference • Designers tend to make the same mis- path through a program section or proce-
in anomaly rates identified by static analy- takes. dure. Sophisticated tools give expressions
sis was found. Unhappily, the hack-it-and- for the precise mathematical relationship
bash-it methodology is still prevalent Main Static Analysis Techniques between inputs and outputs from a partic-
among many software developers. and Methods ular program section: They effectively give
Control Flow Analysis (Including the transfer function for that program sec-
Static Analysis Means It Is Safe Cyclomatic Complexity) tion [12]. They step through the code,
The phrase “static analysis means it’s safe” Control flow analysis can be conducted assigning expressions instead of values to
is heard quite often. Static analysis only using tools or done manually at various each variable. Thus the sequential logic is
allows us to argue that the code is as fol- levels of abstraction (module, node, etc.) converted into a set of parallel assign-
lows: and is done for the following reasons: ments in which output values are
• As compliant with the software • Ensure the code is executed in the expressed in terms of input values – this
requirements as present evaluation right sequence. format is easier to interpret. The tools
methods and technology allows. • Ensure the code is well structured. produce an output for each path consist-
• That coding errors have been minimized. • Locate any syntactically unreachable ing of the conditions that cause the path
Static analysis does not prove that the code. to be executed, and the result of executing
requirements the code was developed • Highlight the parts of the code (e.g., that path.
from were correct or show that the com- loops) where termination needs to be Semantic analysis reveals exactly what
piled code is correct. considered. the code does in all circumstances for the
whole range of input variables for each reduce the number of likely coding viola- Other Memory Usage Analysis
program section. However, there is still tions and improve code readability. It is This is required for any resource that is
the need for substantial human involve- normal to define a rule set when designing shared between different partitions of
ment in comparing the tool’s output with safety-critical code to allow the tools to software. It reveals the absence of conflict
the specification. Compliance analysis carry out the analysis more readily and to between the code and other low-level
(formal code verification) provides a remove some of the more problematic components such as device drivers and
reduction in human requirements and features. It has been found that the size of resource managers.
greater automation. the rule sets is dependent on the language,
such as the following: Object Code Analysis
Formal Verification • C has some 220 rules suggested [13]. Object code analysis demonstrates that
(Also Called Compliance Analysis) • Ada 83 has approximately 80 rules. the object code is an accurate translation
This is the process of proving, in an auto- • Southampton Program Analysis and of the source code and that the compiler
mated process and as far as is possible, Development Environment (SPADE) has introduced no errors. The analysis
that the program code is correct with Ada Kernel (SPARK Ada) has an may be carried out by manual inspection
respect to the formal specification of its extensively defined rule set; sometimes of the machine code produced by the
requirements. All possible program execu- a reduction in the rules can be agreed compiler – this can be made easier if the
tions are explored, which is not feasible by on. compiler vendor provides details of the
dynamic testing alone. Depending on the mappings from source code to object
power of the tool being used, and its sim- Range Checking code.
plification ability, the involvement of ana- Range checking analysis aims to verify
lysts may be large or small. that the data values remain within their Limitations
Verification conditions can enhance specified ranges, as well as maintain spec- Although the various forms of static code
compliance analysis. They consist of con- ified accuracy. This technique can detect analysis offer many advantages to the sys-
ditions that should be valid at the start and the following: tem developer, they also impose some
end of a block of code (pre- and post- constraints. Using these techniques
conditions) and are stated at the start of restricts language choices that may be used
that block. In a way, it is like a different
form in which the programmer can
“All software contains and the choice of the structures used
within these languages. Furthermore,
explain the purpose of a block of code. errors, and computer these analytical methods require highly
The analysis might start with the post- skilled and experienced staff to carry out
condition and work backward to the start programs rarely work the tests and analyze the results. It is not a
of the block. If, on reaching the start, the complete answer for the validation and
pre-condition is generated, then the block the first time. Usually, verification of safety-critical software
of code is provably sound. even with the use of automated tools.
Compliance analysis effectively per- several rounds of Other forms of testing (for example
forms a proof of the code against a low- dynamic) are required to verify certain
level mathematical specification. In this rewriting, testing, and aspects, like executing critical features.
respect, it is by far the most rigorous of Some of the restrictions of static analysis
the static analysis techniques. However, its modifying are required using automated tools are the following:
value depends on the availability of a • Multitask applications software must
specification expressed in a suitable form.
before a working be analyzed a task at a time. Another
Furthermore, this rigor is at the expense form of testing is required to check
of cost; productivity is around five lines
solution is produced.” task interactions.
of code per man-day. • Dynamic aspects of the software (for
• Overflow and underflow analysis. example sequences of program execu-
Independent Evaluation • Rounding errors. tion) are difficult to model with static
Since the 1970s, independent code inspec- • Array bounds. analysis techniques.
tions to reduce code error have been • Stack usage analysis. • Most automated tools require transla-
found to be efficient and cost effective. This is a form of shared resource tion to an intermediate language before
Experience from aircraft static code analy- analysis that establishes the maximum they can analyze the code. Automatic
sis carried out to date shows that code required size of the stack, and whether translators are available for some lan-
walkthrough finds about 60 percent of all there is sufficient physical memory to sup- guages, but for others one must either
the anomalies found. port it. translate manually or write a new trans-
lator. Some language features do not
Other Techniques Timing Analysis have an equivalent in the intermediate
Syntax Checks Timing analysis ascertains the temporal language even with the automatic
Syntax checks aim to find language rules properties of the input/output dependen- translators; they must be manually
violations such as using a variable of the cies, including the worst-case execution translated. The static analysis of the
wrong type or before it is declared. The time for the correct behavior of the over- software depends on its translation
compilers of some languages such as Ada all system. It can be made difficult by lan- model and the more skilled the analyst,
and Pascal will carry out syntax checks guage features such as manipulation of the more skilled the model produced.
automatically, whereas languages like C dynamic data structures, loops without The validation of the intermediate lan-
and assembler need additional tools. static upper bounds, and by using hard- guage model needs to be considered, as
To allow the use of analysis tools, ware with built-in pipe and cache. this can be a major problem.
Air Vehicle Software Analysis ments in mind. More commonly, com- attractive (but little used) language such
The practical application of static code mercial factors such as productivity and as Modula-2 might not be chosen for a
analysis has produced some interesting ease of use steer the development. safety-critical application.
results. The range of software systems Some languages are better suited to
that have been subjected to analysis the production of safety-critical software Tools Available
include the following: than others because they make it easier to There are a number of static code analy-
• Automatic flight control. write dependable code, and easier to sis tools available. They offer different
• Engine control. demonstrate its freedom from errors. depths of analysis, and some will only
• Fuel and center-of-gravity manage- However, you must bear in mind that the operate on a few languages. Most of
ment. language itself is a product that is sus- them run on uncompiled source code
• Warning systems. ceptible to design flaws: Perfect code and first translate to an intermediate lan-
• Anti-icing systems. could still produce errors when run. guage, which the analysis tool itself can
• Flight management. The software lines of code per anom- read.
• Stores management. aly in Table 1 show some of the metrics The time taken for the tool to analyze
• Air data units. found from various programs. Table 1 the code may be only a small fraction of
• Radio altimeters. shows that the poorest language for safe- the time taken to carry out static analysis
• Anti-skid brakes. ty-critical applications is C with consis- of the code. Many tools produce reams
These systems vary in size from 3,000 tently high anomaly rates. The best lan- of data that must be laboriously analyzed
lines of code to 300,000 lines of code guage found is SPARK (Ada), which con- and processed; staff requires skill and a
and include languages from assembler, C, sistently achieves one anomaly per 250 lot of training.
Pascal, Ada to Lucol, and SPARK Ada software lines of code. The average num-
[14]. ber of safety-critical anomalies found is a Main Tools
small percentage of the overall anomalies There are three main, well-established
Effects of Previous Development found with about 1 percent identified as tools used on UK military programs.
Methodologies (RTCA DO-178B) [8] having safety implications. Automatically
It is worth reiterating that when compar- generated code was found to have con- Malvern Program Analysis Suite
ing RTCA DO-178B [8] Levels A and B siderably reduced syntactic and data flow Malvern Program Analysis Suite
code, no discernible difference was found errors. (MALPAS) was developed by Royal
by static code analysis demonstrating that Software development is often per- Signals and Radar Establishment
static code analysis is something you formed on a different system than that Malvern based on research they carried
carry out in addition to testing. Even the used for the final application. Therefore, out and by Southampton University in
most extensive testing does not remove the portability of the code is another fac- the 1970s. It is now mature and since
the anomalies found by static code analy- tor to be considered when choosing a 1986 has been supplied and supported by
sis. Surprising amounts of dead code have language. That is, how easily it will run in Advantage.
been found in code developed to RTCA a different environment to the one in Although automatic translators exist
DO-178B Levels A and B. which it was developed. for most languages, the main ones cov-
The quality of the available compilers ered are Ada and Pascal. There is no con-
Effects of Language is also important. Modern programming cept of pointers in the MALPAS
The choice of language for a computer languages are very complex and sophisti- Intermediate Language and so to analyze
program is important. Not only should cated and hence difficult to understand. C, for example, the code would first have
the functionality of the language itself be It is therefore challenging to write high to be purged of the use of pointers – a
considered, but also the availability and quality, dependable compilers for them potentially formidable task.
quality of support tools and the expertise [15]. Widely used compilers and develop-
within the development team. Unfor- ment tools should be used whenever SPARK
tunately, safety-critical software repre- possible, so that there has been plenty of SPARK is a subset of Ada for high
sents only a small subset of the global opportunity for errors to be found (and integrity programming first formalized
programming effort; most languages are hopefully rectified). This also applies to by Bernard Carré and Trevor Jennings of
not designed with high integrity require- the language used, reflecting why an Southampton University in 1988. It has
continually evolved and nowadays it is
Table 1: Software Language Anomaly Rates being more widely used and is gaining
general acceptance particularly as its
Software Range Software Anomalies Per
Language Lines of Code Thousand Lines
tools now run within a lunchtime for an
Per Anomaly of Code average-sized safety-critical avionics pro-
C Worst 2 500 gram. In addition, SPARK now supports
Average 6 - 38 167 - 26 tagged types, tasking (Ada95 Ravenscar),
Best (Auto Code Generated) 80 12.5 and proof of exception freedom, which have
Pascal Worst 6 167 particular benefits in the context of
Average/Best 20 50 RTCA DO-178B [8].
PLM Average 50 20 Control flow analysis is not needed as
Ada Worst 20 50 it is subsumed into the SPARK grammar,
Average 40 25 and thus performed on the fly. Data and
Best (Auto Code Generated) 210 4.8 information flow requirements have
Lucol Average 80 12.5 been expressed as SPARK program
SPARK Average 250 4
design rules.