Software Engineering Technology

The Qualification of Software Development

Tools From the DO-178B Certification Perspective
Dr. Andrew J. Kornecki Dr. Janusz Zalewski
Embry Riddle Aeronautical University Florida Gulf Coast University

Software development tools are in wide use among safety-critical system developers. Examples of such use include aviation, auto-
motive, space, nuclear, railroad, medical, and military applications. However, verification of tool output to ensure safety, man-
dated in highly regulated industries, requires enormous effort. If a tool is qualified, this effort can be reduced or even eliminat-
ed. The Radio Technical Commission for Aeronautics Document Order-178B and related documents provide guidelines by which
to qualify these tools. However, current regulations, business models, and industry practice make this goal difficult to accomplish.
This article discusses the qualification of development tools and the potential impact of this process on the aviation industry.

S oftware development tools are com-

puter programs that help developers
create other programs. Such tools have
(SC-167) in 1989 with the objective of
updating the DO-178A by focusing on
five areas: documentation integration and
context of a specific airborne system.” It is
the certification authority that decides on
the outcome of the qualification process.
been in use since the early days of com- production, system issues, software devel- Moreover, qualification, if claimed, is a
puting to improve the efficiency of the opment, software verification, and soft- requirement in getting a system certified.
development process by automating mun- ware configuration management and soft-
dane translation operations and bringing ware quality assurance. The resulting doc- Types of Software
the level of abstraction closer to the appli- ument, DO-178B, provides guidelines for
cation engineer. Nowadays, development applicants developing software-intensive
Development Tools
DO-178B differentiates between verifica-
tools are used in a variety of safety-critical airborne systems [1, 2]. It discusses objec- tion tools that cannot introduce errors but may
applications, including the aviation, auto- tives that need to be met to show that the fail to detect them and development tools
motive, space, nuclear, railroad, medical, software development process provides whose output is part of airborne software and
and military industries, and contribute to specified levels of safety assurance. It also thus can introduce errors. There is a signifi-
the risks associated with using respective describes the processes and means of cant amount of effort involved to qualify
products. Despite these risks to society, compliance. a verification tool, and much more to
development tools are rarely qualified in a Systems are categorized by DO-178B qualify a development tool. However,
sense comparable to product certification as meeting safety assurance levels A numerous development tools have been
in regulated industries. The objective of through E based on their criticality in sup- used successfully in many certified pro-
this article is to look at the current state of porting safe aircraft flight. The level A sys- jects without being qualified. To define a
the tool qualification process, identify the tem is the most critical: The failure of subject matter more narrowly, we need to
issues, and propose recommendations for such a system could result in a catastroph- take a closer look at the entire domain of
potential improvement, focusing on the ic failure condition for the aircraft. The software development tools.
aviation industry. level E system is the least critical: Such a The landscape of modern software
system has no effect on the operational development tools is very broad, as illus-
System Certification Versus capability of the aircraft or pilot workload. trated in Figure 1 (see page 20). Following
Although the RTCA DO-178B is the lead- the traditional model of the development
ing source of guidelines for software
Software Tool Qualification
Certification of airborne equipment is typ- process from requirements to implemen-
ically achieved through the Federal developers engaged in such system con- tation, we can identify the following:
Aviation Administration (FAA) authoriza- struction, two other documents have criti- • The requirements category that includes
tion of a type certificate (the entire air- cal bearing on the subject. RTCA DO- tools used early in the life cycle to
craft), supplemental type certificate (new 248B [3] clarifies some of the misinterpre- identify and specify the software
equipment in a specific aircraft), or a tech- tation of the DO-178B. The FAA Order requirements.
nical standard order (minimum perfor- 8110.49 compiles a variety of guidelines • The design category that includes tools
mance standard for materials, parts, and related to the use of software in airborne allowing developers to create architec-
appliances used on civil aircraft). A special systems. Chapter 9 is specifically dedicated tural and detailed design of the soft-
committee (SC-145) of the Radio Tech- to tool qualification [4]. ware in a notation of their choice sup-
nical Commission for Aeronautics (RTCA) A key component of the updated ver- ported by the tool; often in this cate-
convened in 1980 to establish guidelines sion of DO-178B is the concept of tool gory, tools translate the model to
for developing airborne systems. The qualification elaborated in Section 12. source code.
report “Software Considerations in Air- Qualification is a supplementary process • The implementation category that
borne Systems and Equipment Certifica- that the applicant may elect to follow in includes all support required to trans-
tion” was published in January 1982 as the the course of certifying an airborne sys- late the computer code and transfer it
RTCA Document Order (DO)-178 (and tem. According to the definition given in to the target computer.
revised as DO-178A in 1985). DO-178B, tool qualification is defined as, As illustrated in Figure 1, three other
Due to rapid advances in technology, “The process necessary to obtain certifica- categories of tools can be identified: those
the RTCA established a new committee tion credit for a software tool within the related to analysis, testing, and target.

April 2006 www.stsc.hill.af.mil 19

Software Engineering Technology

a Tool Configuration Management Index,

Tool Development Data, Tool Verification
e.g.:

Records, Tool Quality Assurance Records,

Structural Rhapsody Typically With
Code Generator
Design RoseRT

Tool Configuration Management Records,

STOOD Functionality
Requirements Tool

etc. These requirements are also described

Artisan
Integrated
Tool

in [4]. Tool qualification data are approved

e.g.: Development
or/and SCADE
Functional Environment

only in the context of the overall software

e.g.:
Reqtify Matlab (IDE)
Design

development for the specific system

DOORS BEACON e.g.:
Tool Sildex Tornado

where the intention to use the tool is stat-

SpecTRM
DOME Multi

Analysis Implementation ed in the PSAC. The tool itself does not

receive a separate qualification stamp of
Tool Tool

approval. Therefore, using the tool on

another system/project requires a separate
e.g.: Testing
RapidRMA Tool
qualification, although some qualification
TimeWiz

credits may be reused.

e.g.: Target

Surveys requesting which tools are used

CodeTest
TestRT
(with RTOS)

by industry were conducted at two national

VectorCast e.g.:

conferences: the 2002 FAA National

Insure++ VxWorks
QNX

Software Conference and the 2004 Embry

OSE
Tool Categories
Riddle Aeronautical University/FAA Soft-
Integrity
LynxOS

Figure 1: Software Tool Categories ware Tool Forum. In addition, two follow-
up e-mail solicitations were sent to more
However,
Figure for this article, we
1:Software Tool will focus pri- producer) is to transform an input artifact than 500 professionals working on air-
Categories
marily on the tools used in the design into output, thus creating another software borne systems. These surveys and solicita-
phase, the central component of the soft- artifact. The current process mandates ver- tions resulted in a relatively small sample of
ware development life cycle. They reflect ification after each transformation. If this responses that did not provide a base for
two diverse viewpoints on real-time, safety- transformation has an impact on the final statistically significant results. The com-
critical systems development, which result airborne product, NOthe producer needs to be ments included industry discouragement
Can

from different developers’ backgrounds: qualified, but only if the transformation regarding the rigor of development tool
tool insert error
into airborne

• Control engineers consider a system to output would not be verified and the trans- qualification, and a justified perception of
software? NO QUALIFICATION

YES model consisting of well- formation leads to elimination, reduction, the extensive cost of qualification.
be a dynamic
NECESSARY
defined blocks of specific functionality or automation of any of the DO-178B Potential solutions to assist in commer-
(logic, arithmetic,Will dynamic). The func- processes. The conditions under which a cial off-the-shelf (COTS) development
tional paradigm the model is the development tool requires qualification are tool qualification included extensive ven-
of output
basis for system verified and analysis presented in Figure 2 [4].
simulation
the tool’s

dor collaboration and using alternate means

of its behavior. Subsequently, the model Software development tool qualifica-
NOT be
TOOL MUST
allowed in DO-178B. The limited feedback
(as specified in
can be translated automatically into an tion is attempted only as an integral com-
NO BE QUALIFIED
shows that there has been interest in qual-
DO-178B)?

equivalent code, typically without any ponent of a specific application program

ifying software development tools classi-
additional developer’s involvement. requiring the FAA’s certification. The soft-
fied in the function-based/block-oriented
YES
• Software engineers, on the other hand, ware tools to be used are referenced with-
category, which cannot be said about
Are

are familiar with theDO-178B of oper- in the Plan for Software Aspects of
conceptseliminated,
processes of

ating systems, programming automated Certification (PSAC) and the Software structure-based/object-oriented tools.
reduced, orlanguages,
Accomplishment Summary documents of A short list of qualified development
software development methodologies,
tools includes code generators (Gener-
NO
and notations. The graphic notations the original certification project. If devel-
by use of

ation Automatique de Logiciel Avionique,

tool?

(classes, packages, states, transitions, opment tool qualification is required, the

Graphical Processing Utility, Virtual
YES
events)2:Conditions
allow developersWhen to represent applicant should present Tool for review the
Application Prototyping System Code
Figure a Software Development Requires Qualification
the structure and behavior of the target Tool Operational Requirements (TOR) – a
system software as a set of components document describing tool functionality, Generator, Safety Critical Avionic
that can be translated into programming environment, installation, operation man- Development Environment Qualifiable
constructs (data structures, objects, ual, development process, and expected Code Generator) and configuration-
functions, etc.) using the automatic code responses (also in abnormal conditions). scheduling table generators (Universal
generation functionality of the tool. Two documents must be submitted and Table Builder Tool, Configuration Table
Consequently, the software design approved: a Tool Qualification Plan, and a Generation Tool), most of them being in-
tools, which assist developers in translat- Tool Accomplishment Summary as de- house products. According to several
ing the software requirements into source scribed in [4]. To make an argument for informal exchanges with industry, many of
code, can be categorized into two groups: qualification, the applicant must demon- the modern COTS software development
(a) a function-based, block-oriented approach strate correctness, consistency, and com- suites actually have been used in the cre-
applied by control and system engineers, pleteness of the TOR and show that the ation of software artifacts on certified
and (b) a structure-based, object-oriented tool complies with its TOR. This demon- projects without going through the quali-
approach applied by computer scientists stration may involve a trial period during fication process.
and software engineers. which a verification of the tool output is
performed and tool-related problems are Problems With Development
The Qualification Process analyzed, recorded, and corrected. Tool Qualification
A typical use for a design tool (software Other data required for review include It is clear that qualification of develop-

20 CROSSTALK The Journal of Defense Software Engineering April 2006

 The Qualification of Software Development Tools From the DO-178B Certification Perspective

ment tools is an option rarely exercised in nificant difference between tool software ed to the translation process. The transla-
the airborne software industry. In fact, and application software. Applications run tion component is hidden deep inside the
one could argue that qualification of on a target computer while tools operate tool, which causes problems with tool
development tools is not a viable option. on a general-purpose workstation, typical- qualification. Typically, there is no access
Current interpretation of applicable ly closely interacting with a COTS operat- to a COTS tool’s life-cycle data, which
guidelines makes development tool quali- ing system and conventional programming describe the tool’s requirements, design,
fication a proposition that is not practical environment. Considering this, several and code. Unless the tool has been devel-
from a managerial viewpoint, and not easy DO-178B objectives are not applicable to oped in-house, the qualification efforts
from a technical viewpoint. tool software and thus cannot be met. may be doomed.
There is also no general agreement on
what metrics would allow developers to Potential Solutions
The first group of problems is of a regu- carry an independent tool assessment [6]. The qualification of a stand-alone devel-
Managerial Viewpoint

latory and managerial nature. The major One often-repeated statement regard- opment tool is not feasible in the strict
hurdle is the current state of regulations ing development tool qualification is the sense of existing guidelines. Such concepts
and guidelines. The secondary obstacle is requirement that “only deterministic tools as component-based software, software
the business model and lack of incentives, can be qualified.” The DO-178B refers to reuse, and service history should be
in particular the prohibitive cost of tool determinism as “… tools which produce explored [7] to identify the feasibility of
qualification. The existing tools, often the same output for the same input data such qualification. The issues of tool ver-
used in certification projects, do not have when operating in the same environment.” sion control and the precise definition of
appropriate data to support arguments The definition does not take into account operational environment, constraints, and
about meeting the objectives of DO- how the output is generated. By this defin- e.g.:limitations are the basis for starting discus-
178B. The applicant team’s intent is to cer- ition, one may interpret thatStructural sion about solutions
it is notRhapsody to tool qualification.
tify the product rather than expand effort required to provide proof on the internalRoseRTThe availability of extensive tool software
Typically With
Code Generator

and qualify the tool. The tool vendor does behavior of a tool. An example ofTool development data, often scarce for COTS
this can Artisan
Design Functionality
STOOD

not see the business advantage of qualify- be memory use for a tool running
or/and on the
products, may be a challenge to ever
Requirements
Integrated

ing a tool while disclosing proprietary host workstation multi- accomplish

in a multitasking, Functional COTS tool qualification [8].
Tool e.g.: Development

information to potential competitors. user, networked environment. The prob- It could be conceivable to e.g.: create an
e.g.: SCADE Environment
Reqtify Matlab (IDE)

define what the object code for a independent lab dedicated to tool qualifi-
Design
Development tool qualification lem is to
DOORS BEACON
Tool

requires close collaboration between the tool is. Does it include the operating system cation and encourage commercial vendors
SpecTRM Sildex Tornado
DOME Multi

tool vendor and the applicant. This is the (OS) of the host workstation? A tool clear- to submit their product for assessment. A
reason why in-house tools are more likely ly needs to make explicit calls to the OS similar approach is known Toolfrom other
Analysis Implementation

to be qualified. Internal trade studies [5] areas of verification and validation [9, 10].
Tool
routines, and any verification of these
have shown that the cost of development would require full visibilityRapidRMA
of the host’s OS Another idea would be to require certified
e.g.: Testing

tool qualification is significantly higher and related high assurance of its operation. product e.g.:applicants to disclose information
Tool
TimeWiz

than the cost of verification tool qualifica- The main function of a software regarding CodeTestthe development tool use and
Target

tion. The use of qualified verification development tool is to transform, i.e., qualification VectorCast effort by creating an FAA-
TestRT
(with RTOS)

tools can result in fast savings on the first translate an input artifact into output. This sponsored Insure++ database for DO-178B certi-
e.g.:

program where they are introduced. In is why the qualification, if applicable, fied products. This
QNX could face serious
VxWorks

contrast, the use of qualified development should be focused on this Categoriesobjections from industry
Tooltranslation OSE
due to an appre-
tools may require several programs to component of the tool functionality. hensiveness to disclose any information,
Integrity

make up the cost. However, modern, complex software which may result in the loss of commer-
LynxOS

The intellectual property rights may development tools provide a variety of cial advantage. It would be possible to
need to be waived by the vendor to achieve other functions
Figure that are notTool
1:Software directly relat- research a potential for development tool
Categories
qualification. The tool cannot be qualified
as standalone, but only within the scope of Figure 2: Conditions When a Software Development Tool Requires Qualification
a particular certification project. The tools
that could be considered for qualification Can
are very simple: typically in-house created
tool insert error

utilities where the applicant holds all intel-

into airborne NO

lectual property rights, maintains all tool

software? NO QUALIFICATION

development data, and can reuse the tool

NECESSARY

software artifacts on consecutive projects.

YES

The qualification is accomplished within

the specific certification project and thus is
Will
the tool’s output

not clearly visible from the outside as devel-

NOT be verified

opment tool qualification.

TOOL MUST
(as specified in NO
DO-178B)? BE QUALIFIED

The second group of problems is related

Technical Viewpoint YES
to technical aspects. According to the DO-
Are

178B interpretation, the development tool

processes of
DO-178B eliminated,
needs to be qualified to the same level of
reduced, or automated

scrutiny as the appropriate application it is

by use of NO

helping to develop. However, there is a sig-

tool?
YES
Figure 2:Conditions When a Software Development Tool Requires Qua
April 2006 www.stsc.hill.af.mil 21
Software Engineering Technology

qualification using an approach different Washington, D.C.: RTCA, 10 Dec. 7. Lougee, H. “DO-178B Certified Soft-
than the one outlined in Section 12.2 of 2001 <www.rtca.org/downloads/List ware: A Formal Reuse Analysis Ap-
DO-178B. Service history and formal OfAvailableDocsAPR%202005.htm# proach.” CrossTalk Jan. 2005
methods could both be potential options. _Toc101071717>. <www.stsc.hill.af.mil/crosstalk/2005/
It appears that the industry has a 4. Federal Aviation Administration. 01/0501lougee.html>.
pressing need to come up with methods to “Software Approval Guidelines.” FAA 8. Zalewski, J., W. Ehrenberger, F.
audit a tool that is independent of the spe- Order 8110.49. Washington, D.C.: Saglietti, J. Gorski, and A. Kornecki.
cific program and applications using it. FAA, 2003 (Chapter 9 replaces FAA “Safety of Computer Control Systems:
This would require updating the guidelines Notice N8110.91 of 2001) <www.air Challenges and Results in Software
to consider a model-driven development web.faa.gov/Regulatory_and_Guid Development.” Annual Reviews in Con-
paradigm, redefine the qualification ance_Library/rgOrders.nsf/0/640711 trol 27.1 (2003): 23-37.
process, and allow flexibility regarding B7B75DD3D486256D3C006F034F? 9. Brosgol, B.M. “ADA in the 21st
qualification to be less dependent on the OpenDocument&Highlight=8110.49>. Century.” CrossTalk Mar. 2001
application program using the tool. A 5. Potter, Bill. “Use of the MathWorks <www.stsc.hill.af.mil/crosstalk/2001/
more streamlined method to qualify devel- Tool Suite to Develop DO-178B Cer- 03/brosgol.html>.
opment tools and to keep them current as tified Code.” Slide No. 13. Honeywell, 10. Adams, M., et al. “Conformance
technology advances would be useful. May 2004 <http://faculty.erau.edu/ Testing of VMEbus and Multibus II
Better guidance on how to apply service korn/ToolForum/potter.htm>. Products.” Advanced Multi-Micro-
history and how to address what has to be 6. Kornecki A., and J. Zalewski. Criteria processor Bus Architectures. Ed. J.
done for incremental tool changes would for Software Tools Evaluation in the Zalewski. Los Alamitos, CA.: IEEE
also be needed. These and other issues Development of Safety-Critical Real- Computer Society Press, 1995: 392-399.
have been discussed at the recent Tools Time Systems. Proc. of PSAM-7/ Euro- 11. Embry Riddle Aeronautical Universi-
Forum [11]. The RTCA convened another pean Safety and Reliability Conference, ty/FAA Software Tools Forum,
special committee (SC-205) with a charge Berlin, Germany, 14-18 June 2004. Embry Riddle Aeronautical University,
to recommend modifications to the exist- London: Springer-Verlag, 2004 <http:// Daytona Beach, FL., May 18-19, 2004
ing DO-178B. The qualification of soft- f a c u l t y. e r a u . e d u / ko r n / p a p e r s / <www.erau.edu/db/campus/software
ware tools is being discussed and some ESREL04KorneckiZalewski.pdf>. toolsforum.html>.
changes may be forthcoming.◆

Acknowledgement About the Authors

The presented work was supported in part
by the Aviation Airworthiness Center of Andrew J. Kornecki, Janusz Zalewski, Ph.D.,
Excellence under contract DTFA-0301 Ph.D., is a professor at is a professor of comput-
C00048 sponsored by the FAA. Findings the Department of Com- er science at Florida Gulf
contained herein are not necessarily those
puter and Software Engi- Coast University. Prior to
of the FAA. Additional support was
received from the Florida Space Grant neering, Embry Riddle this, he worked for various
Consortium under Grant No. UCF01- Aeronautical University. nuclear research institu-
E000029751. He has more than 20 years of research tions, including the Data Acquisition
and teaching experience in areas of real- Group of Superconducting Super Collider
References time computer systems. Kornecki con- and Computer Safety and Reliability
1. Radio Technical Commission for tributed to research on intelligent simula- Center at Lawrence Livermore National
Aeronautics, Inc. “RTCA DO-178B, tion training systems, safety-critical soft- Laboratory. He also worked on projects
Software Considerations in Airborne ware systems, and served as a visiting and consulted for a number of private
Systems and Equipment Certifica- researcher with the Federal Aviation companies, including Lockheed Martin,
tion.” Advisory Circular. Washington, Administration (FAA). He has been con- Harris, and Boeing. Zalewski served as a
D.C.: RTCA, 1 Dec. 1992 <www.rtca.
ducting industrial training on real-time, chairman of the International Federation
org/downloads/ListOfAvailableDocs
APR%202005.htm#_Toc101071800>. safety-critical software in medical and avi- for Information Processing Working
2. Federal Aviation Administration. ation industries and for the FAA Certi- Group 5.4 on Industrial Software Quality,
“RTCA Inc., Document RTCA/DO- fication Services. Recently, he has been and of an International Federation of
178B.” Advisory Circular No. 20-115B. engaged in work on certification issues Automatic Control Technical Committee
Washington, D.C.: U.S. Department of and assessment of development tools for on Safety of Computer Control Systems.
Transportation, Nov. 1993 <www.air real-time, safety-critical systems. His major research interests include safety-
web.faa.gov/Regulatory_and_Guid related, real-time computer systems.
ance_Library/rgAdvisoryCirc
ular.nsf/0/DCDB1D2031B19791862
Dept. of Computer and

569AE007833E7?OpenDocument>.
Software Engineering Dept. of Computer Science

3. Radio Technical Commission for

Embry Riddle Aeronautical University Florida Gulf Coast University

Aeronautics, Inc. “RTCA DO-248B,

600 Clyde Morris BLVD 10501 FGCU BLVD

Final Report for Clarification of DO-

Daytona Beach, FL 32114 Fort Myers, FL 33965
178B ‘Software Considerations in
Phone: (386) 226-6888 Phone: (239) 590-7317
Airborne Systems and Equipment Fax: (386) 226-6678 Fax: (239) 590-7330
Certification’.” Advisory Circular. E-mail: kornecka@erau.edu E-mail: zalewski@fgcu.edu

22 CROSSTALK The Journal of Defense Software Engineering April 2006