Tracing Your Way to

Better Embedded Software

Techniques to isolate and debug critical software issues.

your office or lab does not fully exercise system. Running dozens or hundreds of
by Jay Gould
Product Manager your embedded product the same way as “use cases” may seem powerful, but may
Xilinx, Inc. when it is fully assembled and deployed in create a false sense of security if you don’t
jay.gould@xilinx.com real-time conditions. Add other modules actually track metrics like code “coverage,”
from other engineers and your simple unit where you pay more attention to exercising
Steve Veneman tests may still pass for hours at a time. all of the code instead of repeatedly testing
Product Manager Often bugs will not reveal themselves until a smaller subset.
Wind River Systems you run the code for much longer periods Standard software debuggers are a useful
steven.veneman@windriver.com of time or with more complex interactions and much-needed tool in the embedded
with other code modules. development process, especially in the early
Have you ever worked on an embedded So how do you know if you are really stages of starting, stepping, and stopping
processing project where your team invest- testing all your code? Is it possible that a your way though brand-new code units.
ed an exorbitant amount of test time only colleague’s software may be overwriting a Once you have significant code applications
to find out at the end of the project that variable or memory address utilized in developed, testing the system as a whole
your product had a series of intermittent your module? becomes much more representative of an
and hard-to-find bugs? You used your soft- The more complex your embedded end-user product, with all of the interac-
ware debugger throughout the develop- products become, the more sophisticated tions of a real-time embedded system.
ment process and spent man-weeks your development and debugging tools However, even a good software debugger
stepping through the code in your lab, yet must scale. Often “test time” is equated may not improve your chances of finding
when your software passed integration test- with “good testing” practices: the longer an intermittent, seldom-occurring bug.
ing you were stunned to find that your QA you run tests, the better you must be exer- Serious code interaction – made more com-
team (or worse yet, a customer) discovered cising the code. But this is often mislead- plex with asynchronous interrupts of real-
serious system flaws. ing for a number of reasons. Stubbing world embedded devices – mandates the
Some embedded bugs are harder to find code and testing a module in a stand- use of proper tools to observe the entire sys-
than others and require advanced detection alone method will miss many of the inter- tem, creating a more robust validation envi-
methods. Testing small units of code in action mistakes of the rest of the final ronment in which to find the sneakier bugs.

70 Xcell Journal Copyright © 2006 Xilinx, Inc. All rights reserved. XILINX, the Xilinx logo, and other designated brands included herein are trademarks of Xilinx, Inc. All other trademarks are the property of their respective owners. Third Quarter 2006
Trace Your Code Execution lems are not easily found by post-mortem information is next accessed.
Once your code goes awry, locks up your debugging through register and memory With advanced tools like Wind River
system, core dumps, or does something accesses. Embedded systems commonly Trace, you can capture the actual instruc-
seriously unexpected, it is time to improve crash when a function is overwriting a vari- tion and data information that was execut-
the visibility of exactly what your software able that should not be accessed/written, or ed on the target before the event occurred.
was doing before the flaw. Often the symp- when memory is corrupted by executable Using this information, you can back up
tom of the failure is not clearly tied to the code being overwritten by another routine. through the executed code and identify
cause at all. Some debuggers or target tools It is difficult to diagnose these types of what events happened before the crash,
running on the embedded device will die problems when no information exists on identifying the root cause of the problem
with a fatal system flaw, so they cannot be what actions led up to an event, and the (see Figure 2). In addition to a chronologi-
used to track that flaw. In these cases, you crash of the system may not actually take cally accurate instruction sequence, when
can use a separate and external tool with its place until much later when the corrupted used on a PowerPC™ embedded platform
own connections and hardware memory to like Xilinx® Virtex™-II Pro or Virtex-4
provide an excellent view of system execu- devices, the Wind River Trace execution
tion all the way up to a complete failure. information also includes specific time-
“Trace” tools, often associated with JTAG stamp metrics. Coincidentally, the Wind
probes or ICEs (in-circuit emulators), can River ICE actually implements Xilinx
display an abundance of useful informa- Virtex-II Pro FPGA devices in the probe
tion for looking at software instructions hardware itself.
executed, lines of code executed, and even
performance profiling. Coverage Analysis and Performance Profiling
Capturing software execution with a Because the trace tool and probe are able to
trace probe allows more debugging capabil- provide a history of the software execution,
ity than just examining a section of code. they can tell you exactly where you have
An intelligent trace tool provides a wide been – and what code you have missed.
variety of trace and event filters, as well as Figure 1 – The Wind River ICE JTAG You miss 100% of the bugs that you don’t
trigger and capture conditions. One of the probe implements Virtex FPGAs. look for, so collecting coverage analysis
most useful situations for trace techniques
is to identify defects that cause target crash-
es on a random, unpredictable basis.
Connecting a probe (Figure 1) like
Wind River ICE with the integrated
Wind River Trace tool introduces a
method by which you can configure trace
to run until the target crashes. By trigger-
ing on “no trigger,” the trace will contin-
ue to capture in a circular buffer of
memory in the probe until the system
crashes. You may have unsuccessfully tried
to manually find a randomly occurring
defect with your software debugger, but
with a smart trace tool, you can now run
the full battery of system tests and walk
away to let the trace run. When that inter-
mittent event finally occurs again, crash-
ing the target, the tool will capture and
upload the data for your examination.
The execution trace will show a deep
history of software instructions executed
on your system leading up to the fatal flaw,
providing much more insight into what the
system was doing and allowing you to find Figure 2 – Wind River Trace software execution view
and fix the bug faster. Often these prob-

Third Quarter 2006 Xcell Journal 71

metrics is important to your testing strate- space, higher than normal software testing built into probes and trace tools allow you
gy. As code execution is almost invisible standards may be mandated. Certainly the to look at the data in a performance view in
without an accurate trace tool, it is com- software running the ABS brakes in your addition to the go/no-go code coverage
mon for entire blocks or modules of code car, a medical device, or an aircraft control view. By identifying bottlenecks in software
to go unexecuted during test routines or system should be tested to a higher degree execution, you can identify routines that
redundant user-case suites. Testing the than a software game or other non-life- fail to meet critical timing specifications or
same function 10 times over may make threatening application. In these life-criti- optimize routines for better performance.
sense in many cases, but missing a different cal applications code must be executed and With the time-stamp information in
Virtex-II Pro and Virtex-4 devices, you can
also use trace tools to determine how long a
software function or a particular section of
code took to run.
Whether you are trying to find rou-
tines that could use some optimization
recoding (perhaps in-lining code blocks
rather than accruing overhead by repeat-
edly calling a separate function) or identi-
fying a timing flaw in a time-critical
routine, performance-metric tools are
required to collect multiple measure-
ments. Settling for a couple passes of a
routine with a manual debugger is not
sufficient for validating timing specs on
important code blocks. Additionally, if a
real-time system is running an embedded
operating system, verifying the capability
of the interrupt service routines (ISRs) is
essential to determine how quickly the
system responds to interrupts and how
much time it spends handling them.
Figure 3 – Wind River profiling and coverage views
Conclusion
If critical performance measurements or
function altogether means that you will not tested (covered) down into every function code coverage are required before your
find those bugs in your lab – your customer of software instructions, and that execution embedded product can ship, then you need
will find them in the field. may need to be documented to a formal, to used specialized tools to capture and log
Coverage metrics showing which func- higher level governing body. appropriate metrics. Accurate system trace
tions are not executed are useful for writing In many cases code coverage can also be metrics are a must if you have ever been
new, additional tests or identifying unused used to analyze errant behavior and the challenged by a seemingly random, hard-
“dead” code, perhaps leftover from a previ- unexpected execution of specific branches to-reproduce bug that your customers seem
ous legacy project. Because many designs or paths. Erratic performance, failure to to experience regularly but that you cannot
are re-ports or updates of previous prod- complete a task, or not executing a routine duplicate in your office.
ucts, many old software routines become in a repeatable fashion may be just as inap- Utilizing an accurate hardware probe
unnecessary in a new design, but the lega- propriate in some applications as a fatal like Wind River Trace with a powerful
cy code may be overlooked and not crash or other defect. By tracing function trace/coverage/performance tool suite will
removed. In applications where code size is entries and exits, comprehensive data is provide you with the data you require.
critical, removing dead code reduces both collected on software operations and you Stop guessing about what is really going
waste as well as risk; nobody wants to wan- can throw away the highlighter and pro- on and gain new visibility into your
der into an obsolete software routine that gram listing (Figure 3). embedded applications.
may not even interact properly with a Because performance is always a con- For more information about Wind
newer version of hardware. cern in modern embedded applications, River Trace and other products, visit
In safety-critical applications like avion- having an accurate way to measure execu- www.windriver.com/products/development_
ics, medical, automotive, or defense/aero- tion times is paramount. The capabilities suite/wind_river_trace/.

72 Xcell Journal Third Quarter 2006