See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/352059377

Penetration Testing for Automotive Cybersecurity

Article in ATZelectronics worldwide · June 2021

DOI: 10.1007/s38314-021-0629-4

CITATIONS READS

4 1,223

2 authors:

Christof Ebert Ruschil Ray

Vector Consulting Services Universität Stuttgart
332 PUBLICATIONS 6,162 CITATIONS 6 PUBLICATIONS 54 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Christof Ebert on 19 August 2022.

The user has requested enhancement of the downloaded file.

 Grey-Box Pen-Testing for Automotive Cybersecurity

Christof Ebert, Ruschil Ray, Florian Kanis, Vector

Automotive vehicles are safety critical computers on wheels. The ongoing convergence of product IT
and enterprise IT creates high cybersecurity demands. With their immediate safety impact,
cyberattacks on such systems will endanger passengers. The automotive industry is using various
methods for security verification and validation. Yet, we realize that with classic security testing,
vulnerability detection is inefficient and incomplete. In this article, we show how an enhanced Grey-
Box Penetration Test (GBPT) needs less test cases while being more effective in terms of coverage
and indicating less false positives.

1. Penetration Testing
Penetration Testing (or pen testing in short) is particularly helpful to identify and mitigate
cybersecurity attacks which had not been considered during design. Current standardization with ISO
21434 and UNECE R.155 on CSMS and R.156 on SUMS enforces systematic risk mitigation
throughout the entire life cycle [1,2,3,4,5]. For security testing, they all devise classic testing
overlayed with dedicated security testing.
Typically, Penetration Testing also known as ethical hacking is performed in the validation stage of
the security and safety V model. To use it effectively, we will in this article show how to connect Pen
Tests with the left side of the V. The various stages are as follows: Scoping, information gathering
and reconnaissance, threat modelling, vulnerability analysis, exploitation, and reporting. Pen testers
need to think like a hacker and ethically penetrate the system to exploit and expose the vulnerabilities,
to allow mitigating these issues.
Pen testing is of three main types namely White-Box, classic Black-Box, and innovative Grey-Box
Pen Testing (GBPT). White-Box Pen testing approach is a full knowledge test where all source code
and all internal data of the system are available to the tester. Although this approach brings the most
results, it takes a lot of time and effort. Black-Box Pen Testing (BBPT) also called as Classic Pen
Test is a no knowledge test where the tester knows only the publicly available documents and
standards. Brute forcing is applied in this test. It also takes time and effort due to the demand to cover
a huge number of permutations of signals and scenarios.
Since hackers have a lot of architecture knowledge, software, and vulnerability information, they
attack the system by various paths. GBPT helps to make the system more robust to cyberattacks. In
GBPT, basic high-level architecture and result from Threat Analysis and Risk Assessment (TARA),
which is performed in the requirement stage as security analysis, is provided as input to perform the
GBPT. This approach takes less time and effort and is most advantageous than the classic BBPT in
the current scenario.
Classic BBPT, however, neither has a good traceability mechanism with respect to the functional
requirements nor does it consider the high-level architecture knowledge to ensure all vulnerabilities
and attack paths are covered. It has the drawback of not addressing the highest risk vulnerabilities
first. This traditional black-box penetration testing mechanism requires a lot of time and effort;
therefore, this calls for a revised testing scheme that is both effective and efficient.
 Grey-Box Pen-Testing for Automotive Cybersecurity 2

We show in this AutoTest presentation a new greybox pen testing method and provide practical
examples from Vector security test labs.

2. Ten Steps for Grey-Box Penetration Testing

The enhanced TARA-based Grey-Box Penetration Testing Method includes 10 steps and proves to
be more beneficial than classic Black-Box Pen Test in terms of efficiency and coverage. This testing
strategy includes a good traceability mechanism and a risk-oriented approach to ensure all
vulnerabilities are covered with least amount of effort and time. It is based on experiences with classic
(Black-Box) Pen Test and enhances it with knowledge on assets, threats, and architecture.
Specifically, we use the Triple Peak model to build and maintain the underlying traceability [6]. To
avoid creating a blind spot as is the case with mere white-box methods, we keep the external black-
box perspective, thus calling it “grey-box”. Fig. 1 depicts the 10 steps of the enhanced TARA-based
GBPT method along the entire lifecycle.

Fig. 1: Ten steps for Grey-Box Penetration Testing

The Grey-Box Pen Test can be divided into 10 steps for ease of implementation:
1) Extract system information. This involves basic features, user experience, but also negative
requirements (e.g., Misuse, Abuse und Confuse Cases). Features are deduced which is the starting
point of the formal traceability model.
2) Model basic architecture and interfaces. Determine the components of the product. These
components should be vertically traced to circuit level components for top-down recursive pen test.
3) Identify relevant assets. Use safety, operational performance, privacy, and legislation. Trace
features to an asset ID to ensure exhaustive coverage.
4) Analyze hardware and software specifications. Many specs for hardware, protocols, IP stacks etc.
are available online, and hackers will use those too. Use public vulnerabilities databases like Common
Vulnerabilities Exposure (CVE) to identify weakness.
 Grey-Box Pen-Testing for Automotive Cybersecurity 3

5) Perform a Threat Analysis and Risk Assessment (TARA). Deduce damage scenarios by failing the
system test cases that are associated with the requirements via forward trace links towards attack
vectors.
6) Create a minimum viable pen test case repository. The attack vectors are traced via their IDs to
ensure a good test case coverage and generate a minimum and non-redundant pen test case repository.
7) Perform actual pen testing. Use the minimum viable test cases and report the valid and invalid
vulnerabilities.
8) Measure performance with KPIs. The number of valid vulnerabilities and the minimum viable test
cases are used to compute the KPIs for effectiveness and efficiency of the GBPT.
9) Distil further functional security requirements. Use the Triple Peak method to systematically
identify necessary security protection [6].
10) Perform Regression Testing. Each change demands regression testing which is most efficient
when using the traceability model which is created in the initial steps of GBPT.
Due to size restrictions in this article, this description is rather short and cookbook-style. For a more
detailed explanation of the underlying methods, we recommend our related webinars and articles [6].

3. Case Study: Infotainment and Ethernet

With ethernet, software and configuration updates can be distributed faster. Due to technologies like
VLAN and network switches, a stricter separation of concerns (i.e., separate functional domains,
separate functions within one functional domain) can be achieved using ethernet compared to bus
technologies like CAN. Using switches in combination with the IEEE 802.1Q specification of VLAN
and Quality of Service (QoS), as well as firewall concepts, communication paths inside the network
can be restricted and priorities for packets are considered.
Ethernet evolutionary introduction builds upon a heterogeneous legacy system which will stepwise
evolve towards ethernet connectivity. Each distributed communication technology (e.g., CAN, LIN,
and Ethernet) was designed for a specified purpose. They provide certain features and has furthermore
limitations. It is thus challenging to apply a mutual security mechanism to the system, e.g., message
authentication codes, due to differences in computing resources of ECUs and payload sizes of
protocol frames such as CAN (8 Byte), CAN-FD (flexible data rate, 64 Byte), FlexRay (254 Byte)
and Ethernet (flexible, 1500 Byte).
To scope this case study, we look to a simplified mixed communication infrastructure as it can be
found in many current vehicle designs [7]. Fig. 2 provides the architecture which exhibits several
subnets for connectivity, body controllers, driver assistance and infotainment. The scope of initial
ethernet security is between the switches in the connectivity gateway and the different legacy domains
(red box in fig. 2).
Modeling is applied to achieve a good understanding of the underlying architecture and identifying a
minimum viable set of suitable test cases [8]. As an attacker we only use for such modeling publicly
available information, no insights from proprietary designs.
 Grey-Box Pen-Testing for Automotive Cybersecurity 4

Fig. 2 Simplified architecture for basic connectivity

On this basis we will apply a TARA (Threat Analysis and Risk Assessment). Our Vector
SecurityCheck and underlying security engineering methods have adopted the state of the practice in
security evaluation and proposed mitigation. Error! Reference source not found. It is using
significant research work from our worldwide security projects. It also uses external best practices,
such as EVITA, HEAVENS, and other proposed methods for security risk assessment in automotive
development [1,2].
Within this scope, the simplified use case scenarios comprise of:
 Regular usage of car
 Car diagnostics
 After-sales enhancements
First, we analyze the assets. The following assets are present in the given architecture:
 Switching function of switches
 Filter function of switches
 Filter and forward function of firewall
 Cross-domain communication over switches and firewall
 Normal operation of domains
 Privacy-related data and intellectual property (IP)
Specific threats are derived in the scope of ethernet switch and connectivity:
 Degenerate ECU performance
 Accessing personal data, such as contacts, routes, etc.
 Loss of data
 Display wrong warnings to driver
 Clone and introduce counterfeit parts by using valid part numbers
 Chip tuning to improve ECU performance
 Steal manufacturer’s IP
 Installing non-authorized software to break safety of the vehicle
 Disturb and distract driver (situation-specific)
Several attack vectors are identified in the scope of ethernet switch and connectivity:
 Grey-Box Pen-Testing for Automotive Cybersecurity 5

 Denial of Service (DoS) attacks, inhibit communication

 Examine vulnerabilities
 Spoofing on CAN
 Eavesdropping of personal data
 Remotely controlling functions
 Manipulate the Central Gateway Module (CGM) routing tables, change filter policies
 Replay old manifests, messages
 Execute compromised software updates
 Obtain keys to sign compromised software updates as “authentic”
 Replace public keys
 Compromise or delete logs
 Disrupt critical information from infotainment to other ECUs
We are using the Vector COMPASS tool to document the threat and attack vectors and thus trace,
prioritize and document the pen testing (fig.3).

Fig. 3: Sample threats after definition of assets and attack vectors

Given the attack vectors and reference architecture, we can complete the prerequisites of the GBPT
with most common anticipated attacks, such as DoS, MAC Flooding, MAC Spoofing, eavesdropping,
VLAN hopping, and inject malware.
Based on the repeated GBPT, we can map suitable security mechanisms to mitigate critical security
risks, such as Firewall, VLAN, network segregation, IDS, Secure Boot, Secure Logging, Secure
Communication (e.g., MACs, MACSec), Encryption. The presented Security Goals are further
broken down to security requirements to enable implementation and testing. Suitable security
mechanisms shall be selected to implement security goals.
To institutionalize a viable regression strategy, we perform with each increment, a subset of the GBPT
and with each delivery, a SecurityCheck with residual risk analysis based on a given selected set of
design rules as part of an updated safety case. Fig. 4 provides an example from our case study.
 Grey-Box Pen-Testing for Automotive Cybersecurity 6

Fig. 4: Residual risk analysis of automotive ethernet introduction with KPI from Vector
SecurityCheck

4. The Benefits of Grey-Box Pen Test

Grey-Box Pen Test helps to create a more accurate attack tree with all possible attack vectors which
covers all the scenarios, unlike in classic Black-Box Pen Test where the attack tree might consist of
failed paths to the threat and might miss out on some important paths to the threat since the tester
doesn’t have complete knowledge of the schematics of the system. The black-box pen tester can miss
out different architectural configurations. These various configurations and pathways are crucial to
the pen tester to rule out all possibilities of compromising the system.
Compared to the GBPT approach, a black-box pen tester merely knows the information from the
available documents and has no internal architecture knowledge of the system. Brute forcing is
applied in such a test strategy, which leads to a lot of steps to be performed to exploit the system,
thereby leading to more time and effort.
Moreover, the attack vectors would not be prioritized based on the risk levels hence could lead to
missing of the most critical one due to limited resources available or due to failed test cases.
Therefore, a classic Black-Box Pen Test results in many test cases and this increased number of test
cases leads to additional effort, time, resource usage and cost.
The KPI chosen for effectiveness and efficiency are the number of true positive vulnerabilities found
and number of test cases respectively. On comparing the two strategies based on metric for efficiency
and effectiveness, we find that the GBPT leads to a smaller number of test cases thereby reducing the
test cost and increasing the efficiency. The defect detection rate is also higher as compared to classic
Pen Test hence the effectiveness or coverage is also increased. In direct "competition" situations, we
were able to identify more effective threats with GBPT in about half the time of BBPT.
Fig. 5 depicts the difference between GBPT and Classic BBPT with respect to criteria like time and
effort, attack tree accuracy, number of false positive vulnerabilities, steps in test case, number of
vulnerabilities, resource consumption and traceability. Our analysis from practical case studies shows
a reduced number of test cases and a higher hit rate for TARA-based GBPT, we infer that the risk-
oriented GBPT is more effective and efficient test strategy than classic BBPT method.
 Grey-Box Pen-Testing for Automotive Cybersecurity 7

Criteria Grey-Box Pen Test (GBPT) Classic Black-Box Pen Test (BBPT)
Time and Time and effort are saved in the initial More time and effort spent performing pen
Effort stages of pen test like scoping, information test as knowledge of the internal
gathering and reconnaissance as knowledge architectural schematics is not known
of architecture and TARA is known
Attack tree Attack tree has a large test coverage with Attack tree contains many branches where
accuracy attack vectors aligning to the architecture some might even result in failed test cases
Number of Lesser cases of false positive vulnerabilities More false positive vulnerabilities as
false positive which leads to efficient exploitation architecture is not known hence lead to
vulnerabilities more time wasted during exploitation
Steps in test The steps involved in the test cases are Starting with vulnerability scanning to
case fewer, thus leading to lesser resource retrieve the open ports with subsequent
consumption, time, effort, and cost sequential exploit trials means more steps
within a single test case
Number of More vulnerabilities are identified, that is Without knowledge of architecture, it
vulnerabilities because it is aware of the architecture and could miss some of the crucial
hence covers all scenarios vulnerabilities as this testing approach is
not systematic and lacks traceability
Resource Resources are consumed in a prioritized Inefficient resource utilization due to
Consumption manner as the highest risk is addressed first. random utilization with high repetition in
brute-forcing
Traceability This results in a good traceability Since this strategy uses brute forcing, a
mechanism to link all functional good traceability mechanism cannot be put
requirements with the assets, then assets into place thereby leading to inefficient
with the threats and attack vectors and these and incomplete testing.
attack vectors are associated with test cases,
hence yielding a systematic and exhaustive
test strategy.
Fig. 5: Evaluating Grey-Box Pen Test vs. Classic Black-Box Pen Test

Effective cybersecurity demands systematic processes along the life cycle of both components and
the car itself, especially if their effectiveness must be proven at a later point due to legal actions. With
this article we strive to provide guidance for specific misuse cases and how to validate accordingly.
Our Grey-Box Pen Test (GBPT) has proven more efficient and effective than traditional validation
schemes. The mentality of engineers must change towards designing and validating for security –
rather than only for functionality. Enterprise IT has realized many years ago that isolated mechanisms,
such as distributed functionality in proprietary subsystems, protection on component-level, gateways
and firewalls between components, validation of critical functions, etc. are insufficient. It demands
intelligent testing with the perspective of an attacker. Think like a criminal and act as an engineer. Or
in the words of Henry Ford: “Who always does what he is already able to do, always remains what
he already is.” This would be too high a risk in automotive IT.
 Grey-Box Pen-Testing for Automotive Cybersecurity 8

References
[1] Kim, S. and R. Shrestha: Automotive Cyber Security. Springer, 2020.
[2] ISO/SAE 21434. Road vehicles — Cybersecurity engineering.
https://www.iso.org/standard/70918.html
[3] UNECE: R.155 Cybersecurity Management System (CSMS).
https://unece.org/transport/documents/2021/03/standards/un-regulation-no-155-cyber-security-
and-cyber-security
[4] UNECE: R.156 Software Update Management System (SUMS).
https://unece.org/transport/documents/2021/03/standards/un-regulation-no-156-software-update-
and-software-update
[5] Ebert, C. and J.John: “Practical Cybersecurity with ISO 21434”. Springer. atz Elektronik. ISSN:
0001-2785, pp. 16-21, No. 2, Mrc/Apr. 2022.
[6] Ebert, C. and Ray, R.: Test-Driven Requirements Engineering. IEEE Software, ISSN: 0740-
7459, vol. 38, no. 1, pp. 16-24, Jan. 2021.
[7] Brennich, T. and Moser, M.: Putting Automotive Security to the Test. ATZelectronics
worldwide, ISSN: 2524-8804, vol.15, no.1, pp. 46-51, Jan. 2020.
[8] Ebert, C. and R. Ray: “Toward a Formal Traceability Model for Efficient Security Validation”.
IEEE Computer, pp. 68-78, vol. 54, ISSN: 0018-9162, Nov. 2021.

Authors
Christof Ebert is the managing director of Vector Consulting Services and a professor at the
University of Stuttgart and the Sorbonne in Paris. Contact him at christof.ebert@vector.com.

Ruschil Ray is a consultant with Vector Consulting Services. Prior to that, she worked for General
Motors. Contact her at ruschil.ray@vector.com

Florian Kanis is a senior consultant with Vector Consulting Services. Prior to that, he worked for in
AUTOSAR development. Contact him at florian.kanis@vector.com

View publication stats