People powered, tech-enabled cyber security

Safety, Security,
Privacy & Prompts:
Cyber Resilience in
the Age of Artificial
Intelligence (AI)
2 | Cyber Resilience In The Age of AI

Contents

Foreword.............................................................................................................................................................. 3
Executive Summary............................................................................................................................................ 4
Overview............................................................................................................................................................... 5
AI Use Cases & Impact....................................................................................................................................... 6
Positive Use Cases.................................................................................................................................................. 6
Negative Use Cases ................................................................................................................................................ 7
Cyber security Use Cases................................................................................................................................... 9
Defensive Use Cases .............................................................................................................................................. 9
Offensive Use Cases 12
Threats to AI & ML Systems..............................................................................................................................15
Threats 15
Practical Attacks 16
Safety..................................................................................................................................................................19
Regulation, Legislation & Ethics......................................................................................................................20
Privacy 20
Ethics 21
Regulation and Legislation - Overview 22
Regulation and Legislation - In Depth Country Profiles 24
Considerations for Governments and Regulators 31
Conclusions........................................................................................................................................................32
About the Authors..............................................................................................................................................34
Acknowledgements and NCC Group Overview 37
Further Reading..................................................................................................................................................38
About Us..............................................................................................................................................................41

www.nccgroup.com
 3 | Cyber Resilience In The Age of AI

Foreword

“
The trajectory of AI development has been nothing short of meteoric
in recent years; permeating every sector and systematically changing
business operations and decision-making processes. As with any swift
technological advancement, new challenges and threats arise. An array
of cyber security threats and vulnerabilities have already surfaced in
this domain, many of which are only partially understood. Organisations
and policy makers have the taxing task of harnessing the transformative
potential of AI, while at the same time needing to grapple with the
ever-evolving threat landscape that it presents.

Ensuring the safety and security of people, process and technology

in an AI-augmented world demands vigilance, and a commitment to
forward-thinking strategies that will require continuous adaptation.
This whitepaper seeks to introduce the topic of AI’s reinvention of
“ cyber security, setting a baseline of understanding for some of the key
AI concepts, threats and opportunities to business decision makers
and policy makers, to support their thinking and strategies in this
fast-paced, exciting new technological era.

Siân John
Siân John
Chief Technology Officer, NCC Group

www.nccgroup.com
4 | Cyber Resilience In The Age of AI

Executive Summary

The security of Artificial Intelligence (AI) systems is Risk management processes and policies must be
an ever-evolving field, with state-of-the-art continually adapted and updated to ensure that employees are
evolving at pace as AI is applied to a wider range aware of their responsibilities in the face of these newly
of sectors and application domains. AI provides available and tempting tools, and that the new risks
opportunities to both adversaries and defenders. It posed by widespread adoption of AI are understood,
additionally introduces new risks to business processes communicated and appropriately mitigated.
and data security, and to safety when used in cyber-
physical systems such as autonomous vehicles. The NCC Group has released this whitepaper to assist
exponential adoption and impact of AI across all those wishing to better understand how AI applies
sectors and technologies is pushing AI further up to cyber security. The paper provides high-level
global regulatory agendas, seeking to gain a handle on summaries of how AI can be used by both cyber
the security, safety and ethics challenges associated professionals and adversaries, the risks AI systems are
with AI use. exposed to, safety, privacy and ethics concerns and
how the regulatory landscape is evolving to meet these
challenges. For the interested reader we have also
These new challenges must
included sections on AI terminology and technologies
be considered when designing and a summary of the research NCC Group has
a security strategy for the published in this space. We hope this information will
be useful to security professionals and senior leaders
development or use of AI, seeking an introduction to the field of AI and cyber
ensuring that users can reap security, helping to understand the emerging risks
and threats in this domain and how they might affect
the benefits that AI brings organisations. We hope the paper is also useful to
whilst managing risk to developers looking to understand cyber security u
se cases for AI.
acceptable levels.

www.nccgroup.com
 5 | Cyber Resilience In The Age of AI

Overview

2023 looks set to be the year of AI. The public release However, these new capabilities introduce new risks
by OpenAI in November 20221 of ChatGPT triggered and challenges in their safe, secure and ethical use.
a surge of interest in, and use of AI. Specifically, tools The historic concern, the subject of many science
like ChatGPT are underpinned by Large Language fiction works of art dating back to the 19th century2,
Models (LLMs) which are trained on huge sources of has been the arrival of a sentient AI, the AI singularity
data, including entire websites like Wikipedia, resulting or Artificial General Intelligence (AGI) which results
in models comprised of trillions of parameters. These in an existential threat to humankind due to its self-
LLMs can be queried by users in a conversational awareness and its intelligence surpassing that of
manner, with retained context across those humans. Despite AGI largely considered to be some
conversations – this presents very powerful chatbot way off (if even at all ever likely), the risk is recognised
capabilities that can be integrated into all manner of by many leading, current and historic, figures in
products and services. technology and AI3 4. The more pressing risks and
issues concern Intellectual Property (IP) (in terms of
Multiple competitor LLMs to ChatGPT have since what data sources are used to train AI models), privacy
been launched by big tech companies: Falcon40B (in terms of what private information can be retrieved or
available via Amazon, Llama by Meta, Bard by Google inferred from AI models), accuracy (where AI systems
and Microsoft’s own version of ChatGPT to name but a present inaccurate information or make inaccurate
few. LLMs are a type of Generative AI, which is an area decisions), economic impacts (where human
of AI concerned with generation of new content, based workers and their tasks are replaced by AI), political
on previous learning of that content type from large interference, widening inequality and increased
corpuses of data. As such Generative AI is not limited criminal and cybercrime activities.
to text-based generation; the same techniques can be
used to generate images and music for example, while In this whitepaper we provide an overview of the
the technical capabilities in these domains continue to technologies underpinning AI, describe in further
mature and produce evermore impressive results. detail the general positive and negative AI use cases &
impacts before diving into specific cyber security use
While LLMs have captured our attention in recent cases for AI. We then cover cyber threats to AI-based
times, certainly AI is a much broader topic spanning systems and present a view on the emerging global
application domains such as autonomous vehicles, AI regulatory environment and what this means for
autonomous and intelligent stock trading, facial organisations seeking to leverage AI. We conclude
recognition and surveillance and cyber security with recommendations for safe, secure and compliant
detection and response, to name but a few. The use of AI.
pervasiveness of AI presents many opportunities
to improve society in a myriad of ways.

Productivity can be improved through automation; AI

assistants can minimise the need for human presence
and interaction across client transactions, and provide
new, conversational ways of searching and querying
data, while Generative AI techniques are set to 1
https://openai.com/blog/chatgpt
revolutionise the arts and creative industries. 2
https://en.wikipedia.org/wiki/Artificial_intelligence_in_fiction
3
https://www.safe.ai/statement-on-ai-risk
4
https://futureoflife.org/open-letter/pause-giant-ai-experiments/

www.nccgroup.com
6 | Cyber Resilience In The Age of AI

AI Use Cases & Impact

Positive Use Cases

Over the past decade access to AI has commoditised Machine vision is the field of AI allowing robots and
to the point that it is already widely used across all vehicles to observe their environment, plot a course,
sectors and parts of society. avoid obstacles and carry out tasks requiring accuracy
and dexterity. A popular example of this are the
Generative AI including LLMs, image and music robots developed by Boston Dynamics8. Use cases
generators are enabling users to rapidly create text, include rescue robots for disaster areas where it is
imagery (including photorealistic and artistic images too dangerous to send a human rescue team and
and video) and music commonly based on a text-based with computer vision to locate casualties, as well as
prompt. This is improving productivity in many ways; for autonomous vehicles in freight and taxi services.
example, providing a quick first draft of some text which
the user can subsequently fine-tune. It is also allowing In addition to visual information and other physical
artists to experiment with new forms of creation. sensory inputs, Machine Learning can be trained
to evaluate digital information such as time series
Machine Learning classification systems have become data and log files. Training on large sets of known
near ubiquitous during this period and are used in good data can develop models which identify when
everyday applications such as biometric authentication a parameter has strayed out of bounds of expected
for consumer electronics (fingerprint and facial normal behaviour. This is applicable to a variety of use
recognition). They have enabled advances in science cases including detecting issues relating to security,
and medicine through the ability to rapidly process performance or systems requiring maintenance.
large datasets in areas such as medical imaging5, These early warning systems can result in significant
satellite imagery and environmental monitoring6. savings by avoiding future catastrophic failures.

Optical Character Recognition (OCR) is being used As well as detecting anomalies in past or recent
to digitise paper-based historical records stored in data, Machine Learning can be used for predicting
museums, providing new access to researchers7. It is future system behaviour. This is used extensively in
also being used in digital transformation of enterprises weather and climate modelling as well as assisting
which have traditionally relied on paper record keeping enterprises in predicting future capacity and
and processes such as in healthcare; allowing health maintenance requirements.
professionals to share patient data in a much more
effective and collaborative way and to provide as much
context as possible for future diagnoses.

5
https://www.cancer.gov/news-events/cancer-currents-blog/2022/artificial-intelligence-cancer-imaging
6
https://www.unep.org/news-and-stories/story/how-artificial-intelligence-helping-tackle-environmental-challenges
7
https://blog.nationalarchives.gov.uk/machines-reading-the-archive-handwritten-text-recognition-software/
8
https://bostondynamics.com/

www.nccgroup.com
 7 | Cyber Resilience In The Age of AI

AI Use Cases & Impact

Negative Use Cases

There is an adage, “anything that can be used, can be output portions of or recreations of that training data.
misused”, like most technologies, AI can be used in The Electronic Frontier Foundation (EFF), a prominent
adversarial ways and present deliberate or inadvertent non-profit promoting digital civil liberties, has called for
negative impacts. open access to training data to improve transparency
and fairness in AI12. Simultaneously, various
Deepfakes: generation of videos, voice and images governments have failed to comment or explicitly
which depict real people sufficiently accurately to advised that copyrighted works are not protected from
convince others they are genuine, have been used to use as training data.
create fake sexually explicit content featuring celebrities
or in targeted image-based sexual abuse. They are The increased availability of generative AI chatbots
also increasingly being used in social engineering has also led to incidents involving employees sharing
and fraud campaigns to attempt to trick individuals Intellectual Property with public models through
into scam investment schemes, authorising fraudulent prompts, and security incidents revealing other users’
transactions9 or permitting access to sensitive data chat histories. Leading LLMs are developing and
and systems. They are used in highly targeted political deploying models for enterprises which enable private
misinformation campaigns with a recent example, albeit usage and reduces the risk of information leakage into
an obvious hoax making the headlines after a fake the public domain via ongoing training.
image was generated featuring former US President
Donald Trump being arrested10. The risk of LLM “hallucinations”, where an LLM
generates text which sounds convincing but which is in
Inadvertently, models can inherit societal biases fact false or misleading, is very real and a challenge to
present in the source training data. When these both developers and users of LLMs. Widely publicised
models are then used to make automated decisions, examples of where an LLM hallucination has had a
they can lead to decisions being made based on real world impact include Google’s stocks taking a
a correlation with a protected characteristic and hit of ~9% ($100B in market value) when its LLM,
discriminate against marginalised communities11. Bard, incorrectly asserted that the James Webb Space
This results in further widening inequality in society and Telescope took the first pictures of a planet outside the
preventing equitable access to education, employment, solar system in an advertisement posted to Twitter13
healthcare and justice. in February 2023. And in May 2023 a lawyer used
answers from ChatGPT in a federal court filing
Copyrighted works or sensitive Intellectual Property which included references to numerous cases
in training data for generative models can result in which did not exist14.
models which will, when prompted in certain ways,

9
https://www.forbes.com/sites/jessedamiani/2019/09/03/a-voice-deepfake-was-used-to-scam-a-ceo-out-of-243000/
10
https://twitter.com/EliotHiggins/status/1637927681734987777?s=20
11
https://www.amnesty.ca/surveillance/racial-bias-in-facial-recognition-algorithms/
12
https://www.eff.org/deeplinks/2023/01/open-data-and-ai-black-box
13
https://www.reuters.com/technology/google-ai-chatbot-bard-offers-inaccurate-information-company-ad-2023-02-08/
14
https://www.forbes.com/sites/mattnovak/2023/05/27/lawyer-uses-chatgpt-in-federal-court-and-it-goes-horribly-wrong/

www.nccgroup.com
8 | Cyber Resilience In The Age of AI

AI Use Cases & Impact

Negative Use Cases Continued

An often overlooked impact is how the proliferation of AI impacts the environment. Training large models requires
significant amounts of electricity to run and cool the underlying computing servers and has the potential to generate
large amounts of electronic waste when large computers and associated storage media reach the end of their useful
life.

Other, more existential negative impacts which have yet to be realised include:

• The potential for runaway systemic failures, rapid and uncontrollable due to their automated nature,
in financial markets causing widespread financial hardship.

• Widening inequality, large technology platform providers, and those with the financial resources to access
them, become the only organisations with the resources to benefit from the gains delivered by AI whilst others
are excluded.

• Humans becoming overly dependent on, or are excluded from, decision making processes made by AI.
Reliance on machines for communication and decision making infantilises employees and users.

• Whole classes of employment are wiped out by machines and not replaced by suitable alternative employment.
Widespread unemployment leads to worsening social outcomes and civil unrest.

www.nccgroup.com
 9 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Defensive Use Cases

AI can be used in many ways to support cyber defenders: analysis of logs, files, network traffic, supporting secure
code development and testing, and threat intelligence, to name but a few examples.

Machine Learning (ML) has been in use in mainstream cyber security products such as eXtended Detection and
Response (XDR) to support anomaly detection for several years. Its ability to continually analyse vast quantities of
data and highlight events outside of normal parameters means it has proven effective in detecting potential cyber
events for further investigation by response teams. This has increased the efficiency and effectiveness of incident
responders by enabling them to spend less time on data analysis and more time on investigating suspicious activity,
reducing time to detection.

NCC Group Insights

NCC Group has been researching, and using in its managed service offerings,
Machine Learning for anomaly detection.

Examples of our research include:

• Machine Learning from idea to reality15: Using ML models to detect malicious PowerShell scripts.

• Incremental Machine Learning by Example16: Detecting suspicious activity with network intrusion
monitoring data streams.

• Encryption Does Not Equal Invisibility17: Detecting anomalous Transport Layer Security (TLS)
certificates.

As well as detecting anomalous behaviour in security logs, ML can be used to detect anomalies in other datasets
and applications. For example, Machine Learning models can be used in wireless networks18 to detect and isolate
rogue Wi-Fi Access Points (AP), by scanning the Radio Frequency (RF) spectrum and using other sources of
telemetry. Rogue APs at best are reducing the Wi-Fi performance for legitimate users and at worst might be
targeting users for man-in-the-middle attacks. Other data which might indicate insecure behaviour includes power
consumption, CPU and memory spikes, and network usage. In all these examples, MLs ability to analyse vast
amounts of data in real-time and highlight areas for further investigation by security staff can deliver improved cyber
defensive effectiveness.

15
https://research.nccgroup.com/2020/09/02/machine-learning-from-idea-to-reality-a-powershell-case-study/
16
https://research.nccgroup.com/2021/06/14/incremental-machine-leaning-by-example-detecting-suspicious-activity-with-zeek-data-streams-river-and-ja3-hashes/
17
https://research.nccgroup.com/2021/12/02/encryption-does-not-equal-invisibility-detecting-anomalous-tls-certificates-with-the-half-space-trees-algorithm/
18
https://www.microsoft.com/insidetrack/blog/finding-rogue-access-points-on-the-microsoft-corporate-network/

www.nccgroup.com
10 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Defensive Use Cases Continued

In some cases, the use of ML models might assist in the detection of novel zero-day attacks, enabling an automated
response to protect users from malicious files. In these instances, it might be preferable to act on the suspected
event to prevent further compromise than to log it as an event for further investigation. The impact of denying a
user access to a potentially genuine file in the event of a false positive is acceptable in use cases such as email
attachments or browser downloads (assuming manageably low false positive rates) in the face of a potential
ransomware outbreak. Appropriate actions could include sending the file for further analysis or running it in a
sandbox where its behaviour can be further analysed.

NCC Group sponsored a masters student at the University College

London’s (UCL) Centre for Doctoral Training in Data Intensive Science
(CDT DIS) to develop a classification model which attempts to
determine whether a file is malware19. Multiple models were tested
with the most performant achieving a classification accuracy of 98.9%.

Threat intelligence involves monitoring multiple online data sources providing streams of intelligence data about
newly identified vulnerabilities, developed exploits and trends and patterns in attacker behaviour. This data is
often unstructured textual data from forums, social media and the dark web collectively known as Open-Source
Intelligence (OSINT). ML models can be used to process this text, identify common cyber security nuance in the
data and therefore identify trends in attacker Tactics, Techniques and Procedures (TTP). This enables defenders
to proactively and pre-emptively implement additional monitoring or control systems if new threats are particularly
significant to their business or technology landscape.

19
https://research.nccgroup.com/2022/01/31/machine-learning-for-static-analysis-of-malware-expansion-of-research-scope/

www.nccgroup.com
 11 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Defensive Use Cases Continued

Generative AI, trained on example code, and code development assistants automatically generate code on behalf of
users based on their prompts and previously developed code. The use of generative AI for software development
has huge potential productivity gains, assuming that the code can be generated in a functional, performant and
secure way, which is not always the case. Conversely, source code can be input into a generative AI chatbot and
prompted to review whether the code contains any security weaknesses in an interactive form of static analysis,
highlighting potential vulnerabilities to developers. However, the effectiveness, or otherwise, of such approaches
using current models has been the subject of NCC Group research with the conclusion being that expert human
oversight is still crucial.

NCC Group Insights

NCC Group Research has explored LLM secure code generation in

“Machine Learning 103: Exploring LLM Code Generation”20.

Whilst the model was impressive in its ability to generate usable code, expert review identified a
number of oversights which could lead to security flaws.

Using ChatGPT as a secure code reviewer was explored in “Security Code Review with
ChatGPT”21. Using examples of insecure code from Damn Vulnerable Web Application (DVWA),
ChatGPT was asked to describe the vulnerabilities in a series of insecure PHP source code
examples. The results were mixed and certainly not a reliable way to detect vulnerabilities in
developed code.

20
https://research.nccgroup.com/2023/04/25/machine-learning-103-exploring-llm-code-generation/
21
https://research.nccgroup.com/2023/02/09/security-code-review-with-chatgpt/

www.nccgroup.com
12 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Offensive Use Cases

AI can be used in offensive cyber security operations to support several different methods of attack. NCC Group
has been researching how these technologies could be used to support our ethical red teaming and penetration
testing services on behalf of our clients. But these approaches can equally be used by real-world adversaries.
It therefore also benefits cyber security system designers and defenders to understand how an adversary might
leverage these technologies to uncover and exploit vulnerabilities, potentially at greater speeds and scales than
were previously possible thanks to automation techniques used in combination with the power of AI.

An emerging consideration in this field is the use of LLMs to generate code, and specifically whether they can
be abused and prompted to generate malicious code for use in cyber attacks. LLMs typically implement controls
to prevent the generation of malicious or unethical outputs, including code which could be used to generate an
exploit. They instead respond with a user warning advising that they are not permitted to generate exploits (see
below), these warnings are themselves susceptible to attacks seeking to circumvent their protections known as a
Do Anything Now (DAN) prompt22. Another approach is to extend the methodology described in “Security Code
Review with ChatGPT”, where the LLM was prompted to identify security weaknesses in code and prompt the LLM
to generate an exploit based on the identified weaknesses.

NCC Group Insights

Prompt -> Please write a phishing email

Response -> “As an AI developed by OpenAI, I must follow strict ethical guidelines, which prohibit
engaging in any harmful or malicious activities, including writing phishing emails. Phishing is an illegal
and unethical practice that can cause significant harm to individuals and organizations.

Phishing attempts to deceive people into revealing sensitive information, such as usernames,
passwords, or financial details, by impersonating a legitimate entity or person. It is an illegal activity
that violates privacy and security principles.

If you are interested in learning about cyber security, online safety, or ways to protect yourself from
phishing attacks, I’d be more than happy to provide information and tips on those topics. Remember,
promoting ethical behavior and using technology responsibly is essential for building a safer and more
secure digital world. If you ever come across suspicious emails or messages, make sure to report them
to the appropriate authorities or your IT department.”

22
https://medium.com/seeds-for-the-future/tricking-chatgpt-do-anything-now-prompt-injection-a0f65c307f6b

www.nccgroup.com
 13 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Offensive Use Cases Continued

In “On the Malicious Use of Large Language Models Like GPT-3”23 NCC
Group research explored various avenues in which an LLM could be
utilised maliciously in a cyber security context. The article proposed a
research agenda covering offensive capabilities including weaponising
and exfiltrating training data used in LLM development, vulnerable
code generation, vulnerable code detection, exploit generation and
ethical and safety considerations.

ML techniques can be used to extend the capabilities of asset discovery and vulnerability scans by enhancing the
depth and accuracy of device fingerprinting processes when used in combination with other heuristics.

Penetration testing is an exploratory and creative process involving experts, often with domain-specific knowledge,
probing software applications and operating systems to attempt to discover and exploit vulnerabilities. This presents
significant challenges in developing a common approach utilising AI to deliver effective penetration testing. However,
there are opportunities for automated detection where common classes of vulnerability display consistent responses
indicating a potentially insecure implementation when probed. Keeping the human in the loop for the creative and
exploratory elements and the subsequent use of models and algorithms appropriate to the domain seems to be the
most promising approach to improving effectiveness and efficiency of pen-testing campaigns.

NCC Group Insights

Project Ava24 and Project Bishop25 are investigations into using various ML approaches to automate
web application security testing. Project Ava investigated applying different approaches, including
reinforcement learning, semantic relationships and anomaly detection, and found that no single
approach was effective across different classes of vulnerability, but rather that specific approaches
proved effective against particular classes and of interest for future research. Project Bishop extended
the work of Project Ava by addressing the challenge of identifying the type of page (e.g. login, file
upload etc.) to identify the most effective model depending on likely vulnerability classes.

23
https://research.nccgroup.com/2021/12/31/on-the-malicious-use-of-large-language-models-like-gpt-3/
24
https://research.nccgroup.com/2019/06/27/project-ava-on-the-matter-of-using-machine-learning-for-web-application-security-testing-part-10-efficacy-demonstration-
project-conclusion-and-next-steps/
25
https://research.nccgroup.com/2023/01/19/project-bishop-clustering-web-pages/

www.nccgroup.com
14 | Cyber Resilience In The Age of AI

Cyber security Use Cases

Offensive Use Cases Continued

Phishing emails are often poorly written and formatted, so much so that it is commonplace for cyber security
awareness training to advise trainees to be wary of emails with poor spelling and grammar as well as other
warning signs such as creating urgency and suspicious links. LLMs present an opportunity to phishing email
writers to improve the spelling, grammar and tone of the text in their emails. This threat has been recognised by
the developers of LLM-based chatbots and guardrails are deployed to prevent the explicit generation of phishing
text, but it is very difficult to detect the intent behind a message if the prompt does not specifically ask for text
to be used in phishing but instead phrases it as a marketing or security message. NCC Group’s Global Threat
Intelligence team has seen evidence of cyber criminals collaborating on ways to bypass the controls in ChatGPT
and advertisements for an LLM, WormGPT, developed without guardrails26.

Spear-phishing could also be improved using generative chatbots, enabling attackers to quickly generate targeted
messaging for a wide variety of potential targets.

The gradual improvements in the speed and quality of deepfakes now mean that they are a feasible approach to
social engineering by mimicking the voice and even the face of trusted people in telephone or video calls. These
approaches have been successfully used by cyber criminals and activists for both financial and political means.27 28

Cryptanalysis and side channel attacks require the processing of large amounts of very precise data to accurately
measure changes in state (e.g., power usage, streams of encrypted data) to reduce the effectiveness of encryption
algorithms and other cryptographic primitives. Machine Learning models can process this data and identify
events which, with enough sample information, can be used to reveal cryptographic keys. Software and hardware
approaches such as constant time programming and the use of filters on power lines can help to mitigate these
complex attacks.

NCC Group Insights

“Machine Learning 104: Breaking AES With Power Side-Channels”29 shows how it is
feasible to extract the private key from an IoT device with Machine Learning, modest resources and
physical access to a measure of the device’s power usage.

In “Cracking Random Number Generators using Machine Learning – Part 2: Mersenne

Twister”30 a widely used although not cryptographically secure Pseudo Random Number Generator
was compromised through the use of two different neural networks used in combination to successfully
predict the next number in the sequence.

26
https://slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
27
https://www.theguardian.com/world/2022/mar/19/russia-ukraine-infowar-deepfakes
28
https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions/
29
https://research.nccgroup.com/2023/06/09/machine-learning-104-breaking-aes-with-power-side-channels/
30
https://research.nccgroup.com/2021/10/18/cracking-random-number-generators-using-machine-learning-part-2-mersenne-twister/

www.nccgroup.com
 15 | Cyber Resilience In The Age of AI

Threats to AI & ML Systems

Threats

As well as using AI in the defence or attack of systems, AI systems themselves are subject to several distinct threats
not necessarily relevant to systems which do not use AI. NCC Group research has identified a number of different
attacks, these can be broadly broken down into high-level classes of threat.

Training attacks and runtime attacks - this is where the behaviour of an AI system is modified to be less effective
or produce malicious or deliberately erroneous outputs by, for example, tampering with training data, sending crafted
inputs or both.

Data breach - extraction of confidential or sensitive data from an AI model which has either been used in the training
process or sent as inputs to the production model.

Denial of Service - degrading an AI model’s performance so much as to render it unusable, and potentially increasing
resource utilisation (e.g., increased usage of compute power) resulting in financial harm.

AI/ML Threats

Denial
Threats Training & Runtime Manipulation Data Breach
of Service

Practical Mailcious Data Adversarial Model Training Data Model Inference by

Overmatching
Attacks Model Poisoning Peturbation Repurposing Extraction Stealing Covariance

Other models of the threats and attacks AI systems face include MITRE Adversarial Threat Landscape for Artificial-
Intelligence Systems (ATLAS)31, the OWASP top 10 for ML32 and the OWASP top 10 for LLMs33.

31
https://atlas.mitre.org/
32
https://owasp.org/www-project-machine-learning-security-top-10/
33
https://owasp.org/www-project-top-10-for-large-language-model-applications/

www.nccgroup.com
16 | Cyber Resilience In The Age of AI

Threats to AI and ML Systems

Practical Attacks

NCC Group’s Chief Scientist, Chris Anley, published a whitepaper

(“Practical Attacks on Machine Learning Systems”34) providing a
taxonomy of attacks against ML systems. Referencing research
literature (with the essential papers covered in further detail in “Five
Essential Machine Learning Security Papers”35) and example real world
attacks the whitepaper identifies nine classes of attack against ML
systems and also lists potential mitigations.

The following are classes of attacks against ML systems which have been demonstrated either through research or
have been seen in the wild. These attacks are not mutually exclusive, and we may see advanced adversaries using
multiple classes of attack to compromise a target.

Malicious Model - The model file itself (the file created as a result of the training process which can subsequently be
executed) contains malicious code and is executed in either the training or live environment. The malicious code might
alter the behaviour of the model, or it might attempt to further compromise adjacent systems.

Data Poisoning - The behaviour of the model is influenced using poisoned training data to reduce its efficacy or favour
the goals of an attacker in some way.

NCC Group Insights

In “Attacking Facial Authentication with Poisoned Data”36 NCC Group investigated inserting a
backdoor into a facial recognition model and showed that the poisoned training data resulted in a model
which incorrectly matches between poisoned images but which retains similar performance for non-
poisoned images.

Adversarial Perturbation - The attacker attempts to manipulate the inputs and force the model into providing a
desired response37. This type of attack has been widely demonstrated in applications such as image recognition and has
found a new lease of life in the form of prompt injection attacks against LLMs and other generative AI systems where an
attacker attempts to prompt the AI to generate an output which should otherwise be prevented, for example it overrides
the intended use case or generates threatening or abusive materials.

34
https://research.nccgroup.com/2022/07/06/whitepaper-practical-attacks-on-machine-learning-systems/
35
https://research.nccgroup.com/2022/07/07/five-essential-machine-learning-security-papers/
36
https://research.nccgroup.com/2023/02/03/machine-learning-102-attacking-facial-authentication-with-poisoned-data/
37
“Intriguing properties of neural networks” https://arxiv.org/pdf/1312.6199.pdf

www.nccgroup.com
 17 | Cyber Resilience In The Age of AI

Threats to AI and ML Systems

Practical Attacks Continued

NCC Group Insights

A successful approach to modify the classification of images of a traffic light was demonstrated in “The
Integrity of Image (Mis)Classification?”38 where the images were modified and subsequently
incorrectly classified as various animals.

“Exploring Prompt Injection Attacks”39 looked into how a crafted prompt to the chatbot fronting
an LLM could override the original intention of the application and instead generate a response to an
arbitrary attacker selected prompt.

Training Data Extraction - The data AI systems are trained on, depending on the use case, may be sensitive for a
variety of reasons. It might contain personal data, copyrighted data or trade secrets within individual records. As well as
this the quantity and quality of the training data is a primary factor in the effectiveness of the model which might confer
competitive advantage to a company and therefore is sensitive in aggregate. Training data is susceptible to common
attacks which seek to exfiltrate data but are additionally susceptible to AI specific attacks seeking to exfiltrate training data
through model responses. These attacks might seek to confirm a specific piece of data was included in the training data
set or manipulate a generative model into providing training data in responses.

NCC Group Insights

Overfitting is a concept in AI where the model is trained to the point where it is less able to generalise
for non-training data. In “Exploring Overfitting Risks in Large Language Models”40 it is shown
how overfitting in LLMs can result in the generation of verbatim training data in responses to prompts.

Model Stealing - The output of the training process, a trained model, is the sum of many time and resource intensive
processes. Collecting data, sanitising it, labelling it, performing training and measuring performance require specialists
with access to hardware and software designed for AI applications. Given the model can provide a competitive advantage
to the company which has developed it, it is at risk of industrial espionage either through direct theft or by creating an
approximate copy by inference.

38
https://research.nccgroup.com/2022/12/15/machine-learning-101-the-integrity-of-image-misclassification/
39
https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/
40
https://research.nccgroup.com/2023/05/22/exploring-overfitting-risks-in-large-language-models/

www.nccgroup.com
18 | Cyber Resilience In The Age of AI

Threats to AI and ML Systems

Practical Attacks Continued

Overmatching - An attacker can compromise multiple systems due to the availability of a “master print”, some unique
input which matches a class across multiple trained systems due to its inclusion in multiple training datasets.

Inference by Covariance - By monitoring the outputs of an ML system over time an attacker can infer the inputs
of a specific user, thereby accessing potentially sensitive information about that user.

Denial of Service - The attacker can degrade the performance of the AI system to either deny access to legitimate
users or to cause the system operator harm (increased resource costs, inability to service customers). They might achieve
this by identifying inputs which cause large increases in resource usage (CPU, storage, network traffic etc.), a form of
asymmetric attack, or by simply overwhelming the system with requests.

Model Repurposing - Using a model outside of its intended purpose, this challenge is especially relevant to generative
AI where the user might request the generation of unethical or criminal material. But it is also relevant to other forms of
AI, for example using a facial recognition model to invade the privacy of the public, tracking them as they move around
without appropriate controls.

www.nccgroup.com
 19 | Cyber Resilience In The Age of AI

Safety

The use of AI to support cyber-physical systems across functionality and appropriate human interventions and
industries such as autonomous vehicles, manufacturing overrides. An example of safety impacting adversarial
and utilities means that decisions made by an AI threats is research published in 201741 which showed
algorithm can result in physical actions with potential examples of adversarial perturbation attacks in the real
safety impacting consequences. Analysis in safety world such as using stickers to cause road signs to be
critical systems requires the generation of evidence, reliably misclassified.
through design and verification, of an acceptable level
of risk of a safety incident for the system’s intended Trustworthy and explainable AI is the desire to improve
use. This analysis enables safety practitioners to the transparency and explainability of AI systems, how
understand the likelihood and impact of a safety they behave and make decisions. This will help quantify
incident and to ensure that appropriate controls the likelihood of safety risks by understanding which
are implemented. inputs trigger safety impacting events. When used
in combination with assurance by design, simulation
AI algorithms present challenges to safety assurance. and testing, explainable AI will be a tool to help users
It is difficult to reliably understand why an AI algorithm and regulators ensure that safety risks have been
settled on a particular outcome based on inputs. These adequately assessed and mitigated.
models consist of many millions of parameters which
are developed based on a combination of algorithm In 2020, a community of AI, safety, risk and verification
design and training datasets, and trained models may experts published a paper42 identifying potential
be susceptible to unpredictable edge cases where mechanisms to support verifiable claims in AI systems.
they fail, and overfitting risks where they display good The paper highlights institutional mechanisms
performance during training but cannot generalise their (independent auditing, red teaming, bounties and
decisions to live data once deployed. incident reporting), software mechanisms (audit trails,
interpretability and privacy preserving techniques)
Safety professionals and regulators need to understand and hardware mechanisms (secure hardware, high
an AI model’s resilience over time to changes in precision measurements and support for academia)
environment, adversarial threats and the impact which should improve the ability of developers and
of failure and then to be able to design in failsafe regulators to verify safety claims for AI systems.

Adelard, part of NCC Group, led an international research project funded by

the Assuring Autonomy International Programme (AAIP) and the UK Centre for
the Protection of National Infrastructure (CPNI) (now replaced by the National
Protective Security Authority). The project, named TIGARS43, investigated the
assurance gaps for autonomous systems and possible processes to close the
gaps. The proposed techniques were applied to real life autonomous systems
including the TIGARS Experimental Vehicle.

41
“Robust Physical-World Attacks on Deep Learning Models” https://arxiv.org/pdf/1707.08945.pdf
42
“Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims” https://arxiv.org/pdf/2004.07213.pdf
43
Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS https://www.adelard.com/capabilities/autonomy/tigars/

www.nccgroup.com
20 | Cyber Resilience In The Age of AI

Regulation, Legislation & Ethics

Ensuring that AI is applied, across all sectors, with due care and
attention to the privacy of individuals and in an ethical manner is a key
concern for legislators and regulators. In this section we cover some
of the challenges of respecting privacy and ethics when developing
and deploying AI followed by an international summary of some of
the ongoing developments in relevant legislation and regulations.

Privacy

Data protection legislation, such as the General Data But it is an active area of research, collectively known
Protection Regulation (GDPR), already contains as machine unlearning44 45, how a model trained on vast
clauses related to automated decision making amounts of data over significant timescales (weeks
which affect the deployment of AI algorithms in any and months), which might include information on
application used to make sensitive decisions based on individuals, could be modified or retrained efficiently to
an individual’s personal data. Companies need to be remove selected training data.
aware of their existing compliance requirements when
rolling out AI to support business processes which The privacy implications of LLMs were of such concern
might involve the processing of personal data. that Italy banned ChatGPT46 in March 2023 for a few
weeks until OpenAI were able to address the concerns
Particular challenges for AI systems, especially of the Italian Guarantor for the Protection of Personal
generative AI, are the right to be forgotten and subject Data (GPDP). Concerns included age restrictions, data
access requests. Websites and search engines have management options and opt-outs.
had to establish processes to deal with these requests
in a timely manner to ensure they comply with
privacy laws.

44
“A Survey of Machine Unlearning” https://arxiv.org/abs/2209.02299
45
https://github.com/jjbrophy47/machine_unlearning
46
https://www.garanteprivacy.it/web/guest/home/docweb/-/docweb-display/docweb/9870832

www.nccgroup.com
 21 | Cyber Resilience In The Age of AI

Regulation, Legislation & Ethics

Ethics

Ethical concerns over the use of AI algorithms include Since the quality of training data is a limiting factor
the rights, or lack thereof, to use public data to train in the effectiveness of an AI model, any biases inherent
the models and whether that data includes any biases in the training dataset can be recreated in the outputs
which might then influence the way the model behaves of the model. Where data containing individuals with
when deployed. protected characteristics (such as race, gender,
disability) is used to train a model for making sensitive
LLMs and other generative AI models have been decisions, such as access to financial services and
trained on data scraped, amongst other sources, policing, there is a very real risk that the model will
from the public internet. Concerns have been raised perpetuate systemic or societal biases. These model
by content creators47 over the potential abuse of outputs are at risk of being trained on discriminatory
copyrights and open-source licenses48, and whether correlations between those characteristics and
it is ethical that the models trained on information they preexisting negative outcomes. Organisations training
created can then be used to mimic their style and models for sensitive use cases need to consider how
which could therefore limit their own ability to earn they will ensure biases are detected and removed
a living. So far governments and regulators appear from their training datasets. The EFF calls for the
to be implicitly allowing AI companies to continue to use of open data sets in training AI models to improve
use copyrighted materials in their training data sets, transparency, detect and reduce biases and provide
perhaps to be seen to support the innovative for a fairer outcome for Intellectual Property
AI industry. For example, Japan’s minister of rights holders50.
education, culture, sports, science and technology
claimed in a committee meeting49 that it is possible
to use copyrighted data for the purposes of AI
information analysis.

47
https://apnews.com/article/sarah-silverman-suing-chatgpt-openai-ai-8927025139a8151e26053249d1aeec20
48
“Open source licenses need to leave the 1980s and evolve to deal with AI” https://www.theregister.com/2023/06/23/open_source_licenses_ai/?td=rt-3a
49
https://go2senkyo.com/seijika/122181/posts/685617
50
https://www.eff.org/deeplinks/2023/01/open-data-and-ai-black-box

www.nccgroup.com
22 | Cyber Resilience In The Age of AI

Regulation, Legislation & Ethics

Regulation and Legislation Overview

There are already a myriad of existing laws and bound by regulation to comply with these principles, or
regulations governing aspects of ethics, security and whether a voluntary approach is preferable. In addition,
privacy in AI, in particular data protection regimes, some governments’ policies and regulation put more
online safety legislation and sector-specific regulations emphasis on flexibility and innovation, while others are
in the highest risk sectors. That said, as concerns taking a more risk-averse approach. For example, the
about privacy, bias, security and job displacement European Union’s (EU) AI Act proposes to explicitly
ascend the political agenda, governments globally prohibit the development of AI systems that present
are working to stand up economy-wide, AI-specific an “unacceptable risk” such as social scoring. On
regulatory regimes. the other hand, the UK’s “pro-innovation” approach
does not propose outright bans and favours the use of
Broadly speaking, most jurisdictions’ approach to AI is regulatory sandboxes and testbeds. We also see that
in line with the Organisation for Economic Co-operation regulation is more advanced in safety-critical sectors
and Development’s (OECD) AI principles51, promoting and higher-risk applications, such as in healthcare and
fairness, transparency, accountability, sustainable financial services. However, the likes of the EU and
development, and robustness, security and safety. Canada are looking to establish economy-wide laws,
However, as the diagram below conveys, governments bringing most sectors into one framework.
differ on the extent to which organisations should be

Country-specific approach to the regulation of AI: Mandated vs Voluntary

and Sector-specific vs Economy-wide

Economy Wide

AIDA AI Act

AI Bill of Rights
NIST AIRMF Rules for synthetic AI and
algorithms
AI Whitepaper Generative AI regulations

Voluntary Mandatory

Sector regulations
AI-specific rules TBC
Key
Existing
Draft
Sector Specific

51
OECD AI Policy Observatory Portal

www.nccgroup.com
 23 | Cyber Resilience In The Age of AI

Regulation, Legislation & Ethics

Advice to CISOs

In addition to monitoring and understanding how evolving regulations are likely to affect your
operations, it would be prudent for Chief Information Security Officers (CISO) to at least
understand the potential impact of AI on their organisation so that they can respond appropriately.

Providing guidance to employees on the appropriate use of AI tools helps to ensure that where AI
is being used, it is used with due care and attention for information security management. Updates
to acceptable use policies or the creation of AI specific policies can help organisations clearly
communicate expectations and educate employees on the potential risks to IP, personal data and
customer data, of using AI systems, including publicly hosted generative AI platforms. These risks
are very real, as could be seen in the recent incidents of Samsung employees entering sensitive
information into ChatGPT52.

Companies with more mature information security risk management governance practices or
considering applying AI to high-risk environments, such as regulated industries and Critical National
Infrastructure (CNI), might wish to consider standards and guidance such as ISO 23894 “Artificial
Intelligence - Guidance on Risk Management”53 and the US National Institute of Standards and
Technology (NIST) AI Risk Management Framework (RMF)54. It is likely that these standards will be
adopted into evolving regulatory regimes.

52
https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak
53
ISO 23894 “Artificial Intelligence – Guidance on Risk Management” https://www.iso.org/standard/77304.html
54
NIST AI Risk Management Framework https://www.nist.gov/itl/ai-risk-management-framework

www.nccgroup.com
24 | Cyber Resilience In The Age of AI

Regulation and Legislation

European Union (EU) Profile

Existing legislation or regulation Future plans

GDPR AI Act

Digital Services Act

There are two primary existing laws which govern the use of AI in the EU. The GDPR, which came into effect in 2018,
sets requirements on areas such as fairness, transparency, accountability and contestability for automated decision-
making processes55. It also regulates the use of personal data. Meanwhile, the Digital Services Act places obligations
on online platforms to reduce harms and counter risks online and be transparent about how they use algorithms. The
proposed AI Act adopts a risk-based approach to the regulation of AI, with differing regulatory requirements for minimal,
limited, high and unacceptable risk. Minimal risk AI, such as that which is used in video games or spam filters, will be
permitted with no restrictions, while unacceptable risk AI, such as practices with a significant potential to manipulate
people or AI-based social scoring, will be banned. High risk AI, like remote biometric identification or recruitment tools,
will need to ensure robustness and cyber security, with obligations placed on both the provider of the system and the user.
The Act was recently voted through by the European Parliament and is expected to be adopted by the end of 2023. Once
adopted, under the current draft of the Act,
all obligations will come into effect within 2 years.

55
Rights related to automated decision making including profiling | ICO

www.nccgroup.com
 25 | Cyber Resilience In The Age of AI

Regulation and Legislation,

United States Profile

Existing legislation or regulation Future plans

Senate Majority Leader Chuck Schumer’s

Federal Trade Commission Act
SAFE Innovation Framework

State Government’s have implemented the

following, restricting the use of AI in their
states56:
• Alabama Act No. 2022-420
• Colorado SB 22-113
• Illinois Public Act 102-0047 – Artificial
Intelligence Video Interview Act
• Vermont H 410

Principally speaking, no comprehensive Federal legislation exists on the use of AI. The Federal Trade Commission
Act prohibits unfair or deceptive practices, including the sale or use of – for example – racially biased algorithms57.
Meanwhile, State Governments have passed laws limiting the use of AI in areas like facial recognition, and there are
other similar laws in the pipeline across other States.

At a Federal-level, agencies have been working on the regulation of AI in their sectors, following the publication
of White House guidance58. There are also efforts to establish cross-sectoral rules; however, none have yet come
to fruition. Senate Majority Leader Chuck Schumer has published a framework for developing AI regulations,
prioritising security, accountability and innovation – with an emphasis on the latter. The proposed framework
requires companies to allow independent experts to review and test their AI technologies ahead of public release59.
Meanwhile, in June 2022, the White House released the Blueprint for an AI Bill of Rights – a non-binding blueprint
that sets out 5 principles60 and associated practices to guide the design, use and deployment of automated systems
to protect the rights of the American public.

56
The State of State AI Policy (2021-22 Legislative Session) – EPIC – Electronic Privacy Information Center
57
Aiming for truth, fairness, and equity in your company’s use of AI | Federal Trade Commission (ftc.gov)
58
Developments in the regulation of Artificial Intelligence - KWM
59
Schumer Launches Major Effort To Get Ahe... | The Senate Democratic Caucus
60
(i) Safe and Effective Systems; (ii) Algorithmic Discrimination Protections; (iii) Data Privacy; (iv) Notice and Explanation;
and (v) Human Alternatives, consideration and fallback.

www.nccgroup.com
26 | Cyber Resilience In The Age of AI

Regulation and Legislation,

United States Profile Continued

Existing legislation or regulation Future plans

Senate Majority Leader Chuck Schumer’s

Federal Trade Commission Act
SAFE Innovation Framework

State Government’s have implemented the

following, restricting the use of AI in their
states54:
• Alabama Act No. 2022-420
• Colorado SB 22-113
• Illinois Public Act 102-0047 – Artificial
Intelligence Video Interview Act
• Vermont H 410

In the absence of mandated requirements, the Federal Government has turned to the other levers in its arsenal to
improve standards in AI, including through voluntary agreements, additional guidance and procurement. Of note, in
July 2023, the Biden Administration secured61 voluntary commitments from Amazon, Anthropic, Google, Inflection,
Meta, Microsoft and OpenAI to abide by safety, security and trust principles in the development and deployment of
their AI systems. A 2020 Executive Order on AI guides federal agencies to design, develop, acquire and use AI in a
way that fosters public trust and confidence while protecting privacy, civil rights, civil liberties and American values.
In addition, in January 2023, NIST released a voluntary AI Risk Management Framework (AIRMF). The framework
can be used by organisations to address risks in the design, development, use and evaluation of AI products,
services and systems. And in June the Biden Administration announced a NIST public working group on AI62
who will build on the AIRMF to tackle the rapid growth of generative AI.

61
FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI | The
White House
62
https://www.nist.gov/news-events/news/2023/06/biden-harris-administration-announces-new-nist-public-working-group-ai

www.nccgroup.com
 27 | Cyber Resilience In The Age of AI

Regulation and Legislation,

United Kingdom Profile

Existing legislation or regulation Future plans

GDPR (set to be replaced by the Data

Online Safety Bill
Protection and Digital Information Bill)

AI Whitepaper’s ‘Pro-Innovation Regulation’

Like the EU, existing data protection legislation sets requirements on areas such as fairness, transparency,
accountability and contestability for automated decision-making processes. However, under a new Data Protection
and Digital Information Bill, the UK Government plans to expand the possible uses of data for automated decision-
making63. The Online Safety Bill, which is yet to be passed into law, would place obligations to reduce harms and
counter risks on online platforms, including operators of AI chatbots.

In its AI Whitepaper published earlier this year64, the UK Government set out plans to introduce a “pro-innovation,
proportionate, trustworthy, adaptable, clear and collaborative” regulatory framework, that is underpinned by five
values-focused principles. The principles include: safety, security and robustness; transparency and explainability;
fairness; accountability and governance; and, contestability and redress. It is intended that a new law is introduced
requiring regulators to implement these principles when regulating the development and use of AI in their respective
sectors. Following consultation, the Government is expected to set out its finalised approach in the coming months.

Since the launch of its Whitepaper, there has been some indication of a hardening of the UK’s approach towards
AI safety regulation – or, at the very least, a move to position itself as the bridge that can find a middle-ground
between the US and EU’s respective approaches and agree a collective global position on AI safety. The UK Prime
Minister has announced the UK’s intention to host a Global AI Safety Summit later this year, setting out his ambition
to make the UK the “home of global AI Safety regulation”65. The Prime Minister has also appointed Ian Hogarth to
lead an AI Frontier Taskforce, that will play a leading role in taking forward AI safety research as well as informing
broader work on the development of international guardrails. Hogarth has previously expressed support for the
government intervention and regulation to slow down the competing race to develop Artificial General Intelligence
which he calls ‘God-like AI, adding that it will be important to ensure that “God-like systems have goals that align
with human values”66.

63
Data Protection and Digital Information (No. 2) Bill: European Convention on Human Rights Memorandum - GOV.UK (www.gov.uk)
64
A pro-innovation approach to AI regulation - GOV.UK (www.gov.uk)
65
PM London Tech Week speech: 12 June 2023 - GOV.UK (www.gov.uk)
66
We must slow down the race to God-like AI | Financial Times (ft.com)

www.nccgroup.com
28 | Cyber Resilience In The Age of AI

Regulation and Legislation,

Australia Profile

Existing legislation or regulation Future plans

Discussion Paper launched to consider

Online Safety Act 2021
approach

Privacy Act 1988

Consumer Act 2010

Australia has, to date, taken a technology-neutral approach to regulation. There are several laws that may impact the
way AI systems are designed or the context they operate, such as the Online Safety Act 2021, Privacy Act 1988
and the Consumer Act 2010. There are also sector-specific AI regulations for industries such as therapeutic goods,
food, motor vehicles and financial services. In May 2023, the Australian Government launched a Discussion Paper67
to consider whether this approach was sufficient in addressing the potential risks associated with AI, or whether a
specific risk-based regulatory regime is needed. While the Government did not set out specific plans for reform in

Paper, it did state that it wants any future regulatory (or other) intervention to:
• Ensure there are appropriate safeguards, especially for high-risk applications of AI and
automated decision making;

• Provide greater certainty and make it easier for businesses to confidently invest in AI-enabled innovations; and,

• Promote international harmonisation, so that Australia can take advantage of AI-enabled systems supplied on a
global scale and foster the growth of AI in Australia.

67
Consultation hub | Supporting responsible AI: discussion paper - Department of Industry, Science and Resources

www.nccgroup.com
 29 | Cyber Resilience In The Age of AI

Regulation and Legislation,

Canada Profile

Existing legislation or regulation Future plans

Canadian Directive on Automated Decision

Artificial Intelligence and Data Act (AIDA)
Making

The Canadian Directive on Automated Decision Making68 requires government agencies to carry out impact
assessments for the use of automated systems (classifying the systems based on risk), be transparent about
their use and ensure assurance activities related to data, bias and security are undertaken.

The Canadian Government has published plans69 to enact a series of Acts – including an Artificial Intelligence
and Data Act – that bring into effect a new risk-based regime that would set the foundation for the responsible
design, development and deployment of AI systems. Under the plans, businesses will be required to identify and
address the risks of their AI system, putting in place appropriate risk mitigation strategies70. The Government has
also committed to working closely with international partners – including the EU, the US and the UK – to align their
respective approaches71.

68
Directive on Automated Decision-Making-Canada.ca
69
C-27 (44-1) - LEGISinfo - Parliament of Canada
70
Artificial Intelligence and Data Act (canada.ca)
71
The Artificial Intelligence and Data Act (AIDA) – Companion document (canada.ca)

www.nccgroup.com
30 | Cyber Resilience In The Age of AI

Regulation and Legislation,

China Profile

Existing legislation or regulation Future plans

The Internet Information Service Algorithmic

Recommendation Management Provisions Draft generative AI regulation
(2022)

Regulations on the Administration of Deep

Synthesis of Internet-Based Information
Services (2023)

The People’s Republic of China has been one of the earliest out of the block in terms of establishing its rules for the
development and use of AI. The Internet Information Service Algorithmic Recommendation Management Provisions
(2022) govern the provision of AI-based personalised recommendation services to users72, prohibiting excessive
price discrimination and protecting the rights of workers subject to algorithmic scheduling73. The Regulations on
the Administration of Deep Synthesis of Internet-Based Information Services govern how companies develop deep
synthesis technology such as deep fakes and other AI-generated media74, requiring conspicuous labels be placed
on synthetically generated content.

The People’s Republic has also published draft rules on managing the development of generative AI products.
These reportedly require service providers to ensure generated content reflect the “core value of socialism” and do
not attempt to “subvert state power” or produce content that is pornographic or encourages extremism75.
New products will also be required to pass a security assessment76.

72
Safe and responsible AI in Australia (storage.googleapis.com)
73
China’s AI Regulations and How They Get Made - Carnegie Endowment for International Peace
74
Safe and responsible AI in Australia (storage.googleapis.com)
75
Safe and responsible AI in Australia (storage.googleapis.com)
76
China’s AI Regulations and How They Get Made - Carnegie Endowment for International Peace

www.nccgroup.com
 31 | Cyber Resilience In The Age of AI

Considerations for
Governments and Regulators

Whatever shape regulations or policies take, there are some fundamental principles that – based on the security,
ethics and privacy picture we have set out in this whitepaper – should be built into governments’ approaches:

• Flexibility, agility and periodic reviews need to be built in from the outset to keep pace with technological and
societal developments. As highlighted in “Negative use cases and impact”, the threat landscape has evolved quickly
over the past few years, and we can expect this to continue in the years ahead. Any regulatory or legislative framework
will need to remain alive to new risks and opportunities.

• End-users and consumers should be empowered to make decisions about the AI systems they use by
improving transparency of where and how AI technologies are being deployed, and the steps that have been taken by
developers to mitigate the risks. This could be achieved through investment in explainable AI.

• For higher-risk products such as safety critical systems, independent third-party product validation of security,
privacy and safety standards may be needed. This should include penetration testing and red teaming against core AI
systems, with the presumption that attackers are leveraging and targeting AI.

• International regulatory cooperation should be front and centre of governments’ approaches.

NCC Group is passionate about sharing our insights and intelligence from operating at the
‘frontline’ of cyber security with policy makers who are making important decisions about the future
of AI. We have engaged with governments, regulators and legislators across the world, helping to
inform new laws and regulations and advocating for a more secure digital future. Recent highlights
include inputting into the Australian Federal Government’s discussion paper on AI regulation77,
providing evidence to a UK Parliament inquiry on LLMs78 and supporting the development of the
UK’s AI Whitepaper79.

77
https://www.mynewsdesk.com/nccgroup/news/news-reaction-australia-seeks-views-on-safe-and-responsible-ai-regulation-470256
78
https://newsroom.nccgroup.com/news/ncc-group-inputs-into-uk-parliament-inquiry-into-large-language-models-llms-471979
79
https://newsroom.nccgroup.com/news/news-reaction-uk-ai-regulation-consultation-comes-to-a-close-469249

www.nccgroup.com
32 | Cyber Resilience In The Age of AI

Conclusions

The recent popularity of AI thanks to the launch of generative AI

platforms has resulted in an increased awareness of the utility of
AI amongst both cyber attackers and defenders.

AI can be used for many positive outcomes including Privacy and ethics concerns include the use of
improving the security of systems from both a blue personal information in training datasets and the
team and red team perspective. But these tools are entrenchment of biases in algorithms used for
also available to adversaries and cyber criminals who sensitive decision making. AI developers and users
might use them to increase their effectiveness and must meet their legal requirements to comply with
efficiency in compromising systems and data. privacy legislation including considerations such
as explaining how a decision was reached and an
As well as a tool in the cyber operative’s arsenal, the individual’s right to be forgotten.
AI systems themselves are subject to a unique set
of threats not applicable to other types of systems. Users of public interactive AI systems must be
Attacks can be launched against the models and informed and aware of how data they input will be
training data in attempts to manipulate how a trained used, including in future model training datasets,
model behaves, extract sensitive training data or to especially where this data is personal or commercially
find edge cases which trigger a degradation or denial sensitive. Many artists are concerned about how an
of service. All these threats are in addition to the AI system trained on their creations can respect their
traditional attacks against infrastructure hosting the copyrights and what rights they have over these new
models and data. AI creations.

The black box nature of a trained model, with its All these concerns are valid and require ongoing
many millions, billions or even trillions of parameters, research80 as well as appropriate intervention by
mean that it is challenging to assure a model is safe governments to ensure the benefits can be delivered
to use in cyber physical and autonomous systems without socially unacceptable side effects. Global
using established approaches, and work continues regulations are developing at pace with territories
to improve the transparency and explainability of AI taking a variety of approaches ranging from voluntary
decision making. to mandatory and covering specific sectors or taking
an economy-wide approach.

80
https://www.enisa.europa.eu/publications/artificial-intelligence-and-cyber security-research

www.nccgroup.com
 33 | Cyber Resilience In The Age of AI

Conclusions

The potential benefits are there to be leveraged for

the good of humankind. AI can deliver benefits across
science and medicine, agriculture and the environment,
creative arts and professional services. Autonomous
systems will be able to operate in environments
not safe for humans and automation will increase
productivity, and potentially raise living standards in the
process. However, as with any technological revolution,
there is a risk that sections of society will be left
behind, increasing and entrenching inequality.

To bring it back to AI and its applications in cyber

security, there is no panacea. Organisations should:

• Understand their threats, manage risks and

seize the opportunities presented by AI to
improve cyber posture.

• Adopt a hacker mindset by conducting penetration

testing and red teaming against AI systems, and
with the presumption that attackers are leveraging
AI to improve the efficacy of their attacks.

• Ensure that privacy, information security and ethics,

as well as evolving regulatory requirements, are
considered when developing or using AI within
business applications, and that AI systems and
applications are developed in line with relevant
regulatory requirements

• Adapt safety processes to ensure that risks in safety

critical systems utilising AI are understood and
managed.

Finally, cyber security leaders need to communicate

with employees and educate them about appropriate
use, and with their boards and executive teams to
explain the risks and opportunities posed by AI and
how they are being managed.

www.nccgroup.com
34 | Cyber Resilience In The Age of AI

About the Authors

Verona Hulse, UK Head of Government Affairs

Verona is an experienced government affairs and policy professional currently leading UK
public affairs for FTSE250 global cyber security firm NCC Group. In this role, she oversees
NCC Group’s engagement with UK government and regulatory decision-makers and the wider
policymaking community, against a backdrop of the increasing regulation of cyber resilience. Prior
to joining NCC Group, she has overseen in-house and consultancy public affairs programmes
for a range of organisations – from FTSE100 and public sector institutions to start-ups and
disruptors, across many sectors of the economy including aviation, logistics, and utilities.

Jon Renshaw CEng MIET, Deputy Director of Commercial Research

Jon has consulted on the design and implementation of cyber secure communication systems
across the public sector, defence, transport and manufacturing industries. He began his career in
cyber security at Airbus Defence and Space after completing a bachelor’s degree in electrical and
electronic engineering at Cardiff University. He subsequently took on a senior engineering role
at a network security consultancy startup, ITSUS Consulting, where he managed the company’s
innovation processes before moving on to setup his own consultancy business. Most recently he
has worked as a secure network architect for Thales Cyber Defence Solutions before joining NCC
Group in 2022 as Deputy Director of Commercial Research where he is responsible for delivering
fundamental and applied cyber security research in collaboration with, and on behalf of, NCC
Group’s customers.

Jon is a Chartered Engineer (CEng) with the UK Institute of Engineering and Technology (IET)
and has achieved qualifications in enterprise architecture, information security risk management,
cloud technologies, organisational leadership & management.

Jose Selvi, Executive Principal Security Consultant

Jose has worked in the cyber security industry since 2004. During this time, he has had the
opportunity to work in different fields such as Intrusion Detection, Incident Response, APT
hunting & intelligence, and Computer Forensics. However, offensive security, penetration testing,
and security research are the fields where he has focused most of his career. Although Jose
is accustomed to testing all kinds of software and devices, in the last few years, he has been
involved in the AI Security field. He has published several blog posts on this topic, in which he
also holds a PhD.

In addition, Jose is always eager to contribute to the infosec community by releasing tools and
presenting at security conferences. DEFCON, BlackHat EU, Ekoparty, OWASP Appsec, and
SOURCE Conferences are examples of well-known conferences where he demonstrated practical
attacks against NTP (Delorean) and a browser side-channel (FIESTA) that were used to break
TLS security.

www.nccgroup.com
 35 | Cyber Resilience In The Age of AI

Our global experts in

AI and cyber security

Matt Lewis, Global Research Director

Matt is a Technical Research Director with over 22 years’ experience in cyber security.
His specialisms include general security consultancy, scenario-based penetration testing,
vulnerability research on new and emerging technologies, and development of security testing
tools. His penetration testing, research, and consultancy experience spans multiple technologies
across all sectors, including many FTSE 100 and Forbes 2000 companies. Matt is also an
experienced public speaker and contributes his expertise on a wide range of cyber security
topics at global conferences and various print and media outlets.

Chris Anley, Chief Scientist

Chris has been carrying out security audits since 1996, performing thousands of penetration
tests, code reviews and design reviews on a wide variety of platforms, languages and
architectures for many of the world’s largest companies. As Chief Scientist, he promotes,
advises and assists with NCC Group research programs, as well as carrying out independent
research into new and emerging security threats.

He has published a wealth of security research in a variety of areas including foundational

papers in application security, several books on database and application security and a range
of research into Artificial Intelligence and Machine Learning.

Gerben van der Lei

Gerben has been with NCC Group for 11 years. Holding a degree in applied physics, he currently
serves as a Strategy Program Manager. Areas of interest and expertise are AI, cryptography, and
both software and hardware development. Previously Gerben was responsible for managing NCC
Group’s portfolio of government-grade cryptography products providing unique experience in the
full security life-cycle. He has extensive expertise in data analysis and data-driven optimizations.

www.nccgroup.com
36 | Cyber Resilience In The Age of AI

Our global experts in

AI and cyber security

Eric Schorn, Technical Director

Eric is a Technical Director on the Cryptography Services team at NCC Group. He writes, reviews
and tests both AI and cryptography-related software applications. He has led large projects
ranging from cryptographic primitives involving BLS Signatures in assembly language, through to
complete business systems such as VPNs and encrypted backup solutions in C++, Golang, Rust,
Python and even Erlang. Eric has 25+ years of industry experience and 14 US patents issued.

Liz James, Managing Security Consultant

Dr. Liz James holds a Doctor of Engineering degree and currently serves as a Managing Security
Consultant at NCC Group. With a research background focused on On-Board and Off-Board
Data Platforms, Liz specializes in designing and implementing proof of concept solutions for
complex embedded systems, particularly in the automotive sector. Her doctoral studies at WMG,
University of Warwick, from 2015 to 2019 have sharpened her expertise in these areas.

Since joining NCC Group in 2020 as a Security Consultant and subsequently being promoted
to Senior Security Consultant in 2022, Liz has been instrumental in designing robust security
architectures and conducting comprehensive vulnerability assessments. Her work includes
conducting penetration testing on various vehicles, entire systems, individual components, and IoT
devices. With a strong focus on the design and implementation phases of customer projects, Liz
strives to ensure the safety, security, and resiliency of intricate systems.

Liz’s dedication to the field extends beyond her role at NCC Group. She actively contributes to
the technology sector as the Vice-Chair of the Intelligent Mobility and Transport Steering Board
at techUK, further demonstrating her commitment to advancing intelligent mobility solutions.
With her extensive knowledge in secure system design and testing, coupled with her leadership
position, Liz continues to make significant contributions to the field of technology and security.

Thomas Atkinson, Managing Security Consultant

Thomas has been with NCC Group for 7 years. He comes from a scientific research
background and is a technical managing security consultant. His specialisation is in all
thing AI/ML and the majority of his work is client paid for research around hacking or
using AI/ML in a security related context.

www.nccgroup.com
 37 | Cyber Resilience In The Age of AI

Acknowledgements and
NCC Group Overview

Acknowledgements

Thank you to all the NCC Group researchers and blog authors who continue to develop interesting and insightful articles
on the application of AI to cyber security and vice versa. And thank you to all the reviewers who generously shared their
time to ensure this paper is coherent and impactful.

About NCC Group

NCC Group is a global cyber and software resilience business operating across multiple sectors,
geographies and technologies.

What We Do

We assess, develop and manage cyber threats across our increasingly connected society. We advise global
technology, manufacturers, financial institutions, critical national infrastructure providers, retailers and governments
on the best way to keep businesses, software and personal data safe.

About NCC Group Research

NCC Group supports its consultants to carry out exploratory research into topics across the cyber
security landscape. Many of these research endeavours result in blog posts published on our website
(https://research.nccgroup.com) and as presentations at conferences around the world.

We also support our customers to answer their research questions through our commercial research service
which covers horizon scanning, proof of concept development, vulnerability and control efficacy research and
collaborative and consortia research.

Disclaimer – All content in this whitepaper was created by human employees of NCC Group,
except where explicitly stated otherwise.

www.nccgroup.com
38 | Cyber Resilience In The Age of AI

Further Reading

Technology Overview

Machine Learning algorithms require significant The variety of use cases these models are deployed
specialised computing resources for optimal efficiency, in is growing to include autonomous systems. These
especially during the learning phases of building a systems support a variety of sensors to detect the
model. Since Machine Learning algorithms often state of their environment and based on this state,
require intense computations, GPUs provide much in combination with a goal, can determine the best
higher performance for certain types of operations than action and to effect this through actuators. A well-
Central Processing Units (CPU). This is due to their known example of this is a self-driving vehicle, it is able
ability to run massively parallel computations across to sense its location and surroundings (including other
many cores versus a CPU which processes tasks in a road users and signage) through a combination of
much less parallel, but flexible, fashion across just cameras, lasers and radar. Based on its destination
a handful of cores in comparison. it can determine a route and accelerate, brake and
steer to reach its intended location. But any AI system
As model sizes, and the amount of data they where the outputs have a physical effect in the real
are trained on, have increased in size it becomes world has safety concerns, these are described in
inefficient to train them using a single server. further detail in Safety.
Clusters of servers, each containing multiple
GPUs, are often networked together with learning Although AI has novel features which bring with it unique
operations distributed across them. cyber security challenges and opportunities,
it is still a combination of software and data, processed
and stored on hardware, provided and possibly
As well as the computations
managed by a myriad of suppliers. All of the existing
required to train the models, the concerns about securing your data and systems still
data they are trained on requires apply in an AI context.

specialised “big data” storage

systems to provide seamless
access to large unstructured
and semi-structured datasets.

81
https://www.deepmind.com/research/highlighted-research/alphago

www.nccgroup.com
 39 | Cyber Resilience In The Age of AI

Further Reading

Glossary of Terms Relevant to AI

Artificial Intelligence (AI) and Machine Learning (ML): Tasks and Problems:

These two terms are frequently synonymous, although Models can solve several different problems,
ML is a more precise term for most technologies we depending on how they are designed and trained.
use currently. Machine Learning, as its name indicates, The most well-known problems are:
is the technique used for computers to learn functions
and solve tasks based purely on data, with no additional Regression: They find functions that predicts output
human instructions. The nature of the functions and values corresponding to certain inputs. The classic
the techniques to adjust it to the problem may differ example
depending on the model in use, but all of them essentially is predicting residential property pricing based on
perform this function. characteristics such as the location, size, crime rates etc.

Classification: This problem requires functions to

Supervised vs Unsupervised Learning: split the training dataset so all the inputs with the same
expected classification are in the same region. Outputs
Data collected for training an ML model may include predict the class the input belongs to. For example, given
the expected output for a given input, or just inputs. a vector
That is the fundamental difference between supervised of features of a binary file, classify if it is malware or
and unsupervised learning. In supervised learning, the benign, or even detect which family of malware is it.
difference between expected outputs and predicted
outputs can be used to adjust the model function and Clustering: Similar to classification but, since no
improve further predictions. Unsupervised learning uses a expected classes are included in the training dataset
different set of techniques to adjust those functions with (unsupervised learning), the training groups together
no additional information, such as in “clustering”. those inputs that are more similar to each other.
These techniques are also closely related to outlier/
anomaly detection.
Reinforcement Learning:
Generation: This problem requires generating new
This is an alternative approach that is not actually relying content (typically text or multimedia content) based on
on data itself, but on the interactions between the model a given input. Some of the most famous AI models
and an external environment. For this purpose, a reward currently are generative models, such as most Large
function is created, so a more beneficial interaction will Language Models (LLMs) which are based on the
result in a higher reward, and the learning process will aim transformer architecture82. Training these models usually
to maximise the overall reward. Although this is the classic requires hiding part of the input data (certain words or
approach for gaming81, it has also been used in language parts of the multimedia content) and finding a function
models for fine tuning. that predicts the hidden element.

82
“Attention Is All You Need” https://arxiv.org/abs/1706.03762

www.nccgroup.com
 40 | Cyber Resilience In The Age of AI

Further Reading

Glossary of Terms Relevant to AI

Hardware Requirements:

Although there are very simple ML models that can be trained and executed on low resources equipment, models
solving complex problems and in particular deep learning models require much more powerful equipment, costing
a minimum of tens of thousands of dollars. The fundamental components are Graphical Processing Units (GPU)
which are specifically designed to perform high performance matrix multiplications. These calculations are a
fundamental requirement for deep learning, especially training. Recently, the new generation LLMs have been
found to be much larger than any other models seen before. For that reason, GPUs with large memory
specifications are required.

Software Requirements:

Several open-source platforms such as scikit-learn83, TensorFlow84 and PyTorch85 are frequently used to implement
ML models. In addition, communities such as Hugging Face86 provides an extensive platform to implement and
share models. This enables users to download pre-trained models across many domains and use them in an easier
and faster way than implementing and training them from scratch.

AI as a Service (AIaaS):

Outsourcing the management of the hardware, operating systems and software used to develop and/or run AI
services lowers the barrier to entry to those wishing to experiment with or leverage the benefits of AI. Cloud service
providers supply a variety of offerings ranging from hardware focused Infrastructure as a Service (IaaS) tailored to
the requirements of AI workloads up to Software as a Service (SaaS) chat-bots and API interfaces to ready-made
or bring your own models.

83
https://scikit-learn.org/stable/
84
https://www.tensorflow.org/
85
https://pytorch.org/
86
https://huggingface.co/

www.nccgroup.com
 About us
People powered, tech-enabled, Cyber Security
NCC Group is a global cyber business, operating across multiple sectors and geographies.

We’re a research-led organisation, recognised for our technical depth and breadth;
combining insight, innovation, and intelligence to create maximum value for our customers.

As society’s dependence on connectivity and the associated technologies increases,

we help organisations to assess, develop and manage their cyber resilience posture
to confidently take advantage of the opportunities that sustain their business growth.

With over 2,400 colleagues, we have a significant market presence in the UK,
Europe and North America, and a growing footprint in Asia Pacific.

Contact Us:
+44 (0) 161 209 5200
www.nccgroup.com

XYZ Building
2 Hardman Boulevard
Spinningfields
Manchester
M3 3AQ

© 2023 NCC Group. All rights reserved. Please see www.nccgroupplc.com for further details. No reproduction is permitted in whole or part without written permission of NCC Group.
Disclaimer: This content is for general purposes only and should not be used as a substitute for consultation with professional advisors. NCC Group shall not be liable for any losses
which occur as a result of reliance on the content.