0% found this document useful (0 votes)
96 views21 pages

The State OF AI Sercurity

The Cisco State of AI Security report provides an overview of AI security developments, focusing on emerging threats, policy changes, and research insights. It highlights the increasing risks associated with AI infrastructure, supply chains, and specific attack vectors such as jailbreaking and data poisoning. The report emphasizes the need for robust security measures as AI technology becomes more integrated into critical sectors, while also introducing Cisco AI Defense as a comprehensive solution for enterprise AI security.

Uploaded by

alhanbly
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
96 views21 pages

The State OF AI Sercurity

The Cisco State of AI Security report provides an overview of AI security developments, focusing on emerging threats, policy changes, and research insights. It highlights the increasing risks associated with AI infrastructure, supply chains, and specific attack vectors such as jailbreaking and data poisoning. The report emphasizes the need for robust security measures as AI technology becomes more integrated into critical sectors, while also introducing Cisco AI Defense as a comprehensive solution for enterprise AI security.

Uploaded by

alhanbly
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

© 2025 Cisco and/or its affili es. All rights reserved.

Contents

Executive Summary 3
The AI Threat Landscape 4

Overview 4

Emerging AI Security Risks and Attack Vectors 4

Looking Ahead: New and Improved AI Threat Vectors 10

Developments in AI Policy 11

Overview 11

Domestic AI Policy Developments in 2024 11

International AI Policy Developments in 2024 13

Looking Ahead: Direction for AI Policy in 2025 14


AI Security Research 16
Overview 16
Algorithmically Jailbreaking Large Language Models 16
Fine-Tuning Breaks Internal Model Guardrails 17

Training Data Extraction via Decomposition 18


Poisoning Web-Scale Training Datasets 18
Recommendations for Implementing AI Security 19
AI Security at Cisco 20
Contributors 21

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 2
Executive Summary

Artificial intelligence (AI) has emerged as one of This is Cisco’s inaugural State of AI Security report. Its aim is to
provide a comprehensive overview of important developments in
the defining technologies of the 21st century. It
AI security across several key areas: threat intelligence, policy,
has transformed both our personal and and research. We’ll reflect on progress from the past year while
professional lives, and its rapid advancement will simultaneously looking at what’s ahead and highlighting the ways
continue to reshape the ways in which in which Cisco is investing in a safer, more secure future for AI.
Ultimately, we want to help our customers better understand the
businesses operate. Business leaders largely
AI landscape so that they might be better equipped to manage
recognize the generational opportunity that AI the risks and reap the benefits that AI brings.
presents and feel tremendous pressure to
harness this potential. Findings from our Cisco The State of AI Security report will cover:

2024 AI Readiness Index show that the race to · In-depth analysis of threats to AI infrastructure, AI supply
chains, and AI applications and evaluation of the implications AI
integrate AI into critical business functions is
threat vectors such as model backdoors, prompt injections, and
impeded by a few practical challenges—of which, data extraction.
AI security is the most prominent.
· Important developments in U.S. and international AI policy,
As AI systems handle increasingly sensitive workloads in vital highlighting common themes and macro trends from hundreds of
sectors such as healthcare, finance, and defense, the need for AI-related legislation, executive orders, partnership agreements,
robust safety and security measures becomes nonnegotiable. and security frameworks.
The threat landscape for AI is novel, complex, and not
effectively addressed by traditional cybersecurity solutions. · Original research into algorithmic jailbreaking, dataset
Similarly, streamlining the integration of AI capabilities while poisoning, data extraction, and several other cutting-edge AI
adhering to new compliance frameworks and regulations can security topics led by Cisco’s own AI research team.
make AI adoption feel overwhelming and costly. We are also excited to introduce Cisco AI Defense, the first truly
comprehensive solution for enterprise AI security. Announced in
January of this year, AI Defense builds on our decades of
networking and security experience to help enterprises protect the
development, deployment, and usage of AI across their
organizations.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 3
The AI Threat Landscape

Overview Emerging AI Security Risks and


2024 witnessed the continued market expansion of artificial Attack Vectors
intelligence and machine learning applications, to include AI
business integrations and tools that provide productivity Direct Compromise of AI Infrastructure
gains. As of early 2024, 72 percent of 1,363 surveyed
organizations said they adopted AI capabilities in their Attackers are focused on targeting infrastructure
business functions. Meanwhile, the Cisco AI Readiness supporting AI systems and applications, particularly on the
Index reported that only 13 percent of 7,985 senior unique vulnerabilities of AI deployment environments.
business leaders surveyed said they are ready to leverage Compromises in AI infrastructure could result in cascading
AI and AI-powered technologies to their full potential. effects that can impact multiple systems and customers
Organizations across industries have increasingly integrated simultaneously, and attackers can proceed to conduct
AI into their products or workflows. In cybersecurity, for additional operations targeting model training jobs and
example, AI enhances threat and vulnerability detection, model architecture, models’ training data and
automates response, and bolsters organizations’ overall configurations, hijacking expensive computational
security postures. resources, data exfiltration, or numerous other end goals.
We confidently assess that addressing security risk to AI
While the advancement and adoption of AI technology has models, systems, and applications themselves is an
paved the way for copious new business opportunities, it overlooked aspect of the AI development lifecycle.
also complicates the risk and threat environments: the rapid
adoption of AI technology or AI-enabled technology has led In 2024, attackers successfully compromised NVIDIA’s
to an expanded attack surface and novel safety and Container Toolkit, which could allow attackers to access
security risks. Cisco’s AI security team—the threat and control the host file system, conduct code execution,
researchers and developers behind Cisco’s new AI denial of service, escalation of privileges, information
Defense security solution—is watching this space closely. In disclosure, and data tampering.
addition to maintaining our taxonomy of security and safety
risks, here are the potential threats in AI we are most Earlier in 2024, attackers also compromised Ray, an
worried about: open-source AI framework GPU cluster management
system, hijacking computational resources for other ends
• Security risk to AI models, systems, applications, and such as cryptocurrency mining, while potentially
infrastructure from both direct compromise of AI assets accessing model training data and other sensitive
as well as vulnerabilities in the AI supply chain information. This incident was widely considered the first
in-the-wild attack (i.e., an attack that occurred outside of
• The emergence of AI-specific attack vectors targeting a research setting) against an AI framework.
large language models (LLMs) and AI systems (e.g.,
jailbreaking, indirect prompt injection attacks, data AI systems are increasingly embedded in critical
poisoning, data extraction attacks) applications, from finance and healthcare to national
security and other autonomous systems. These incidents
• Use of AI to automate and professionalize threat actor show the variability of AI infrastructure attacks and
cyber operations, particularly in social engineering underscore the need to protect against them to prevent
cascading impact on business operations, public safety,
While these threats might be on the horizon for 2025 and or even national security.
beyond, threats that emerged in 2024 mainly featured AI
enhancing existing malicious tactics rather than aiding in AI Supply Chain Compromise
creating new ones or significantly automating the kill-
chain. Most AI threats and vulnerabilities are low to The AI ecosystem's reliance on shared models, datasets,
medium risk by themselves, but those risks combined with and libraries expands the attack surface into the AI supply
the increased velocity of AI adoption and the lagging chain. Supply chain attacks exploit the trust organizations
development, implementation, and adherence to place in third-party components—whether they be pre-
accompanying security practices will ultimately increase trained models, open-source libraries, or datasets used to
organizational risks and magnify potential negative impacts train AI systems. When parts of the supply chain are
(e.g., financial loss, reputational damage, or violations of compromised, it can introduce hidden vulnerabilities that
laws and regulations). may not be discovered until significant damage has been
done. Adversaries targeting an AI system’s building blocks
and related components can be particularly concerning
due to their potential for widespread impact across
multiple downstream applications and systems.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 4
Developers frequently integrate pre-trained models, Compromised machine learning libraries (e.g.,
software libraries, and datasets from external sources, TensorFlow and PyTorch have both been targets of
which can create several risks, such as backdoored attack) can introduce vulnerabilities that can manifest
models, where attackers embed a hidden functionality into across numerous applications and put them at risk. What
a pre-trained model, allowing them to manipulate outputs makes supply chain compromises particularly nefarious is
under specific conditions or run arbitrary code when the that they have the potential to infiltrate AI infrastructure
model is loaded. and avoid detection until serious harm occurs.

Some AI applications rely on models trained by third AI-Specific Attack Vectors


parties and made available through open-source
repositories like Hugging Face, PyTorch Hub, or Direct Compromise of AI Infrastructure
TensorFlow Hub. A survey of IT decision makers revealed
that around 60 percent of respondents use open-source Direct prompt injection is a technique used to manipulate
ecosystems as an AI tool source, and 80 percent of model responses through specific inputs to alter its
respondents note that at least a quarter of their company’s behavior and circumvent an AI model’s built-in safety
AI solutions or platforms are based on open source. While measures and guardrails, usually to re-task an LLM or LLM
open-source repositories have security checks, attackers application to conduct some other task. These can either
remain savvy enough to avoid detection, and organizations be intentional (i.e., a malicious attempt to exploit the
risk installing those malicious components. model) or inadvertent (i.e., a user providing input that
triggers unexpected behavior).
Case Study: Sleepy Pickle
Jailbreaking is a specific direct prompt injection
In our June 2024 AI Threat Roundup blog, we covered technique where an attacker provides inputs that cause
Sleepy Pickle, a technique shared on the Trail of Bits blog the model to disregard its alignment or safety protocols
that enables adversaries to directly and discreetly entirely, particularly in chatbots. LLMs such as chatbots
compromise a model itself. are often designed with guardrails to prevent them from
generating harmful, unethical, or illegal outputs. Still
Pickle is a common Python serialization format in machine attackers can implement adversarial prompts or inputs to
learning with well-understood security risks. Adversaries circumvent these restrictions. Jailbreaking can also
can insert malicious code into pickle files overwrite or reveal the underlying system prompt (i.e., the
to deliver payloads after distribution and deserialization. initial set of instructions given to an AI model that defines
Instead of distributing malicious models, Sleepy Pickle its core behavior, capabilities, constraints, and
executes a custom function to compromise the model after personality). When system prompts are revealed, attackers
deserialization. This delay makes the technique dangerous, can more effectively craft prompts to bypass the model's
customizable, and more difficult to detect. safety measures and behavioral guardrails or identify and
exploit vulnerabilities in how the model processes
instructions.

© 2025 Cisco and/or its affili es. All rights reserved. Cisco State of AI Security Report 5
Early jailbreaking attempts often relied on direct instruction
manipulation, such as asking the model to “pretend” or
“roleplay” scenarios that would normally be restricted.
However, as models became more robust to these
basic approaches, adversarial techniques grew more
sophisticated. Additional advanced jailbreaking techniques
now include token smuggling, where malicious instructions
are encoded within seemingly benign prompts; adversarial
prompting, where attackers craft carefully worded prompts
designed to trick a model into ignoring its guardrails; and
context contamination, where the model’s context window
is deliberately filled with content intended to alter its
behavior. Despite advances in jailbreaking defenses, Cisco
research has revealed that simple jailbreaks continue to be
effective against advances in AI safety.

Indirect Prompt Injection


While direct prompt injection attacks involve entering
text prompts that lead to unintended actions, indirect
prompt injection attacks focus on providing compromised
source data, such as malicious PDFs or web pages, or
even non-human-readable text (e.g., binary, base64), to
inject malicious instructions to manipulate LLM responses.
Indirect prompt injections are more difficult to detect
because the attack does not require direct access to an
AI model, meaning they can bypass traditional prompt
injection defenses, and the threat can persist in systems
over time.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 6
Training Data Extraction and Tampering
AI models often process and store vast amounts of data,
making them attractive targets for data exfiltration,
tampering, and unauthorized access. Training state-of-
the-art LLMs requires trillions of tokens of contextual
information throughout their training lifecycle, and deep
learning model architectures can memorize their training
data. Security researchers have hypothesized that models
have the potential to reveal their training data and
demonstrated numerous scenarios that can result in
training data extraction. Attacks targeting the extraction
of training data from deployed AI models risks revealing
sensitive or confidential information that was used to train
the model.

Cisco’s AI research has also revealed the capability to


extract memorized training data through a simple method
that tricks a chatbot into regurgitating individual sentences
in news articles, allowing us to reconstruct portions of the
source article. If methodologies such as these prove
replicable at scale, the data privacy and security
implications are widespread, especially when AI models
are trained on proprietary or private information.
Organizations could face a complete loss of information
privacy, loss of proprietary data and intellectual property,
or violations of copyright or fair use principles, and face
consequences such as financial loss, reputational damage,
and privacy violations.

Attackers can also tamper with data used by AI models,


compromising the integrity of the model's outputs and
potentially leading to incorrect decisions or harmful
actions. Setting inappropriate or overly lenient privileges
may also compromise access to AI models and allow
attackers access to sensitive data or infrastructure.

Figure: Reference article (top) and our LLM prompting


flow to extract training data (middle) and our results
(bottom)

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 7
Data Poisoning Campaigns Case Study: Talos Research on
Data poisoning is when threat actors inject malicious Malicious LLMs
samples into training datasets to introduce weaknesses or
backdoors into AI models, enabling them to influence the Cybercriminals that cannot or do not wish to bypass
data that the model produces, engage in criminal security built into legitimate LLMs sometimes opt to build
operations, or gain unauthorized access. Researchers their own. Cybercriminal-designed LLMs do not include
have also demonstrated the capability to poison AI-based any of the restrictions against malicious use. In fact, some
malware detection technology, causing the model to of these LLMs are specifically designed to facilitate
misclassify malware samples as benign. Financial services criminal activity, including applications like GhostGPT,
organizations can face similar challenges in their fraud DarkBard, DarkGPT, and FraudGPT. Most of these LLMs
detection models if attackers can access fraud detection are advertised for sale to cybercriminals on hacking
models, alter the system’s training dataset, and shift its forums, Telegram channels (a social media and messaging
decision boundary. application where illicit activity often occurs), and also on
the dark web, costing as little as $75 per month.
Model Extraction and Model Inversion Cisco Talos has observed cybercriminals conducting
A model extraction attack is a type of attack where phishing attacks with the assistance of LLMs to generate
an attacker tries to steal or duplicate a machine learning more authentic, customized phishing message content,
model by repeatedly querying it and using the responses which can also increase the likelihood of bypassing email
to train their own copy. Similarly, a technique called model security filtering. Some malicious LLM apps also advertise
inversion, where attackers repeatedly query the model features such as:
and iterate on its outputs to gather more information,
could allow attackers to reconstruct training data by • Malicious code obfuscation
exploiting the model's learned parameters and outputs.
Both techniques can potentially expose sensitive training • Exploit code generation
data or disclose detailed patterns about a model from
private training data. • Scanning sites for known vulnerabilities

• Checking the authenticity of credit card numbers


How Threat Actors Leverage AI as a Tool
for Cyber Attacks • Outbound email sending capability

Generative AI is powerful and has a staggering potential to • API access for automation of these tasks
influence the threat landscape, but in 2024, threat actors’
use of AI did not significantly enhance attackers’ tactics, Figure: Screenshot of a cybercriminal LLM
techniques, and procedures (TTPs). Although threat actors (DarkGPT) dashboard
have the potential to harness AI and develop novel
capabilities, we have not yet observed those capabilities
deployed at scale in-the-wild. In the meantime, we have
observed both state-sponsored adversaries and
cybercriminals use of AI for social engineering and
influence operations, and task automation and other
productivity improvements in the threat actors’ attack
lifecycle.

Generative AI for Social Engineering


The accessibility of generative AI tools, such as large
language models (LLMs) and deepfake technologies, has
led to a surge in sophisticated social engineering attacks,
but this increase can be broken down into two distinct
parts: the use of AI for social engineering and the use of AI
for automating malicious activities. By combining these
two components, attackers can increase their success
rates exponentially, as they can produce higher volumes
of socially engineered lures of higher quality with the
assistance of LLMs and generative AI.

As such, we expect phishing and other social engineering


techniques such as vishing (AI-generated voice cloning)
and deepfakes to continue improving with AI’s assistance,
while spam and phishing detection races to catch up.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 8
State-sponsored advanced persistent threat (APT) groups Task Automation and Productivity Gains
and other sophisticated actors may leverage aspects of
these features, such as deepfake video and audio and in the Attack Lifecycle
supporting materials (e.g., resumes, cover letters) for
conducting interviews or phone calls or automating social Threat actors have attempted to leverage chatbots to
engineering. Governments such as North Korea have assist in malware development and task automation to
explicitly stated their intention to develop AI capabilities, improve their attack success rates. For example, as a
though no direct evidence or open sources have indicated summation tool, malicious actors have queried chatbots to
that the country’s cyber forces have applied AI or ML to gather open-source intelligence on their targets.
enhance its offensive cyber programs. Other organizations
have observed that North Korean-affiliated actors Research has proven that LLMs can exploit one-day
attempted to use chatbots to debug their malicious code. vulnerabilities (i.e., vulnerabilities that have been disclosed
but not patched in a system). Threat actors have
In 2024, cybercriminals leveraged these technologies to leveraged LLMs to assist with basic scripting tasks and
create convincing phishing campaigns and manipulate code debugging. For example, there is evidence to
individuals into divulging sensitive information or granting suggest that accounts originating in China are leveraging
unauthorized access to their organization’s networks and chatbots to debug code related to communications
systems. For example, the cybercriminal threat actor group surveillance technology, among other activities. But we
Scattered Spider has successfully used AI voice cloning to have not yet observed threat actors deploying an
conduct vishing attacks against numerous sectors, advanced capability for vulnerability scanning and
including healthcare. Using voice samples from corporate exploitation in real-world scenarios.
videos and social media, they generated convincing voice
clones of executives to authorize security changes and Cybercriminals have developed and sold multiple tools
network access requests. Criminals have also leveraged AI that can aid in vulnerability research, reconnaissance,
to bypass regulations and know-your-customer practices exploit writing, and task automation. Cybercriminals also
for cryptocurrency organizations. take advantage of AI-powered agents to mimic human-
like behaviors that bypass bot detection (e.g., random
mouse movements, real-time form completion) and fraud
Threat actors have also leveraged chatbots to generate
detection techniques (submitting micro-transactions to
content in non-native languages to conduct influence
validate card details).
operations. Examples include either translating or
optimizing content in a targeted language for social media
posts, short articles, and longform articles on topics such
as geopolitical conflict, criticism of United States and
European policy, or security-related content.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 9
As agentic systems increasingly integrate with disparate
Looking Ahead: New and Improved services and vendors, the opportunity for threat actor
AI Threat Vectors exploitation or vulnerability is ripe. Attackers could
potentially leverage agentic systems to conduct multi-
Agentic AI, “AI systems and models that can act stage attacks, find creative ways to access restricted
autonomously to achieve goals without the need for data systems, chain seemingly benign actions into
constant human guidance,” and has the capability to harmful sequences, or learn to evade detection by
conduct planning and reasoning, to memorize and recall network and system defenders.
information, and to take action and use tools to accomplish
tasks, all of which could reap productivity benefits and Continued social engineering at scale: From social
unlock new insights for organizations. engineering to propaganda proliferation, cybercriminal
and state-sponsored actors will continue to leverage AI
Additional Resources: OWASP Guide to technologies to improve the personalization and
Agentic AI Threats professionalization of their malicious activities. While not
realized yet, malicious use of multimodal AI, which
The international web security nonprofit OWASP released integrates text, images, voice, and sophisticated coding,
the first version of their guide to Agentic AI threats in could enable attackers to streamline and automate entire
February 2025. As agentic systems continue to evolve and attack chains. Theoretically, these attacks could conduct
become more sophisticated, so too does their risk profile. reconnaissance on targets, craft realistic phishing
This document from the OWASP Agentic Security Initiative content, find zero-day exploits, generate evasive
(ASI) provides a reference of emerging agentic threats malware, and automate lateral movements within
while simultaneously suggesting practical mitigation networks, leading to faster exploitation and increased risk
strategies. Cisco is a proud contributor to and supporter of across both the public and private sectors.
this guide.
Numerous areas of risk could emerge in the development
Agentic AI systems could also imperil organizations that of capabilities targeting AI models and systems
are neither prepared nor equipped to handle agentic themselves, including using adversarial inputs to trick AI-
systems and their potential for compromise. At least 14 powered security filters, hijacking AI agents used in
distinct threat vectors have been identified with agentic business operations workflows, as well as attacking
systems, including: memory poisoning, where false or elements of the AI supply chain (e.g., corrupting training
misleading data is introduced into an AI’s memory systems data, compromising a model’s cloud infrastructure).
to exploit the agent’s context; misaligned and deceptive Traditional cyber attacks against AI systems (as well as AI
behaviors, where an AI agent is used to conduct harmful or laboratories and developers) will remain a salient threat
disallowed actions; and unexpected remote code as attackers seek to conduct intellectual property theft,
execution and code attacks, where attackers inject user data theft, or disrupt, degrade, or destroy elements
malicious code or execute unauthorized scripts. of the AI development lifecycle.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 10
Developments in AI Policy

Overview
A significant number of new AI policy developments occurred The following sections are only intended to be a snapshot
in 2024, largely in response to the increasing prevalence of of trends seen in 2024 and do not account for all AI policy
AI-powered technologies and their market expansion. In the developments, both domestically and internationally.
United States alone, state lawmakers introduced over 700 Given the rapid evolution of the AI regulatory landscape,
AI-related bills—113 of which were enacted into law— changes to the below efforts may have occurred since the
across 45 states in 2024. The pace of policy activity has publication of this report. The information provided in this
not slowed in 2025. Within the first couple of weeks of report is meant to be a helpful resource only and is not
2025, 40 AI-related bill proposals have been introduced at intended to constitute legal advice.
both the state and federal levels. The swift and complex
nature of these changes has presented challenges to Domestic AI Policy Developments
players across the market navigating the evolving
landscape. in 2024
AI introduces social and economic risks alongside potential Fragmented State-by-State Legislation
substantial economic growth opportunities, challenging In the absence of federal policies on AI, states have taken
jurisdictions to balance the desire to foster innovation independent action to regulate the technology. A flurry of
against managing associated risks. As countries around new bills introduced at the state level put some
the world develop and implement AI legislation and restrictions on AI development and use.
regulations, no one standard approach to regulating AI has
emerged. In their efforts to respond to both the challenges · Colorado became the first state to pass a comprehensive
and opportunities brought by AI, governments have drawn AI Act (SB 24-205). The bill requires developers and
on a wide-ranging AI policy toolkit: drafting deployers of “high-risk” AI systems to comply with
comprehensive laws, regulations for specific use-case additional precautionary measures to ensure they avoid
applications, national AI strategies, and voluntary discrimination and other safety harms. The new law, part of
guidelines and standards. We have observed that AI Colorado’s Consumer Protection Act, mirrored the risk-
governance often begins with the rollout of a national based approach of the recently passed EU AI Act.
strategy before moving towards legislative action.
· Utah AI Policy Act bill (SB 149) came into effect on May 1,
Highlights of global developments in AI policy throughout 2024. This legislation is part of Utah’s consumer protection
2024 include: laws and introduced disclosure obligations for the use of
generative AI systems in both the private and public sectors.
· Country-level focus on promoting AI safety amidst In addition, it established the Office of AI Policy and the AI
rapid technological developments, through actions such as Learning Laboratory Program, with the potential to establish
AI Safety Summit voluntary commitments, as well as cybersecurity auditing procedures for higher risk AI
transatlantic and global partnerships; applications.

· Domestically, a fragmented state-by-state AI · States such as Connecticut, Maryland, Vermont, and


legislation approach has emerged in the absence of Virginia mandated that state agencies conduct impact
federal-level action; and assessments to test AI systems for safety risk.

· European Union AI Act officially entered into force on


August 1, 2024, meaning Europe is now enforcing the
world's first comprehensive AI law.

In 2025, early actions suggest the focus of governments


has shifted to place greater emphasis on security and AI
innovation. This recent shift is exemplified by President
Trump's focus on national security implications of AI and
creating an enabling environment for development and
adoption of AI. The AI Action Summit held in Paris in
February 2025, which brought together Heads of State,
government officials, and leaders of international
organizations, similarly demonstrated growing support for a
pro-innovation environment. French and British leaders in
particular highlighted the need for greater investments in
AI infrastructure.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 11
Federal Interest in AI Safety and Security Importance of AI Security Standards
In 2024, there were various efforts across federal agencies This past year there was a lot of activity around the
to promote safe and secure AI development and use. development of AI security standards, providing
organizations guidance on how to secure AI applications.
· The Department of Justice leveraged existing statutes to
· The National Institute of Standards and Technology
seek harsher sentences for certain crimes involving the
(NIST), an agency of the U.S. Department of Commerce
misuse of AI.
that promotes domestic innovation by advancing
measurement science, standards, and technology,
· The bipartisan House Task Force on AI issued a
published its Adversarial Machine Learning (ML)
comprehensive report on AI including guiding principles and
Taxonomy. This resource, which is co-authored by
forward-looking recommendations to advance America’s
members of the Cisco AI Defense team, provides a
leadership in AI innovation responsibly.
conceptual hierarchy of attack lifecycles, attacker goals
and objectives, and attacker capabilities. In addition, it
· The Department of Commerce launched the
suggests corresponding methods for mitigating and
U.S. AI Safety Institute Consortium (AISIC). The National
managing the consequences of attacks.
Institute of Standards and Technology (NIST) launched the
consortium to: “[establish] guidelines and processes to · MITRE, a non-profit organization bridging public and
enable developers of generative AI to conduct AI red- private sectors through federally funded research centers,
teaming tests to enable deployment of safe, secure, and extended their Adversarial Threat Landscape for AI
trustworthy systems.” Systems (ATLAS) framework to cover generative AI
systems. The ATLAS matrix is a living community
· The U.S. Department of the Treasury released a report on knowledge base of adversarial tactics and techniques
managing AI-specific cybersecurity risks in the financial based on real-world attack observations. It’s a resource
services sector. In the report, “significant opportunities and used by security professionals, developers, and operators
challenges that AI presents to the security and resiliency of protecting AI-enabled systems.
the financial services sector.”

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 12
· In a landmark agreement, the UK and US AI Safety
International AI Policy Developments Institutes committed to a partnership to jointly test AI
in 2024 models and share frameworks, best AI safety practices, and
expertise.
Transnational Partnerships
· A second Safety Summit was hosted in Seoul, Korea in
In 2024, transnational partnerships were the primary policy May 2024, successfully securing commitments from sixteen
vehicle to promote safe and secure AI development and companies at the forefront of AI development to share risk
use globally. and safety frameworks and avoid high-risk models.

· The United Kingdom (UK) and Canada signed an · The UN unanimously adopted a US-led resolution on
agreement to work closely together on AI safety. As part of AI technologies. The draft resolution aims to lay out a
the agreement, the two countries agreed to share expertise comprehensive vision for “safe, secure, and trustworthy AI”
to enhance evaluation and testing work and “inspire and is based on the voluntary commitments put forth by
collaborative work on systemic safety research,” with an President Biden’s administration in partnership with leading
eye toward growing the network of AI safety institutes AI companies last fall. This marked a critical step towards
following the first AI Safety Summit in Bletchley in 2023. establishing international agreement on guardrails for the
ethical and sustainable development of AI. At its core, the
· EU and US AI experts from the EU-U.S. Trade and resolution encourages protecting personal data, monitoring
Technology Council developed an updated edition of the AI AI for risks, and safeguarding human rights.
Taxonomy and Terminology. This taxonomy helps to align
international governance efforts and creates a shared · Japanese Prime Minister Kishida Fumio announced the
understanding of how to effectively secure AI systems. The launch of the Hiroshima AI Process Friends Group at an
joint council also announced a new research alliance: AI for Organization for Economic Cooperation and Development
Public Good, focused on applying AI systems to the most gathering. The initiative, supported by 49 countries and
important global challenges. regions, aims to align global efforts on safe, secure, and
trustworthy generative AI. This initiative supported the
implementation of international guidelines as outlined in the
Hiroshima AI Process Comprehensive Policy Framework.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 13
National and Regional AI Governance • Singapore released its Model AI Governance Framework
for Generative AI, providing a voluntary framework for
In 2024, the EU AI Act became the world’s first organizations to adopt while deploying AI systems to meet
comprehensive AI law to come into force, while other best practices for AI risk management.
countries took national approaches to AI governance.
• Early in 2024, Japan signaled they were heading toward
• EU AI Act officially entered into force on August 1, 2024, the development of new legislation to regulate AI. Two
and outlines regulations on AI development, deployment, publications, from the Liberal Democratic Party and the
and use, imposing stricter rules on high-risk AI systems (as Japanese Cabinet Office’s AI Strategy team, recommended
stipulated on page 127 of the official EU AI Act text) and introducing regulations for large-scale foundation models.
banning "unacceptable" AI applications, with penalties for However, by the end of the year Japan’s attitude shifted
non-compliance up to 7% of an organization’s total towards a ‘light touch’ regulatory approach. As stipulated by
worldwide turnover. a second AI white paper, Japan aims to become the “most
AI-friendly country” by adopting principles from the
• The Australian Government released a new policy Hiroshima AI Process and “consider minimum necessary
for the responsible use of AI in government. The policy measures through legal regulations.”
positions the government to play a “leadership role in
embracing AI for the benefit of Australians while ensuring its Looking Ahead: Direction for AI
safe, ethical and responsible use, in line with community
expectations.” The policy is mandatory for non-corporate Policy 2025
Commonwealth entities and took effect on September 1,
This year’s AI policy developments have already signaled a
2024.
significant shift in the direction that emerging regulation is
headed, marking an evolution of the AI policy conversation
• There was a push in Africa to start regulating AI, as the use
toward effectively balancing the need for AI security with
of AI systems has been expanding across the continent. The
accelerating the speed of innovation and increasing
African Union, including 55 member nations, began
investment in AI infrastructure.
preparing an AI policy to further develop and regulate the
use of AI. However, there was ongoing debate about
whether regulation is warranted and the impact it might have In 2024, policymakers were primarily concerned with AI
on innovation. Seven African nations have already developed safety and mitigating any social and economic harm
national AI policies, and in February of last year the African associated with the use of AI. The AI safety conversation will
Union Development Agency published a policy draft to serve likely continue to be relevant for policymakers’ approach to
as the blueprint of further AI regulations by African nations. regulations in 2025 but addressing security-related risks and
supporting pro-innovation policy are clear priorities.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 14
• The Trump Administration takes action to support AI • The UK published its AI Opportunities Action Plan: The UK
innovation and protect national security: In the opening government detailed a slew of policy objectives, ranging from
days of his presidency, President Trump revoked President investments in infrastructure to fostering the development of
Biden’s AI Executive Order, and shortly thereafter announced UK Sovereign AI, further indicating a greater focus towards AI
a new one which the Administration positioned as fostering opportunity and growth. The three key categories of
innovation, supporting economic growth, and protecting recommendations include: laying the foundations to enable AI,
national security. This position was buttressed by Vice changing lives by embracing AI, and securing their future with
President JD Vance’s speech at the AI Action Summit, homegrown AI.
outlining the U.S. Administration’s priority of harnessing AI
innovation. The U.S. government is also increasingly • The Indian Ministry of Electronics and Information
concerned about the potential export of foundational Technology (MEITY) is seeking input on AI governance
technologies that may provide a technological advantage to guidelines: MEITY published a report on AI Governance
foreign adversaries in their development of AI. Guidelines Development, on January 6, 2025, seeking
comments from stakeholders. The governance guidelines
• The United States and UK decline to sign the 2025 AI adopt a risk-based approach and align closely with the OECD
Action Summit’s declaration on safety: The United States AI principles. In addition, the report recommends establishing
(along with the UK) declined to sign the Summit’s a technical advisory body to serve a similar role as an AI
declaration on safety, citing concerns over global safety institute.
governance and national security.
• South Korea signed an AI Framework Act into law: This
• French President Emmanuel Macron urges EU to simplify makes South Korea the second jurisdiction, following the EU,
regulatory efforts: While hosting the AI Action Summit in to enact a comprehensive regulatory AI law. It adopts a risk-
Paris, President Macron proposed a lighter approach to AI based approach, focusing on ‘high-impact’ AI systems. High-
regulation in Europe to boost member states’ impact AI, in this context, refers to AI systems that pose risks
competitiveness in the global AI race. to human life, physical safety, and fundamental rights.

• The UK rebrands its AI Safety Institute to focus on • Japan announces plans for AI Act: Japan introduced a
security: The UK AI Safety Institute officially rebranded as draft AI Act bill to its Parliament. The proposal does not take a
the UK AI Security Institute, signaling an increased focus on strict regulatory approach and does not include penalties for
combatting the use of AI to facilitate crime and threaten non-compliance. Instead, the bill focuses on operationalizing
national security. the Hiroshima Process Principles, supporting R&D and
empowering the government to investigate malicious uses of
• European Commission withdraws AI Liability Directive: AI that are not covered by existing legislation.
The European Commission formally abandoned the 2022 AI
Liability Directive, which aimed to “[lay] down uniform rules • AI security standards update to reflect new and
for certain aspects of non-contractual civil liability for damage emerging risks: The Open Worldwide Application Security
caused with the involvement of AI systems.” According to a Project (OWASP), a well-recognized global non-profit
press release on the Commission’s newly adopted 2025 organization that works to improve web-application and
work program, this decision was motivated by efforts to software security, released an updated version of the "Top
“reduce administrative burden and simplify EU rules.” Ten for Large Language Model Applications for 2025,”
including new additions like ‘misinformation’ and ‘vector and
• The European Union and France announce significant
investment plans for AI: During the AI Action Summit in embedding weaknesses,’ and announced a new
Paris, European Commission President Von der Leyen “Generative AI Red Teaming Guide.” These resources will be
announced the InvestAI plan which will seek to mobilize up to leveraged by organizations and governments to better
EUR200B in investments in AI infrastructures and four understand the AI security landscape and best practices.
gigafactories. France President Macron announced more
than EUR109B in private investments in AI in France.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 15
AI Security Research

Overview
Over the last year, Cisco’s AI security research team has • Automatic: Manual inputs and human supervision aren’t
led and contributed to several pieces of groundbreaking necessary.
research in key areas of AI security. These efforts reflect
our commitment to advance the AI security community • Black box: The attack doesn’t require knowledge of the
while simultaneously ensuring our customers are LLM architecture.
protected against novel threats and emerging
vulnerabilities. • Transferable: Prompts are written in natural language
and can be reused.
This section provides a high-level overview of our
methodologies, key findings, and real-world implications • Prompt efficient: Fewer prompts make attacks more
of Cisco’s various AI security research initiatives, including: discreet and harder to detect.
• Algorithmic jailbreaking attacks models with zero The success of TAP against sophisticated models like
human supervision, enabling adversaries to automatically GPT-4 and Llama 2 also demonstrates the relatively
bypass protections for even the most sophisticated LLMs. low cost of algorithmic jailbreaking and suggests that
This method can be used to exfiltrate sensitive data, disrupt more capable LLMs can oftentimes be easier to break.
services, and harm businesses in other ways.

• Fine-tuning models can break their safety and security


alignment, meaning that improved contextual relevance for
AI applications can inadvertently make them riskier for
enterprise use.

• Simple methods for poisoning and extracting training


data demonstrate just how easily the data used to train an
LLM can be discreetly tampered with or exfiltrated by an
adversary.
Table: Fraction of jailbreaks achieved as per the GPT4-
As AI itself and the threats to AI systems continue to evolve
Metric. For each method and target LLM, we report the
rapidly, we combine findings from this first-party research
fraction of jailbreaks found on AdvBench Subset by the
with our third-party threat intelligence pipeline to deliver AI
GPT4-Metric and the number of queries sent to the target
protections that are relevant and resilient.
LLM in the process. For both TAP and PAIR we use
Algorithmically Jailbreaking Large Vicuna-13B-v1.5 as the attacker. Since GCG requires
white-box access, we can only report its results on open-
Language Models sourced models. In each column, the best results are
bolded.
To govern model behavior and prevent malicious, sensitive,
or otherwise harmful outputs, developers add safety and
For organizations exploring potential business applications
security guardrails to their LLMs. While these boundaries
for AI, this research reaffirms the importance of
are important, they are not infallible. Model jailbreaks
independent security measures that are more resilient
undermine these protections and coerce models to
than built-in guardrails and protect LLMs in real-time.
produce restricted outputs.

Cisco AI researchers, working in collaboration with


researchers from Yale University, developed an algorithmic
method for jailbreaking LLMs known as the Tree of Attacks
with Pruning (TAP). TAP uses two LLMs—an attacker model
and an evaluator model—to create and continuously refine
harmful prompts. The research highlights several reasons
why algorithmic jailbreak methods like TAP are particularly
damaging and difficult to mitigate:

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 16
Applying Algorithmic Jailbreaking to Fine-Tuning Breaks Internal Model
Frontier Reasoning Models Guardrails
The emergence of advanced reasoning models like Fine-tuning foundational models is a common approach
OpenAI o1 and DeepSeek R1 prompted AI researchers businesses employ to improve the accuracy, domain
from Cisco and the University of Pennsylvania to develop expertise, and contextual relevance of an AI application in
Adversarial Reasoning. This automated approach to model a flexible and cost-effective way. However, research by
jailbreaking uses advanced model reasoning to effectively the Cisco AI team reveals a danger to fine-tuning that is
exploit the feedback signals provided by an LLM to bypass often overlooked—namely, that fine-tuning can throw off
its guardrails and execute harmful objectives. model alignment and introduce new safety and security
risks.
Adversarial Reasoning was instrumental for the Cisco
security evaluation of DeepSeek R1 which revealed a This phenomenon is broadly applicable and can even
concerning 100% attack success rate (ASR). In a broader occur with completely benign datasets, making fine-tuned
sense, this research suggests that future work on model AI applications generally easier to jailbreak and more likely
alignment must consider not only individual prompts but to produce harmful or sensitive results. Specifically, this
entire reasoning paths to develop robust defenses for AI research was conducted using Llama-2-7B and three
systems. AdaptLLM chat models fine-tuned and released by
Microsoft researchers to cover the domains of
biomedicine, finance, and law.

Evaluations found fine-tuned variants more than 3 times


more susceptible to jailbreak instructions and over
22 times more likely to produce a harmful response
than the original foundation model. The purpose of this
research is not to disparage fine-tuning entirely, but rather
to highlight that fine-tuning can introduce new dimensions
of risk to even the most well-aligned foundation model. It
emphasizes the need for an independent safety and
security layer that can protect the model without being
impacted by fine-tuning.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 17
Training Data Extraction via
Decomposition
Chatbots will typically refuse to answer prompts that
attempt to reconstruct copyrighted or paywalled data
because the underlying models are trained with specific
guidelines and restrictions on reproducing copyrighted or
paywalled materials verbatim. However, Cisco AI
researchers were able to leverage a simple method to trick
chatbots into regurgitating portions of news articles,
allowing for reconstruction of the source material and
raising concerns about greater information security risks
such as the extraction of sensitive, proprietary, or non-
public information.

With a method known as decomposition, researchers


would break the primary objective—extraction of private
training data—into smaller, successive requests that could
bypass the model’s internal guardrails. This was run
against two frontier LLMs for a corpus of 3,723 New York
Times articles and 1,349 Wall Street Journal articles
published between 2015 and 2023. Researchers were
able to retrieve at least one verbatim sentence from 73
NYT articles for LLM-α and 11 articles for LLM-β. They re-
ran prompts against the top 100 performing articles to
successfully reconstruct over 20% of the text from six
articles from LLM-α and two articles from LLM-β.

These results demonstrate that this decomposition method


can successfully induce the chatbot to generate texts that
are reliable reproductions of news articles, meaning that
they likely originate from the source training dataset. If this
methodology proves replicable at scale, the data privacy
and security implications are widespread—from a complete
loss of information privacy to violations of copyright.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 18
Recommendations for Implementing AI Security

AI applications have additional security considerations • Prioritize security in areas where adversaries
compared to traditional web applications, which can feel seek to exploit weaknesses. Equipped with a deeper
like entirely new territory and overwhelm enterprise understanding of the AI security threat landscape, prioritize
security teams. We would like to make this enormous new your defenses, institute controls, and harden your
threat landscape easier to grasp by highlighting the technological assets where you know adversaries and
commonalities between AI security and traditional criminals are targeting.
cybersecurity practices.
• Educate your workforce in responsible and safe AI
Each business will have to tailor its AI security strategy usage. As with any new technology, employee misuse or
around distinct implementation parameters. For example, misunderstanding of AI can be a tremendous source of
what models and datasets are you leveraging? What is the organizational risk. Clearly communicate internal policies
specific AI use case? How sensitive is the data being around acceptable AI use within legal, ethical, and security
handled? What end users does this AI application serve? boundaries to mitigate risks like sensitive data exposure.
While these are unique aspects, we outline some general
considerations and recommendations for all businesses AI security can still feel like an overwhelming challenge for
defining their AI security strategies below. most businesses: a dynamic threat landscape, evolving
standards, and new pieces of legislation—not to mention
• Manage risk at every point in the AI lifecycle. As
breakthroughs in AI technology itself—can be difficult to
outlined in our threat intelligence section, there is a degree
track and reflect organizationally. That’s why partnering
of risk at virtually every step of the AI lifecycle from
with the right vendors and investing in purpose-built AI
development to deployment. Ensure your security team is
security solutions is important. Cisco introduced AI
equipped to identify and mitigate these in every phase:
Defense precisely for this reason; with a straightforward
supply chain sourcing (e.g., third-party AI models, data
solution for managing AI risk from development to
sources, and software libraries), data acquisition, model
deployment, businesses can focus their efforts on
development, training, and deployment.
breakthrough AI applications knowing security is covered.
• Maintain familiar cybersecurity best practices. AI may
be new and unique, but familiar concepts like access
control, permission management, and data loss prevention
remain critical. Approach securing AI the same way you
would secure core technological infrastructure and adapt
existing security policies to address AI-specific threats.

• Uphold AI security standards throughout the AI


lifecycle. Consider relevant legislation; refer to resources
and frameworks like the NIST AI Risk Management
Framework, OWASP Top 10 vulnerability lists, and the
MITRE ATLAS matrix to assist in managing risk at your
organization. Apply these best practices to your AI
development and deployment processes.

• Determine risk thresholds for AI in your organization.


Consider how your business is using AI and implement risk-
based AI frameworks to identify, assess, and manage risks
associated with these applications. Clearly communicated
thresholds ensure all stakeholders have a shared
understanding for when to accept or reject any risks and
issues that arise from the deployment of AI technologies.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 19
AI Security at Cisco

Cisco is building on decades of leadership in networking • Protecting Cisco Secure Endpoint and Email Threat
and cybersecurity to pave the way for rapid AI innovation Protection customers from malicious AI supply chain
and resilient AI security. In 2024 alone, we made artifacts downloaded from Hugging Face, shared via
tremendous progress integrating new capabilities into our email, or downloaded from a shared drive for customers
existing portfolio and launched the first truly using Cisco Secure Endpoint and Cisco Secure Email
comprehensive solution for enterprise AI security: Cisco Threat Defense.
AI Defense.
This State of AI Security report validates that the AI
At a high level, Cisco AI Defense addresses the two
landscape has and continues to evolve rapidly. As we
primary areas of enterprise AI risk. The first is risk of
sensitive data exposure from employees using third- drive towards future breakthroughs in AI technology and
party systems and sharing intellectual property, PII, and applications, Cisco remains committed to AI security
other confidential information with these tools. The through our contributions to the community and cutting-
second is risk for businesses developing and deploying edge solutions for customers pushing the envelope of AI
their own AI applications. Vulnerabilities exist all innovation.
throughout the AI development lifecycle; businesses
creating AI applications need to ensure that these
systems are safe and secure for customers.

Bringing AI Defense to the market is just one part of our


ongoing commitment to fostering a safer, more secure
future for enterprise AI. Here are a few other examples
from the past year of ways we’re protecting AI and using
AI to enhance our broader security portfolio.

• Using AI to enhance Cisco Secure Email Threat


Defense by processing and accurately classifying
malicious business email compromise (BEC) attacks—
one of the fastest-growing and most financially
damaging cyber threats, according to the FBI.

• Safeguarding companies from the security risks of


third-party AI applications with Cisco Secure Access,
protecting against threats and sensitive data loss while
restricting employee access to unsanctioned tools.

• Enabling security analysts to work faster and


smarter using AI capabilities in Cisco Extended
Detection and Response (XDR) that streamline resource-
intensive tasks like security event correlation, incident
summarization, and reporting.

• Bolstering Cisco Secure Firewall with AI capabilities,


like Encrypted Visibility Engine (EVE), which uses
machine learning to identify traffic without having to
decrypt it, and the AI Assistant, which simplifies tasks
like policy identification, troubleshooting, and lifecycle
management.

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State
State of AI Security Report
Report 20
Contributors

Emile Antone (Product Marketing Manager, Cisco)


Lead Contributor

Amy Chang (AI Researcher, Cisco)


Lead Contributor

Alie Fordyce (Engineering Product Manager, Cisco)


Lead Contributor

Mark Loewenstein (Product Marketing Leader, Cisco)

Paul Kassianik (AI Researcher, Cisco)

Adam Swanda (AI Researcher, Cisco)

Hyrum Anderson (Director of Software Engineering, Cisco)

© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 21

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy