The State OF AI Sercurity
The State OF AI Sercurity
Contents
Executive Summary 3
The AI Threat Landscape 4
Overview 4
Developments in AI Policy 11
Overview 11
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 2
Executive Summary
Artificial intelligence (AI) has emerged as one of This is Cisco’s inaugural State of AI Security report. Its aim is to
provide a comprehensive overview of important developments in
the defining technologies of the 21st century. It
AI security across several key areas: threat intelligence, policy,
has transformed both our personal and and research. We’ll reflect on progress from the past year while
professional lives, and its rapid advancement will simultaneously looking at what’s ahead and highlighting the ways
continue to reshape the ways in which in which Cisco is investing in a safer, more secure future for AI.
Ultimately, we want to help our customers better understand the
businesses operate. Business leaders largely
AI landscape so that they might be better equipped to manage
recognize the generational opportunity that AI the risks and reap the benefits that AI brings.
presents and feel tremendous pressure to
harness this potential. Findings from our Cisco The State of AI Security report will cover:
2024 AI Readiness Index show that the race to · In-depth analysis of threats to AI infrastructure, AI supply
chains, and AI applications and evaluation of the implications AI
integrate AI into critical business functions is
threat vectors such as model backdoors, prompt injections, and
impeded by a few practical challenges—of which, data extraction.
AI security is the most prominent.
· Important developments in U.S. and international AI policy,
As AI systems handle increasingly sensitive workloads in vital highlighting common themes and macro trends from hundreds of
sectors such as healthcare, finance, and defense, the need for AI-related legislation, executive orders, partnership agreements,
robust safety and security measures becomes nonnegotiable. and security frameworks.
The threat landscape for AI is novel, complex, and not
effectively addressed by traditional cybersecurity solutions. · Original research into algorithmic jailbreaking, dataset
Similarly, streamlining the integration of AI capabilities while poisoning, data extraction, and several other cutting-edge AI
adhering to new compliance frameworks and regulations can security topics led by Cisco’s own AI research team.
make AI adoption feel overwhelming and costly. We are also excited to introduce Cisco AI Defense, the first truly
comprehensive solution for enterprise AI security. Announced in
January of this year, AI Defense builds on our decades of
networking and security experience to help enterprises protect the
development, deployment, and usage of AI across their
organizations.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 3
The AI Threat Landscape
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 4
Developers frequently integrate pre-trained models, Compromised machine learning libraries (e.g.,
software libraries, and datasets from external sources, TensorFlow and PyTorch have both been targets of
which can create several risks, such as backdoored attack) can introduce vulnerabilities that can manifest
models, where attackers embed a hidden functionality into across numerous applications and put them at risk. What
a pre-trained model, allowing them to manipulate outputs makes supply chain compromises particularly nefarious is
under specific conditions or run arbitrary code when the that they have the potential to infiltrate AI infrastructure
model is loaded. and avoid detection until serious harm occurs.
© 2025 Cisco and/or its affili es. All rights reserved. Cisco State of AI Security Report 5
Early jailbreaking attempts often relied on direct instruction
manipulation, such as asking the model to “pretend” or
“roleplay” scenarios that would normally be restricted.
However, as models became more robust to these
basic approaches, adversarial techniques grew more
sophisticated. Additional advanced jailbreaking techniques
now include token smuggling, where malicious instructions
are encoded within seemingly benign prompts; adversarial
prompting, where attackers craft carefully worded prompts
designed to trick a model into ignoring its guardrails; and
context contamination, where the model’s context window
is deliberately filled with content intended to alter its
behavior. Despite advances in jailbreaking defenses, Cisco
research has revealed that simple jailbreaks continue to be
effective against advances in AI safety.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 6
Training Data Extraction and Tampering
AI models often process and store vast amounts of data,
making them attractive targets for data exfiltration,
tampering, and unauthorized access. Training state-of-
the-art LLMs requires trillions of tokens of contextual
information throughout their training lifecycle, and deep
learning model architectures can memorize their training
data. Security researchers have hypothesized that models
have the potential to reveal their training data and
demonstrated numerous scenarios that can result in
training data extraction. Attacks targeting the extraction
of training data from deployed AI models risks revealing
sensitive or confidential information that was used to train
the model.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 7
Data Poisoning Campaigns Case Study: Talos Research on
Data poisoning is when threat actors inject malicious Malicious LLMs
samples into training datasets to introduce weaknesses or
backdoors into AI models, enabling them to influence the Cybercriminals that cannot or do not wish to bypass
data that the model produces, engage in criminal security built into legitimate LLMs sometimes opt to build
operations, or gain unauthorized access. Researchers their own. Cybercriminal-designed LLMs do not include
have also demonstrated the capability to poison AI-based any of the restrictions against malicious use. In fact, some
malware detection technology, causing the model to of these LLMs are specifically designed to facilitate
misclassify malware samples as benign. Financial services criminal activity, including applications like GhostGPT,
organizations can face similar challenges in their fraud DarkBard, DarkGPT, and FraudGPT. Most of these LLMs
detection models if attackers can access fraud detection are advertised for sale to cybercriminals on hacking
models, alter the system’s training dataset, and shift its forums, Telegram channels (a social media and messaging
decision boundary. application where illicit activity often occurs), and also on
the dark web, costing as little as $75 per month.
Model Extraction and Model Inversion Cisco Talos has observed cybercriminals conducting
A model extraction attack is a type of attack where phishing attacks with the assistance of LLMs to generate
an attacker tries to steal or duplicate a machine learning more authentic, customized phishing message content,
model by repeatedly querying it and using the responses which can also increase the likelihood of bypassing email
to train their own copy. Similarly, a technique called model security filtering. Some malicious LLM apps also advertise
inversion, where attackers repeatedly query the model features such as:
and iterate on its outputs to gather more information,
could allow attackers to reconstruct training data by • Malicious code obfuscation
exploiting the model's learned parameters and outputs.
Both techniques can potentially expose sensitive training • Exploit code generation
data or disclose detailed patterns about a model from
private training data. • Scanning sites for known vulnerabilities
Generative AI is powerful and has a staggering potential to • API access for automation of these tasks
influence the threat landscape, but in 2024, threat actors’
use of AI did not significantly enhance attackers’ tactics, Figure: Screenshot of a cybercriminal LLM
techniques, and procedures (TTPs). Although threat actors (DarkGPT) dashboard
have the potential to harness AI and develop novel
capabilities, we have not yet observed those capabilities
deployed at scale in-the-wild. In the meantime, we have
observed both state-sponsored adversaries and
cybercriminals use of AI for social engineering and
influence operations, and task automation and other
productivity improvements in the threat actors’ attack
lifecycle.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 8
State-sponsored advanced persistent threat (APT) groups Task Automation and Productivity Gains
and other sophisticated actors may leverage aspects of
these features, such as deepfake video and audio and in the Attack Lifecycle
supporting materials (e.g., resumes, cover letters) for
conducting interviews or phone calls or automating social Threat actors have attempted to leverage chatbots to
engineering. Governments such as North Korea have assist in malware development and task automation to
explicitly stated their intention to develop AI capabilities, improve their attack success rates. For example, as a
though no direct evidence or open sources have indicated summation tool, malicious actors have queried chatbots to
that the country’s cyber forces have applied AI or ML to gather open-source intelligence on their targets.
enhance its offensive cyber programs. Other organizations
have observed that North Korean-affiliated actors Research has proven that LLMs can exploit one-day
attempted to use chatbots to debug their malicious code. vulnerabilities (i.e., vulnerabilities that have been disclosed
but not patched in a system). Threat actors have
In 2024, cybercriminals leveraged these technologies to leveraged LLMs to assist with basic scripting tasks and
create convincing phishing campaigns and manipulate code debugging. For example, there is evidence to
individuals into divulging sensitive information or granting suggest that accounts originating in China are leveraging
unauthorized access to their organization’s networks and chatbots to debug code related to communications
systems. For example, the cybercriminal threat actor group surveillance technology, among other activities. But we
Scattered Spider has successfully used AI voice cloning to have not yet observed threat actors deploying an
conduct vishing attacks against numerous sectors, advanced capability for vulnerability scanning and
including healthcare. Using voice samples from corporate exploitation in real-world scenarios.
videos and social media, they generated convincing voice
clones of executives to authorize security changes and Cybercriminals have developed and sold multiple tools
network access requests. Criminals have also leveraged AI that can aid in vulnerability research, reconnaissance,
to bypass regulations and know-your-customer practices exploit writing, and task automation. Cybercriminals also
for cryptocurrency organizations. take advantage of AI-powered agents to mimic human-
like behaviors that bypass bot detection (e.g., random
mouse movements, real-time form completion) and fraud
Threat actors have also leveraged chatbots to generate
detection techniques (submitting micro-transactions to
content in non-native languages to conduct influence
validate card details).
operations. Examples include either translating or
optimizing content in a targeted language for social media
posts, short articles, and longform articles on topics such
as geopolitical conflict, criticism of United States and
European policy, or security-related content.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 9
As agentic systems increasingly integrate with disparate
Looking Ahead: New and Improved services and vendors, the opportunity for threat actor
AI Threat Vectors exploitation or vulnerability is ripe. Attackers could
potentially leverage agentic systems to conduct multi-
Agentic AI, “AI systems and models that can act stage attacks, find creative ways to access restricted
autonomously to achieve goals without the need for data systems, chain seemingly benign actions into
constant human guidance,” and has the capability to harmful sequences, or learn to evade detection by
conduct planning and reasoning, to memorize and recall network and system defenders.
information, and to take action and use tools to accomplish
tasks, all of which could reap productivity benefits and Continued social engineering at scale: From social
unlock new insights for organizations. engineering to propaganda proliferation, cybercriminal
and state-sponsored actors will continue to leverage AI
Additional Resources: OWASP Guide to technologies to improve the personalization and
Agentic AI Threats professionalization of their malicious activities. While not
realized yet, malicious use of multimodal AI, which
The international web security nonprofit OWASP released integrates text, images, voice, and sophisticated coding,
the first version of their guide to Agentic AI threats in could enable attackers to streamline and automate entire
February 2025. As agentic systems continue to evolve and attack chains. Theoretically, these attacks could conduct
become more sophisticated, so too does their risk profile. reconnaissance on targets, craft realistic phishing
This document from the OWASP Agentic Security Initiative content, find zero-day exploits, generate evasive
(ASI) provides a reference of emerging agentic threats malware, and automate lateral movements within
while simultaneously suggesting practical mitigation networks, leading to faster exploitation and increased risk
strategies. Cisco is a proud contributor to and supporter of across both the public and private sectors.
this guide.
Numerous areas of risk could emerge in the development
Agentic AI systems could also imperil organizations that of capabilities targeting AI models and systems
are neither prepared nor equipped to handle agentic themselves, including using adversarial inputs to trick AI-
systems and their potential for compromise. At least 14 powered security filters, hijacking AI agents used in
distinct threat vectors have been identified with agentic business operations workflows, as well as attacking
systems, including: memory poisoning, where false or elements of the AI supply chain (e.g., corrupting training
misleading data is introduced into an AI’s memory systems data, compromising a model’s cloud infrastructure).
to exploit the agent’s context; misaligned and deceptive Traditional cyber attacks against AI systems (as well as AI
behaviors, where an AI agent is used to conduct harmful or laboratories and developers) will remain a salient threat
disallowed actions; and unexpected remote code as attackers seek to conduct intellectual property theft,
execution and code attacks, where attackers inject user data theft, or disrupt, degrade, or destroy elements
malicious code or execute unauthorized scripts. of the AI development lifecycle.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 10
Developments in AI Policy
Overview
A significant number of new AI policy developments occurred The following sections are only intended to be a snapshot
in 2024, largely in response to the increasing prevalence of of trends seen in 2024 and do not account for all AI policy
AI-powered technologies and their market expansion. In the developments, both domestically and internationally.
United States alone, state lawmakers introduced over 700 Given the rapid evolution of the AI regulatory landscape,
AI-related bills—113 of which were enacted into law— changes to the below efforts may have occurred since the
across 45 states in 2024. The pace of policy activity has publication of this report. The information provided in this
not slowed in 2025. Within the first couple of weeks of report is meant to be a helpful resource only and is not
2025, 40 AI-related bill proposals have been introduced at intended to constitute legal advice.
both the state and federal levels. The swift and complex
nature of these changes has presented challenges to Domestic AI Policy Developments
players across the market navigating the evolving
landscape. in 2024
AI introduces social and economic risks alongside potential Fragmented State-by-State Legislation
substantial economic growth opportunities, challenging In the absence of federal policies on AI, states have taken
jurisdictions to balance the desire to foster innovation independent action to regulate the technology. A flurry of
against managing associated risks. As countries around new bills introduced at the state level put some
the world develop and implement AI legislation and restrictions on AI development and use.
regulations, no one standard approach to regulating AI has
emerged. In their efforts to respond to both the challenges · Colorado became the first state to pass a comprehensive
and opportunities brought by AI, governments have drawn AI Act (SB 24-205). The bill requires developers and
on a wide-ranging AI policy toolkit: drafting deployers of “high-risk” AI systems to comply with
comprehensive laws, regulations for specific use-case additional precautionary measures to ensure they avoid
applications, national AI strategies, and voluntary discrimination and other safety harms. The new law, part of
guidelines and standards. We have observed that AI Colorado’s Consumer Protection Act, mirrored the risk-
governance often begins with the rollout of a national based approach of the recently passed EU AI Act.
strategy before moving towards legislative action.
· Utah AI Policy Act bill (SB 149) came into effect on May 1,
Highlights of global developments in AI policy throughout 2024. This legislation is part of Utah’s consumer protection
2024 include: laws and introduced disclosure obligations for the use of
generative AI systems in both the private and public sectors.
· Country-level focus on promoting AI safety amidst In addition, it established the Office of AI Policy and the AI
rapid technological developments, through actions such as Learning Laboratory Program, with the potential to establish
AI Safety Summit voluntary commitments, as well as cybersecurity auditing procedures for higher risk AI
transatlantic and global partnerships; applications.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 11
Federal Interest in AI Safety and Security Importance of AI Security Standards
In 2024, there were various efforts across federal agencies This past year there was a lot of activity around the
to promote safe and secure AI development and use. development of AI security standards, providing
organizations guidance on how to secure AI applications.
· The Department of Justice leveraged existing statutes to
· The National Institute of Standards and Technology
seek harsher sentences for certain crimes involving the
(NIST), an agency of the U.S. Department of Commerce
misuse of AI.
that promotes domestic innovation by advancing
measurement science, standards, and technology,
· The bipartisan House Task Force on AI issued a
published its Adversarial Machine Learning (ML)
comprehensive report on AI including guiding principles and
Taxonomy. This resource, which is co-authored by
forward-looking recommendations to advance America’s
members of the Cisco AI Defense team, provides a
leadership in AI innovation responsibly.
conceptual hierarchy of attack lifecycles, attacker goals
and objectives, and attacker capabilities. In addition, it
· The Department of Commerce launched the
suggests corresponding methods for mitigating and
U.S. AI Safety Institute Consortium (AISIC). The National
managing the consequences of attacks.
Institute of Standards and Technology (NIST) launched the
consortium to: “[establish] guidelines and processes to · MITRE, a non-profit organization bridging public and
enable developers of generative AI to conduct AI red- private sectors through federally funded research centers,
teaming tests to enable deployment of safe, secure, and extended their Adversarial Threat Landscape for AI
trustworthy systems.” Systems (ATLAS) framework to cover generative AI
systems. The ATLAS matrix is a living community
· The U.S. Department of the Treasury released a report on knowledge base of adversarial tactics and techniques
managing AI-specific cybersecurity risks in the financial based on real-world attack observations. It’s a resource
services sector. In the report, “significant opportunities and used by security professionals, developers, and operators
challenges that AI presents to the security and resiliency of protecting AI-enabled systems.
the financial services sector.”
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 12
· In a landmark agreement, the UK and US AI Safety
International AI Policy Developments Institutes committed to a partnership to jointly test AI
in 2024 models and share frameworks, best AI safety practices, and
expertise.
Transnational Partnerships
· A second Safety Summit was hosted in Seoul, Korea in
In 2024, transnational partnerships were the primary policy May 2024, successfully securing commitments from sixteen
vehicle to promote safe and secure AI development and companies at the forefront of AI development to share risk
use globally. and safety frameworks and avoid high-risk models.
· The United Kingdom (UK) and Canada signed an · The UN unanimously adopted a US-led resolution on
agreement to work closely together on AI safety. As part of AI technologies. The draft resolution aims to lay out a
the agreement, the two countries agreed to share expertise comprehensive vision for “safe, secure, and trustworthy AI”
to enhance evaluation and testing work and “inspire and is based on the voluntary commitments put forth by
collaborative work on systemic safety research,” with an President Biden’s administration in partnership with leading
eye toward growing the network of AI safety institutes AI companies last fall. This marked a critical step towards
following the first AI Safety Summit in Bletchley in 2023. establishing international agreement on guardrails for the
ethical and sustainable development of AI. At its core, the
· EU and US AI experts from the EU-U.S. Trade and resolution encourages protecting personal data, monitoring
Technology Council developed an updated edition of the AI AI for risks, and safeguarding human rights.
Taxonomy and Terminology. This taxonomy helps to align
international governance efforts and creates a shared · Japanese Prime Minister Kishida Fumio announced the
understanding of how to effectively secure AI systems. The launch of the Hiroshima AI Process Friends Group at an
joint council also announced a new research alliance: AI for Organization for Economic Cooperation and Development
Public Good, focused on applying AI systems to the most gathering. The initiative, supported by 49 countries and
important global challenges. regions, aims to align global efforts on safe, secure, and
trustworthy generative AI. This initiative supported the
implementation of international guidelines as outlined in the
Hiroshima AI Process Comprehensive Policy Framework.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 13
National and Regional AI Governance • Singapore released its Model AI Governance Framework
for Generative AI, providing a voluntary framework for
In 2024, the EU AI Act became the world’s first organizations to adopt while deploying AI systems to meet
comprehensive AI law to come into force, while other best practices for AI risk management.
countries took national approaches to AI governance.
• Early in 2024, Japan signaled they were heading toward
• EU AI Act officially entered into force on August 1, 2024, the development of new legislation to regulate AI. Two
and outlines regulations on AI development, deployment, publications, from the Liberal Democratic Party and the
and use, imposing stricter rules on high-risk AI systems (as Japanese Cabinet Office’s AI Strategy team, recommended
stipulated on page 127 of the official EU AI Act text) and introducing regulations for large-scale foundation models.
banning "unacceptable" AI applications, with penalties for However, by the end of the year Japan’s attitude shifted
non-compliance up to 7% of an organization’s total towards a ‘light touch’ regulatory approach. As stipulated by
worldwide turnover. a second AI white paper, Japan aims to become the “most
AI-friendly country” by adopting principles from the
• The Australian Government released a new policy Hiroshima AI Process and “consider minimum necessary
for the responsible use of AI in government. The policy measures through legal regulations.”
positions the government to play a “leadership role in
embracing AI for the benefit of Australians while ensuring its Looking Ahead: Direction for AI
safe, ethical and responsible use, in line with community
expectations.” The policy is mandatory for non-corporate Policy 2025
Commonwealth entities and took effect on September 1,
This year’s AI policy developments have already signaled a
2024.
significant shift in the direction that emerging regulation is
headed, marking an evolution of the AI policy conversation
• There was a push in Africa to start regulating AI, as the use
toward effectively balancing the need for AI security with
of AI systems has been expanding across the continent. The
accelerating the speed of innovation and increasing
African Union, including 55 member nations, began
investment in AI infrastructure.
preparing an AI policy to further develop and regulate the
use of AI. However, there was ongoing debate about
whether regulation is warranted and the impact it might have In 2024, policymakers were primarily concerned with AI
on innovation. Seven African nations have already developed safety and mitigating any social and economic harm
national AI policies, and in February of last year the African associated with the use of AI. The AI safety conversation will
Union Development Agency published a policy draft to serve likely continue to be relevant for policymakers’ approach to
as the blueprint of further AI regulations by African nations. regulations in 2025 but addressing security-related risks and
supporting pro-innovation policy are clear priorities.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 14
• The Trump Administration takes action to support AI • The UK published its AI Opportunities Action Plan: The UK
innovation and protect national security: In the opening government detailed a slew of policy objectives, ranging from
days of his presidency, President Trump revoked President investments in infrastructure to fostering the development of
Biden’s AI Executive Order, and shortly thereafter announced UK Sovereign AI, further indicating a greater focus towards AI
a new one which the Administration positioned as fostering opportunity and growth. The three key categories of
innovation, supporting economic growth, and protecting recommendations include: laying the foundations to enable AI,
national security. This position was buttressed by Vice changing lives by embracing AI, and securing their future with
President JD Vance’s speech at the AI Action Summit, homegrown AI.
outlining the U.S. Administration’s priority of harnessing AI
innovation. The U.S. government is also increasingly • The Indian Ministry of Electronics and Information
concerned about the potential export of foundational Technology (MEITY) is seeking input on AI governance
technologies that may provide a technological advantage to guidelines: MEITY published a report on AI Governance
foreign adversaries in their development of AI. Guidelines Development, on January 6, 2025, seeking
comments from stakeholders. The governance guidelines
• The United States and UK decline to sign the 2025 AI adopt a risk-based approach and align closely with the OECD
Action Summit’s declaration on safety: The United States AI principles. In addition, the report recommends establishing
(along with the UK) declined to sign the Summit’s a technical advisory body to serve a similar role as an AI
declaration on safety, citing concerns over global safety institute.
governance and national security.
• South Korea signed an AI Framework Act into law: This
• French President Emmanuel Macron urges EU to simplify makes South Korea the second jurisdiction, following the EU,
regulatory efforts: While hosting the AI Action Summit in to enact a comprehensive regulatory AI law. It adopts a risk-
Paris, President Macron proposed a lighter approach to AI based approach, focusing on ‘high-impact’ AI systems. High-
regulation in Europe to boost member states’ impact AI, in this context, refers to AI systems that pose risks
competitiveness in the global AI race. to human life, physical safety, and fundamental rights.
• The UK rebrands its AI Safety Institute to focus on • Japan announces plans for AI Act: Japan introduced a
security: The UK AI Safety Institute officially rebranded as draft AI Act bill to its Parliament. The proposal does not take a
the UK AI Security Institute, signaling an increased focus on strict regulatory approach and does not include penalties for
combatting the use of AI to facilitate crime and threaten non-compliance. Instead, the bill focuses on operationalizing
national security. the Hiroshima Process Principles, supporting R&D and
empowering the government to investigate malicious uses of
• European Commission withdraws AI Liability Directive: AI that are not covered by existing legislation.
The European Commission formally abandoned the 2022 AI
Liability Directive, which aimed to “[lay] down uniform rules • AI security standards update to reflect new and
for certain aspects of non-contractual civil liability for damage emerging risks: The Open Worldwide Application Security
caused with the involvement of AI systems.” According to a Project (OWASP), a well-recognized global non-profit
press release on the Commission’s newly adopted 2025 organization that works to improve web-application and
work program, this decision was motivated by efforts to software security, released an updated version of the "Top
“reduce administrative burden and simplify EU rules.” Ten for Large Language Model Applications for 2025,”
including new additions like ‘misinformation’ and ‘vector and
• The European Union and France announce significant
investment plans for AI: During the AI Action Summit in embedding weaknesses,’ and announced a new
Paris, European Commission President Von der Leyen “Generative AI Red Teaming Guide.” These resources will be
announced the InvestAI plan which will seek to mobilize up to leveraged by organizations and governments to better
EUR200B in investments in AI infrastructures and four understand the AI security landscape and best practices.
gigafactories. France President Macron announced more
than EUR109B in private investments in AI in France.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 15
AI Security Research
Overview
Over the last year, Cisco’s AI security research team has • Automatic: Manual inputs and human supervision aren’t
led and contributed to several pieces of groundbreaking necessary.
research in key areas of AI security. These efforts reflect
our commitment to advance the AI security community • Black box: The attack doesn’t require knowledge of the
while simultaneously ensuring our customers are LLM architecture.
protected against novel threats and emerging
vulnerabilities. • Transferable: Prompts are written in natural language
and can be reused.
This section provides a high-level overview of our
methodologies, key findings, and real-world implications • Prompt efficient: Fewer prompts make attacks more
of Cisco’s various AI security research initiatives, including: discreet and harder to detect.
• Algorithmic jailbreaking attacks models with zero The success of TAP against sophisticated models like
human supervision, enabling adversaries to automatically GPT-4 and Llama 2 also demonstrates the relatively
bypass protections for even the most sophisticated LLMs. low cost of algorithmic jailbreaking and suggests that
This method can be used to exfiltrate sensitive data, disrupt more capable LLMs can oftentimes be easier to break.
services, and harm businesses in other ways.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 16
Applying Algorithmic Jailbreaking to Fine-Tuning Breaks Internal Model
Frontier Reasoning Models Guardrails
The emergence of advanced reasoning models like Fine-tuning foundational models is a common approach
OpenAI o1 and DeepSeek R1 prompted AI researchers businesses employ to improve the accuracy, domain
from Cisco and the University of Pennsylvania to develop expertise, and contextual relevance of an AI application in
Adversarial Reasoning. This automated approach to model a flexible and cost-effective way. However, research by
jailbreaking uses advanced model reasoning to effectively the Cisco AI team reveals a danger to fine-tuning that is
exploit the feedback signals provided by an LLM to bypass often overlooked—namely, that fine-tuning can throw off
its guardrails and execute harmful objectives. model alignment and introduce new safety and security
risks.
Adversarial Reasoning was instrumental for the Cisco
security evaluation of DeepSeek R1 which revealed a This phenomenon is broadly applicable and can even
concerning 100% attack success rate (ASR). In a broader occur with completely benign datasets, making fine-tuned
sense, this research suggests that future work on model AI applications generally easier to jailbreak and more likely
alignment must consider not only individual prompts but to produce harmful or sensitive results. Specifically, this
entire reasoning paths to develop robust defenses for AI research was conducted using Llama-2-7B and three
systems. AdaptLLM chat models fine-tuned and released by
Microsoft researchers to cover the domains of
biomedicine, finance, and law.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 17
Training Data Extraction via
Decomposition
Chatbots will typically refuse to answer prompts that
attempt to reconstruct copyrighted or paywalled data
because the underlying models are trained with specific
guidelines and restrictions on reproducing copyrighted or
paywalled materials verbatim. However, Cisco AI
researchers were able to leverage a simple method to trick
chatbots into regurgitating portions of news articles,
allowing for reconstruction of the source material and
raising concerns about greater information security risks
such as the extraction of sensitive, proprietary, or non-
public information.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 18
Recommendations for Implementing AI Security
AI applications have additional security considerations • Prioritize security in areas where adversaries
compared to traditional web applications, which can feel seek to exploit weaknesses. Equipped with a deeper
like entirely new territory and overwhelm enterprise understanding of the AI security threat landscape, prioritize
security teams. We would like to make this enormous new your defenses, institute controls, and harden your
threat landscape easier to grasp by highlighting the technological assets where you know adversaries and
commonalities between AI security and traditional criminals are targeting.
cybersecurity practices.
• Educate your workforce in responsible and safe AI
Each business will have to tailor its AI security strategy usage. As with any new technology, employee misuse or
around distinct implementation parameters. For example, misunderstanding of AI can be a tremendous source of
what models and datasets are you leveraging? What is the organizational risk. Clearly communicate internal policies
specific AI use case? How sensitive is the data being around acceptable AI use within legal, ethical, and security
handled? What end users does this AI application serve? boundaries to mitigate risks like sensitive data exposure.
While these are unique aspects, we outline some general
considerations and recommendations for all businesses AI security can still feel like an overwhelming challenge for
defining their AI security strategies below. most businesses: a dynamic threat landscape, evolving
standards, and new pieces of legislation—not to mention
• Manage risk at every point in the AI lifecycle. As
breakthroughs in AI technology itself—can be difficult to
outlined in our threat intelligence section, there is a degree
track and reflect organizationally. That’s why partnering
of risk at virtually every step of the AI lifecycle from
with the right vendors and investing in purpose-built AI
development to deployment. Ensure your security team is
security solutions is important. Cisco introduced AI
equipped to identify and mitigate these in every phase:
Defense precisely for this reason; with a straightforward
supply chain sourcing (e.g., third-party AI models, data
solution for managing AI risk from development to
sources, and software libraries), data acquisition, model
deployment, businesses can focus their efforts on
development, training, and deployment.
breakthrough AI applications knowing security is covered.
• Maintain familiar cybersecurity best practices. AI may
be new and unique, but familiar concepts like access
control, permission management, and data loss prevention
remain critical. Approach securing AI the same way you
would secure core technological infrastructure and adapt
existing security policies to address AI-specific threats.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 19
AI Security at Cisco
Cisco is building on decades of leadership in networking • Protecting Cisco Secure Endpoint and Email Threat
and cybersecurity to pave the way for rapid AI innovation Protection customers from malicious AI supply chain
and resilient AI security. In 2024 alone, we made artifacts downloaded from Hugging Face, shared via
tremendous progress integrating new capabilities into our email, or downloaded from a shared drive for customers
existing portfolio and launched the first truly using Cisco Secure Endpoint and Cisco Secure Email
comprehensive solution for enterprise AI security: Cisco Threat Defense.
AI Defense.
This State of AI Security report validates that the AI
At a high level, Cisco AI Defense addresses the two
landscape has and continues to evolve rapidly. As we
primary areas of enterprise AI risk. The first is risk of
sensitive data exposure from employees using third- drive towards future breakthroughs in AI technology and
party systems and sharing intellectual property, PII, and applications, Cisco remains committed to AI security
other confidential information with these tools. The through our contributions to the community and cutting-
second is risk for businesses developing and deploying edge solutions for customers pushing the envelope of AI
their own AI applications. Vulnerabilities exist all innovation.
throughout the AI development lifecycle; businesses
creating AI applications need to ensure that these
systems are safe and secure for customers.
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State
State of AI Security Report
Report 20
Contributors
© 2025 Cisco and/or its affiliates. All rights reserved. Cisco State of AI Security Report 21