AI Agent Security Risks: Why Autonomous Systems Demand a New Threat Model

Why AI Agents Have a Larger Attack Surface Than Traditional LLM Apps

Traditional LLM applications follow a constrained pattern: prompt in, text out. The model generates a response, and the application returns it. AI agents break this model entirely. Agents take autonomous actions, invoke external tools, persist memory across sessions, and chain decisions without human intervention at every step.

This shift from text generation to autonomous action introduces six distinct attack surface expansions:

1. Tool access: A successful prompt injection in a standard LLM produces misleading text; in an agent, the same injection triggers real-world actions such as sending emails, modifying records, or exfiltrating data through connected APIs. The difference between LLM security risks and agent security risks starts here.
2. Persistent memory: Agents that maintain long-term memory are vulnerable to memory poisoning, where an attacker injects a payload that persists across sessions and activates on a future trigger condition.
3. Autonomous decision loops: Without human checkpoints at each step, agents can compound a single bad decision across multiple iterations before anyone detects the deviation. Even coding agent security patterns demonstrate this compounding effect.
4. Multi-agent delegation: When Agent A passes output to Agent B, a compromised upstream agent propagates tainted outputs downstream, and each receiving agent treats that output as trusted input.
5. Expanded identity and permissions: Agents often operate with service-level credentials; a hijacked agent acts with the full authority of its assigned identity, not the limited scope of a single user prompt.
6. Opaque reasoning chains: Multi-step agent reasoning makes it harder to trace which specific step introduced a vulnerability, especially when intermediate outputs are not logged.

Traditional LLM security controls (input filtering, output moderation) remain necessary but are not sufficient. The attack surface for agents is defined by what they can do, not just what they can say.

Why the OWASP Agentic AI Top 10 Changes How You Threat-Model Agents

The OWASP Agentic AI Top 10 [1], published in 2025, extends threat modeling beyond the original OWASP Top 10 for LLM Applications [2] with the first structured taxonomy of vulnerabilities specific to autonomous AI systems.

Three categories are especially relevant to production agent deployments:

1. Prompt injection escalation: Agents amplify injection attacks because they can act on injected instructions through tool calls and API invocations.
2. Excessive agency and privilege: Agents granted broad permissions without least-privilege constraints create disproportionate blast radius when compromised.
3. Insufficient logging and monitoring: Agents that operate autonomously without continuous telemetry leave security teams blind to threats that emerge across multi-step execution chains.

The original OWASP LLM Top 10 was built around the request-response cycle: a user sends a prompt, the model returns a completion. Agents break that model. A single user request can move through dozens of internal steps, tool calls, memory reads, and sub-agent delegations, each of which is a potential vulnerability surface the request-response model was never designed to cover. An analysis on production agent architectures confirms this expanded surface area in practice.

Here is a pattern we see in production. An agent with broad permissions processes documents from an external source. An attacker embeds an instruction in one of those documents. The agent interprets it, executes a tool call to access internal records, and exports data to an attacker-controlled endpoint. Without continuous monitoring across every step of that chain, the exfiltration looks like normal agent activity.

Enforceable policy applied at the tool-call boundary would restrict the agent to its authorized actions, blocking the unauthorized export before it executes.

How Prompt Injection Escalates When Agents Have Tool Access

In a standard LLM application, a successful prompt injection produces misleading or harmful text, and the damage stays contained to the output. When agents have tool access, the same injection class produces a fundamentally different outcome: unauthorized actions executed with the agent's credentials.

The attack pattern follows three steps:

1. Injection delivery: An attacker embeds a malicious instruction in a document, email, or data source the agent will process. Research on indirect prompt injection [3] demonstrates the range of delivery vectors available to attackers.
2. Instruction interpretation: The agent's LLM interprets the embedded instruction as a legitimate task within its execution context.
3. Tool execution: The agent invokes a connected tool (API call, database query, file operation) according to the attacker's instruction, using its own service-level permissions.

Input sanitization alone cannot prevent this pattern. The injected content may be semantically valid and syntactically clean. It passes input filters because it resembles legitimate task instructions. The vulnerability is not in the input; it is in the agent's authority to act on that input.

The foundational mitigation is least-privilege tooling. Each agent should have access only to the specific tools and permission scopes its task actually requires. Broad tool access turns every prompt injection into a potential privilege escalation. Systematic red teaming techniques can surface these escalation paths before attackers do.

Memory Poisoning as a Persistent Threat

Memory poisoning occurs when an attacker injects a payload into an agent's long-term memory store. Unlike one-shot prompt injection, the poisoned memory persists across sessions and activates when a future interaction matches the trigger condition the attacker embedded. Recent memory poisoning research [4] shows these attacks persist across sessions and are difficult to detect, particularly in long-term semantic memory stores.

This makes memory poisoning significantly harder to detect than session-scoped injection because the attack and its effect are separated in time. Session resets do not clear the threat. The payload lives in the persistent memory layer, not the conversation context.

RAG-based agents are especially vulnerable. Poisoned documents enter the retrieval corpus and surface repeatedly across queries. Every retrieval that includes the poisoned document reintroduces the attacker's payload into the agent's context window. Understanding the full range of data leakage paths is essential for securing these retrieval pipelines.

Mitigation requires continuous monitoring of memory writes and retrieval outputs. Anomaly detection applied to memory operations can flag when new entries introduce instruction-like patterns that diverge from the baseline distribution of the agent's memory store.

Multi-Agent Cascading Failures and the Trust Tax

In multi-agent architectures, Agent A's output becomes Agent B's input. If Agent A is compromised or produces subtly incorrect output, every downstream agent inherits that error. This is the cascading failure pattern, and it is fundamentally different from microservice failures in traditional software. Research documenting simulated agent attacks [5] shows how quickly these cascading compromises propagate in practice.

When a microservice fails, it returns an error code or times out. That failure announces itself. When an agent produces compromised output, the result looks semantically valid. Agent B receives plausible-looking text and processes it as trusted input. The failure propagates silently because each agent's output appears reasonable in isolation. Research on threats in LLM-agent workflows [6] documents how compromised outputs propagate across multi-agent architectures.

Verifying every agent's output at every handoff is the only way to catch compromised data before it propagates. The AI Trust Tax is the per-query cost enterprises incur when calling external APIs to evaluate each agent's output. In multi-agent systems, this cost multiplies with every handoff. Enterprises can incur approximately $260K annually at 500K traces per day; at 1M traces per day, that figure doubles. These figures vary by model, deployment size, and traffic volume.

Because of the AI Trust Tax, teams routinely skip evaluation at inter-agent boundaries. The cost of verifying every handoff is prohibitive when each evaluation requires an external LLM call. This leaves blind spots at the exact points where cascading failures originate.

The alternative is inline evaluation that runs within the agent pipeline without external API overhead. Fiddler Trust Models eliminate the AI Trust Tax by running evaluation in-environment with no external API calls and no per-evaluation cost. This means teams can apply Reliable Evaluation at every agent handoff without the cost scaling that forces blind spots in multi-agent workflows.

Why Agent Security Requires Both Guardrails and Sequence-Level Visibility

Static guardrails are necessary for agent security, but they are not sufficient on their own. Pre-LLM guardrails intercept known-bad inputs before they reach the model. Post-execution guardrails inspect outputs before they are returned or acted upon. Both operate on individual requests. Agent threats often emerge from sequences of individually benign actions that combine into a malicious pattern. No single step triggers a guardrail, but the sequence constitutes an attack.

Observability functions as a security control by providing distributed tracing across every agent step, tool call, and memory access. Span-level telemetry rolls up to aggregate insights across the agentic hierarchy, giving security teams visibility into the full decision tree of agent calls, tool invocations, and sub-agent outputs. Established agent observability patterns provide the foundation for this instrumentation.

Three observable signals indicate a compromised agent:

1. Anomalous tool-call patterns: Unexpected API calls, permission escalation attempts, or tool invocations outside the agent's defined scope.
2. Memory drift: Retrieval outputs diverging from baseline distributions, indicating potential memory poisoning or corpus contamination.
3. Output divergence across agent handoffs: Inconsistencies between what an upstream agent produced and what a downstream agent received or interpreted.

These signals require continuous monitoring, not periodic audits. By the time a quarterly review surfaces an anomaly, the cascading failure has already propagated. Tracking the right guardrails metrics ensures these signals are captured in real time.

What to Watch For in Agent Telemetry

• Tool-call frequency spikes above established baselines
• Memory-write anomalies that introduce new instruction patterns
• Permission escalation attempts outside defined agent scope
• Output consistency drift between upstream and downstream agents

The Fiddler AI Observability and Security Platform delivers this capability across the full agentic hierarchy through standardized OpenTelemetry-based instrumentation. Agentic Observability as a discipline treats every agent action as an observable event, enabling teams to detect threats that static guardrails miss and respond before compromised outputs reach production systems or downstream agents.

Building an Agent Security Practice for Regulated Industries

Financial services, healthcare, and defense operate under regulatory frameworks that require audit trails, explainability, and demonstrable policy enforcement for automated decision-making. AI agents operating in these environments must meet the same standards applied to any system that makes or influences consequential decisions. The AI risk management framework [7] from NIST provides one reference architecture for structuring these controls.

Agent security in regulated contexts goes beyond access controls. It requires auditable governance: immutable logs of every agent decision, tool invocation, and data access, paired with enterprise-wide visibility across all live, testing, and retired AI applications. Every agent action needs a complete decision lineage that can be produced for regulators on demand. A comprehensive approach to enterprise AI security treats these audit requirements as foundational, not optional.

Policy enforcement in these environments must be automatic and continuous. Manual, periodic reviews cannot keep pace with agents that execute thousands of decisions per hour. Financial institutions and government agencies, including the U.S. Navy and organizations like DTCC, are already operationalizing these controls as part of their AI governance programs.

From Threat Model to Action Plan

AI agents require a fundamentally different security model than traditional LLM applications. The attack surface is defined by what agents can do: the tools they invoke, the memory they persist, the decisions they chain, and the downstream agents they delegate to. Threat models built around text generation alone leave the most consequential vulnerabilities unaddressed.

As agents gain more autonomy in multi-agent networks, the organizations that treat observability as a first-class security control will scale deployments more safely than those still treating it as a monitoring afterthought. Agentic Observability is still a maturing discipline, but the teams investing in continuous, inline evaluation and full-stack agent tracing today are building the foundation that production-grade autonomous systems will require.

See how Fiddler Agentic Observability helps teams detect and respond to agent security threats in real time.

References

[1] OWASP, "OWASP Top 10 for Agentic Applications," OWASP Foundation, 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[2] OWASP, "OWASP Top 10 for LLM Applications," OWASP Foundation, 2025. https://genai.owasp.org/llm-top-10/

[3] Greshake, K., et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," arXiv, 2023. https://arxiv.org/abs/2302.12173

[4] Zou, W., et al., "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models," arXiv, 2024. https://arxiv.org/abs/2603.20357

[5] Unit 42 / Palo Alto Networks, "AI Agents Are Here. So Are the Threats," Unit 42 Threat Research, 2025. https://unit42.paloaltonetworks.com/agentic-ai-threats/

[6] Tao, M., et al., "Threats in the LLM-Agent Workflows," arXiv, 2025. https://arxiv.org/html/2506.23260v1

[7] NIST, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," National Institute of Standards and Technology, 2023. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf