How To Detect When A Financial AI Agent Is Behaving Anomalously

The Problem In Production: A Trading Agent That Drifts Overnight

Picture an autonomous trade-surveillance and anti-money laundering (AML) triage agent at a large bank. It reviews flagged transactions overnight and drafts escalation narratives for analysts. One night an upstream data source changes its schema.

By morning, the agent is over-escalating routine transfers. It also cites accounts it never actually queried.

Nothing crashed. Latency looked normal, and the transaction data was clean. The agent itself drifted.

An Anomalous Agent Is Not A Bad Transaction

This is an agent-behavioral anomaly, and it is distinct from a transaction anomaly. A transaction anomaly is a suspicious payment the agent is built to catch. An agent-behavioral anomaly is a deviation in how the agent reasons and acts.

The first is the agent's job. The second is a failure of the agent doing its job.

Four Traits That Make Agent Anomalies Hard To Catch

Agent-behavioral anomalies are hard to catch for four reasons.

• Probabilistic: the same input can yield different outputs.
• Autonomous: no human reviews each decision.
• High Velocity: they act across many tool calls.
• Subtle Failures: they surface as hallucinated narratives or quiet over-escalation rather than crashes.

The stakes are rising. Deloitte projects that US AI-enabled fraud losses reached $12.3 billion in 2023. It expects them to hit $40 billion by 2027, a 32% compound annual growth rate [1].

Why Rules And Infrastructure Monitoring Miss Agent Anomalies

Three Monitoring Tools That Watch The Wrong Signals

Most financial-services teams reach for three familiar tools. Each was built for a different problem.

• Static Rules: encode known-bad patterns as if-then logic. They cannot anticipate novel or low-and-slow agent behavior that no analyst has scripted. Research on AML policy effectiveness finds that less than 0.1% of criminal proceeds worldwide are ever recovered [2], and AML industry reporting widely cites false-positive rates above 90% for traditional transaction-monitoring systems.
• Transaction Monitoring: watches inputs and outputs. It scores the payment, not the reasoning path the agent took to judge it.
• Infrastructure Monitoring: sees latency, error rates, and throughput. It says nothing about decision quality.

Correlation Across Signals Exposes Agent Anomalies

The deeper issue is correlation. Conventional monitoring sees an agent's high-level intent or its low-level system actions. It cannot correlate the two.

Research on the AgentSight system calls this a semantic gap [3]. That blind spot makes benign, malicious, and failing behavior look identical. Correlating intent with actions is what exposes prompt injection, reasoning loops, and coordination bottlenecks [3].

The result is the same across all three tools. Teams learn that something happened, not why, and usually after the fact. This is why observability must precede autonomy.

The Signals That Reveal An Anomalous Agent

Span-Level Telemetry Captures The Agent's Reasoning Path

Effective financial AI agent anomaly detection starts with instrumenting what the agent actually does. This practice is Agentic Observability: capturing the full execution context of an agent so its behavior can be evaluated, not its outputs alone. It rests on Standardized Telemetry, a consistent record of every span in a trace.

A span is a single unit of work, such as one tool call, one retrieval, or one model completion. Span-level telemetry rolls up across the agentic hierarchy, the full decision tree of agent calls, tool invocations, and sub-agent outputs. That hierarchy is where behavior becomes visible.

Three Steps To Detect An Anomalous Agent

Detection then follows three steps.

1. Establish a behavioral baseline. Capture span-level telemetry across normal operation, including tool calls, retrieval patterns, and decision lineage. The baseline describes how the agent behaves when it is healthy.
2. Evaluate every step. Score each span with large language model (LLM) and agentic metrics such as faithfulness, groundedness, toxicity, and jailbreak or PII (personally identifiable information) detection. The RAGAS framework established faithfulness, answer relevance, and context relevance as core evaluation dimensions [4].
3. Score deviation and alert. Compare live spans against the baseline, trigger alerts on divergence, and roll signals up to aggregate insights across the agent timeline.

Continuous evaluation is not optional. Without it, teams are effectively flying blind and cannot detect regressions before they compound [5].

The evaluation step is where naive detection breaks down. A threshold on a single output field misses the reasoning behind it. Contrast the two approaches.

# Naive approach: a static threshold on one output field.
# It sees the final number, not how the agent reached it.
if escalation_score > 0.9:
    alert("possible anomaly")

# Behavioral approach: evaluate the span against a baseline and eval metric.
# Each span carries the agent's reasoning, tool calls, and retrieved context.
for span in trace.spans:
    faithfulness = centor.evaluate(span, metric="faithfulness")
    deviation = baseline.distance(span.behavior_vector)
    if faithfulness < baseline.faithfulness_floor or deviation > baseline.max_deviation:
        alert(span, reason="behavioral drift", score=faithfulness)

The second approach evaluates behavior, not a single number. Running those evaluations in under 100ms per span is the hard part.

In-Environment Evaluation Keeps Agent Data Where It Lives

Fiddler Centor Models (formerly Fiddler Trust Models) are batteries-included and run in-environment. They evaluate with no external API calls and no data exposure. They carry no per-evaluation cost and enforce policies at under 80ms response time, and they stay framework, model, and cloud agnostic.

In our deployments, Centor Models score every span with 100% trace coverage and no sampling.

What To Watch For When Tuning AI Agent Anomaly Detection

We have seen four failure modes derail agent anomaly detection in practice. Watch for each when you tune your setup.

• Thin Baselines: A baseline built on too little traffic or an atypical window like quarter-end flags normal behavior as anomalous.
• Sampling Blind Spots: Down-sampling traces to cut external evaluation cost hides low-and-slow drift.
• The Confident Fabricator: An agent that invents a plausible rationale passes naive checks; hallucination is fluent output lacking grounding in accurate information.
• Aggregate-Only Baselining: Baselining on aggregate metrics alone hides per-span and per-tool deviation, where the real signal lives.

Wells Fargo researchers identify hallucination and toxicity as the primary novel model risks that generative AI introduces in financial institutions, and they note that US banks are obligated to enhance their model risk frameworks in response [6].

From Detection To Enforcement: Acting Before The Agent Does

Detection on its own produces an incident report. It tells you an agent misbehaved after the fact. Enforcement acts on the request and response path, before an action executes.

That distinction separates observing risk from controlling it.

Two Guardrail Types Catch Threats At Different Stages

• Pre-LLM Guardrails: intercept inputs before they reach the model, catching a prompt injection attempt or a malformed instruction before the agent ever reasons over it.
• Post-Execution Guardrails: inspect outputs and proposed actions before they are returned or acted upon, catching a fabricated narrative or an unauthorized trade before it leaves the system.

Every Guardrail Ends In One Of Three Actions

Both apply one of three actions.

• Allow: lets a clean request proceed.
• Redact: removes sensitive content such as PII or protected health information (PHI) while preserving the rest of the request.
• Block: rejects the full request, the right response to an injection attempt or an unauthorized trade.

Redaction keeps the workflow moving; blocking stops it.

This matters because data exposure is broad. It includes any external service, API call, log store, or trace export that receives agent inputs or outputs. It is not limited to text sent to an LLM provider.

Enforceable Policy applies these actions inline on the Fiddler AI Observability and Security platform.

Real-time enforcement is grounded in current guidance. Anthropic recommends that agents pause for human feedback at checkpoints, alongside sandbox testing and guardrails that constrain autonomy [7]. Its research on deployed agents adds that training models to surface their own uncertainty complements external safeguards like permission systems and human oversight [8].

Governance Must Scale in the Era of Autonomous Agents

The governance burden is real. Agentic AI amplifies safety, liability, and autonomy risks, and one-time validation with periodic review is inadequate, so real-time adaptive oversight is required [9]. Yet only one in five organizations has a mature governance model for autonomous AI agents [10].

One problem remains open. Real-time behavioral anomaly detection across multi-agent handoffs, where one agent's output becomes another's input, is still unsolved.

Oversight Has To Scale With Autonomy

Return to the bank's overnight triage agent. With span-level telemetry, that team can baseline how the agent reasons when it is healthy. They can evaluate every span for faithfulness and grounding as work happens.

They can also enforce inline, redacting sensitive data and blocking unauthorized actions before they execute. The schema change that once drifted the agent overnight becomes a caught deviation by the next span.

The engineering implication outlasts any single fix. Oversight has to scale with autonomy. Every increment of agent independence you grant demands an equal increment in your ability to see, evaluate, and act. That's visibility, context, and control, applied continuously, not once at launch.

Reach out to a Fiddler expert if you are looking to monitor, enforce, and govern agents.

References

[1] Deloitte Center for Financial Services, "Generative AI is expected to magnify the risk of deepfakes and other fraud in banking," Deloitte Insights, May 29, 2024. [Online]. Available: https://www.deloitte.com/us/en/insights/industry/financial-services/deepfake-banking-fraud-risk-on-the-rise.html

[2] R. Pol, "Anti-money laundering: The world's least effective policy experiment? Together, we can fix it," Policy Design and Practice, vol. 3, no. 1, 2020. [Online]. Available: https://www.tandfonline.com/doi/full/10.1080/25741292.2020.1725366

[3] Y. Zheng et al., "AgentSight: System-Level Observability for AI Agents Using eBPF," arXiv, Aug. 2025. [Online]. Available: https://arxiv.org/html/2508.02736v2

[4] S. Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation," arXiv, Sep. 2023. [Online]. Available: https://arxiv.org/abs/2309.15217

[5] E. Yan, "Patterns for Building LLM-based Systems & Products," eugeneyan.com, 2023. [Online]. Available: https://eugeneyan.com/writing/llm-patterns/

[6] A. Bhattacharyya et al., "Model Risk Management for Generative AI in Financial Institutions," arXiv, Mar. 2025. [Online]. Available: https://arxiv.org/abs/2503.15668

[7] Anthropic, "Building Effective Agents," Anthropic, Dec. 2024. [Online]. Available: https://www.anthropic.com/research/building-effective-agents

[8] Anthropic, "Measuring AI Agent Autonomy in Practice," Anthropic, Feb. 18, 2026. [Online]. Available: https://www.anthropic.com/research/measuring-agent-autonomy

[9] "The Agentic Regulator: Risks for AI in Finance and a Proposed Agent-based Framework for Governance," arXiv, Dec. 2025. [Online]. Available: https://arxiv.org/html/2512.11933v1

[10] Deloitte AI Institute, "State of AI in the Enterprise 2026," Deloitte, 2026. [Online]. Available: https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html