What Is AI Agent Observability: Traces, Spans, and Signals for Production Agents

In an illustrative scenario, an order-support agent tells a customer that no refund is due on an order that qualifies for one. The answer took six tool calls and two sub-agent handoffs to produce. The orchestrator delegated to a billing sub-agent, which looked up the order and pulled the price list. It then retrieved the refund policy and handed a summary to a second sub-agent that drafted the reply.

Every one of those calls returned a 200. The 99th-percentile (p99) latency stayed inside budget, and the error rate on the application performance monitoring (APM) dashboards read zero. Whoever was on call had nothing to page on.

The failure happened between the steps, where the billing sub-agent passed a stale order status across the handoff. The platform team had no record of that boundary in its APM stack.

An agent can also burn tokens in a retry loop, as InfoQ’s report on Cloudflare’s agent tracing work describes [1].

Platform teams use AI agent observability, also called Agentic Observability, to collect and correlate telemetry from every step an autonomous agent takes to complete a task, then analyze it. That telemetry covers its large language model (LLM) calls, the tools it invokes with their arguments and results, the handoffs between agents, the tokens each step consumes, and the evaluation and guardrail scores attached to each of those steps. Teams organize that telemetry across application, session, agent, trace, and span to assess whether an agent did the right thing and at what cost. When it did not, the team can identify the exact span responsible.

What Is AI Agent Observability

Observability for AI agents has to cover behavior decided at run time, one step at a time.

• Autonomy: The agent chooses its own tools and sub-agents, so the choice itself has to be recorded, together with the arguments it passed, or a wrong tool call is indistinguishable from a right one.
• Multi-step execution: Errors compound along the chain. Cemri et al. annotated 1,642 multi-agent traces across seven frameworks and found failure rates between 41% and 86.7%, with step repetition (15.7%) and reasoning-action mismatch (13.2%) among the most common modes [2].
• Handoffs: Gartner’s Varma and Guttridge note that an agent’s actions and triggers can be recorded but why it acted cannot be verified, and that this widens with multi-agent handoffs [3].
• Token cost: Cost belongs in the same record because a looping agent spends money while reporting success.

How It Differs From APM and LLM Monitoring

Engineers use APM to check whether a service is up and how fast it responds. They also review reported errors, but cannot use APM to explain why an agent spent far more than usual asking the same question three times, as Sabith K Soopy’s CNCF post puts it [4]. With LLM monitoring, teams score one prompt and one completion for hallucination or toxicity. They also measure relevance and review token usage per call. With agent observability, teams keep both forms of monitoring and add the structure between calls.

Engineers who check only status codes and latency miss wrong-but-successful answers because the HTTP layer did exactly what it was built to do. Mulik puts it directly in his InfoQ platform engineering playbook: “A hallucinated response isn’t a stack trace. It passes every traditional health check.” He adds that latency dashboards can show a healthy p99 and a near-zero error rate while the model drifts toward lower-quality outputs that still look syntactically valid [5].

With LLM monitoring, engineers catch that drift at the completion level but inspect each completion independently. From the completion score alone, they cannot determine that it was the eighth step of a trace or that its context arrived through a handoff. They also cannot track whether the same tool was called three times with identical arguments.

The Agentic Hierarchy: Application, Session, Agent, Trace, Span

OpenTelemetry (OTel) standardizes the trace and the span, defines attributes for the session [6] and the agent [7], and leaves the application level to aggregation. The five-level ladder that stacks those into one drill-down path is a platform convention, and each level answers a different operational question:

• Application: Is the system healthy overall? Aggregate token usage, latency, evaluation scores, and guardrail decisions across every session. This is where alerts fire.
• Session: Is the problem isolated to one user or one interaction? The conventions use gen_ai.conversation.id to correlate the turns of a conversation and tell instrumentations not to populate it from a UUID or a trace ID when no real identifier exists; a content hash is also excluded [6]. A session without a real ID cannot be grouped.
• Agent: Is one agent in a multi-agent workflow causing the issue? invoke_agent spans carry gen_ai.agent.name and gen_ai.agent.id; the conventions want a stable ID and advise against transient in-memory ones [7]. Without a stable ID, the billing sub-agent in the opening scenario cannot be compared across runs.
• Trace: What path did one request take? A root invoke_agent span with child chat and execute_tool spans records the sequence, six tool calls and two handoffs in order, with timing.
• Span: Is a specific step consistently underperforming or failing? Each chat, execute_tool, retrieval, or memory operation is its own span with its own attributes, duration, and error state.

The Signals That Matter for Agents

Every signal worth capturing for an agent attaches to a span:

• Tool-call success and arguments: The execute_tool span requires gen_ai.tool.name and carries error.type on failure. It also offers gen_ai.tool.call.arguments and gen_ai.tool.call.result as opt-in attributes [6]. Enable them for the tools that matter, because a tool called with the wrong order ID returns a clean result and a clean status.
• Latency per step: Each chat and execute_tool span has its own duration, so teams can query per-step latency from spans without defining a new metric. A policy retriever that slows down on one agent’s calls appears here before it moves end-to-end p99.
• Token cost: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens are recommended on the invoke_agent span [7]. Aggregate them per session and per agent; Soopy’s post recommends alerting on session cost against a rolling average and on per-agent daily burn rate [4].
• Evaluation scores on spans: The gen_ai.evaluation.result event should be parented to the evaluated span. Its attributes include required gen_ai.evaluation.name and a conditionally required score value or label; an explanation is recommended [8].
• Guardrail trigger rate: Record every allow or block decision, and every decision to redact, as an event on the span it protected, then count decisions per application, session, agent, and tool. Engineers can investigate the tool when its responses have a rising block rate, or investigate the agent when its prompts have a rising rate.
• Handoffs between agents: The published conventions carry agent identity on invoke_agent spans [8], so have the orchestrator or agent SDK record both agent names on the delegating span.

Mapped to metrics, events, logs, and traces (MELT):

• Metrics: the aggregates above, per application and agent, with low-cardinality labels.
• Events: evaluation results and guardrail decisions on their spans.
• Logs: tool responses and errors carrying trace and span IDs.
• Traces: one per agent run, every span carrying the conversation ID.

Instrumenting With OpenTelemetry at the Gateway or Collector

The GenAI semantic conventions are marked Development, so attribute names still move [6]. Normalize them at the two points you already control: the LLM or MCP gateway (Model Context Protocol) and the OTel Collector. The gateway sees model requests and MCP tool calls routed through it and can emit chat and MCP spans with provider and model attributes, along with token attributes, without touching agent code. You can configure the Collector to rename legacy attributes and redact message content before storage. You can also tag spans by team for chargeback. In-process planning steps and memory reads remain outside the gateway’s view, as do in-process sub-agent calls. Frameworks that emit the conventions natively add those spans to the same trace through propagated context. See the guide to OpenTelemetry tracing for production agents.

From a Degraded Metric to Root Cause

An on-call engineer starts the drill-down at an application-level alert and ends at one span. Using the opening scenario:

1. Application: An alert fires because the answer-correctness evaluation score for the order-support application dropped below its threshold while latency and error rate held steady. Token totals also stayed steady. The engineer therefore investigates agent behavior.
2. Session: The engineer filters sessions by the failing score and finds the drop concentrated in conversations that include a refund question. Sessions without one are unaffected, so the engineer rules out a provider or model change.
3. Agent: Within those sessions, the engineer finds low scores on the billing sub-agent’s spans; the orchestrator and the reply sub-agent score normally.
4. Trace: The engineer reviews a representative trace and follows the six tool calls and two handoffs in order. The order lookup ran first, and the summary handed to the reply sub-agent carries a status recorded before the refund was approved.
5. Span: The engineer inspects the execute_tool span for the order lookup, reviewing its arguments and a result containing the outdated status, then reads the mismatch in its evaluation event. The engineer identifies that tool’s data path as the place to fix, using one span as evidence.

A four-stage walkthrough works the same path step by step. The wider operating loop of alerting and intervention is covered under monitoring and controlling agentic applications.

How Fiddler Approaches Agentic Observability

The Fiddler AI Observability and Security platform monitors continuously across application, session, agent, trace, and span, so a degraded application metric drills down to the exact LLM call or tool call behind it. OpenTelemetry-native instrumentation means a gateway deployment needs no agent-side instrumentation; teams that want in-process spans use SDKs for LangGraph, LangChain, AWS Strands Agents, and Google ADK, or REST endpoints. Batteries-included Fiddler Centor Models can score prompts and responses in-environment by default, and the scores attach to traces and appear in the hierarchical drill-down.

Fiddler Guardrails applies policy decisions to requests before the model is invoked and to responses before they reach the user or a downstream system. Depending on the policy, it allows or blocks content; it can also redact content. It records who triggered each decision and which policy applied, which gives teams the controls and evidence to govern agents in production.

Request a demo to review how agent observability across all five levels would fit the gateway and collector pipeline you already run.

References

[1] S.-J. Wiggers, “Cloudflare Adds Agent Tracing, with Truncation Limits and Uneven Payload Defaults,” InfoQ, Aug. 15, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://www.infoq.com/news/2026/08/cloudflare-agent-tracing/

[2] M. Cemri et al., “Why Do Multi-Agent LLM Systems Fail?” in NeurIPS 2025 Datasets and Benchmarks Track, 2025, arXiv:2503.13657. Accessed: Oct. 14, 2026. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2025/hash/b1041e52d3be19f0a9bc491657488e4a-Abstract-Datasets_and_Benchmarks_Track.html

[3] S. Varma and K. Guttridge, “Observability Is a Must,” Gartner, Aug. 17, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://www.gartner.com/en/documents/8271421

[4] S. K. Soopy, “You can’t debug what you can’t see: observability for AI agents,” CNCF blog, Aug. 4, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://www.cncf.io/blog/2026/08/04/you-cant-debug-what-you-cant-see-observability-for-ai-agents/

[5] A. Mulik, “The Platform Engineering Playbook for Production LLMs,” InfoQ, Oct. 5, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://www.infoq.com/articles/platform-engineering-playbook-production-llms/

[6] OpenTelemetry, GenAI spans semantic conventions (gen-ai-spans.md), semantic-conventions-genai repository, main branch, Development status. Accessed: Oct. 14, 2026. [Online]. Available: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

[7] OpenTelemetry, GenAI agent spans semantic conventions (gen-ai-agent-spans.md), semantic-conventions-genai repository, main branch, Development status. Accessed: Oct. 14, 2026. [Online]. Available: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md

[8] OpenTelemetry, GenAI events semantic conventions (gen-ai-events.md), semantic-conventions-genai repository, main branch, Development status. Accessed: Oct. 14, 2026. [Online]. Available: GitHub

[9] G. Liu and S. Solomon, “AI Agent Observability - Evolving Standards and Best Practices,” OpenTelemetry blog, Mar. 6, 2025. Accessed: Oct. 14, 2026. [Online]. Available: https://opentelemetry.io/blog/2025/ai-agent-observability/

[10] OpenTelemetry, MCP semantic conventions (mcp.md), semantic-conventions-genai repository, main branch, Development status. Accessed: Oct. 14, 2026. [Online]. Available: GitHub