Key Takeaways
- LLM tracing records a request's full execution path through your application, capturing every prompt, model call, tool invocation, and retrieval as spans.
- You can start with OpenTelemetry auto-instrumentation in an afternoon, then add manual spans where your logic needs custom context.
- Because traces capture raw prompts and outputs, trace data governance belongs in your design from the first production request.
Why LLM Failures Are Hard To Reproduce In Production
Consider a financial-services support assistant that returns a customer's account balance. Most days it works. Intermittently it reports the wrong balance, and nothing in the logs explains why.
The trouble starts with non-determinism. The same prompt can produce different outputs across runs, so a failure may not reproduce when you replay the request [1]. A bug you cannot reproduce is a bug you cannot fix with confidence.
A single user request also fans out into many steps. Prompt construction, retrieval, one or more model calls, tool calls, and post-processing each shape the final answer. When that answer is wrong, any one of those steps could be the cause.
Traditional logs and application performance monitoring were built for a different shape of system. They record HTTP spans, latency, and stack traces, but not the prompt sent, the tokens consumed, or the reasoning path the model followed. The actual failure point stays invisible.
Retries and hidden state make this worse. A request may succeed on a second attempt with a different retrieved document, so the surface behavior recovers while the underlying fault goes unrecorded. You are left with an error rate and no explanation.
To debug the support assistant, you need a record of what happened inside each request. That record is a trace.
What LLM Tracing Is: Traces, Spans, And The Execution Path
LLM tracing is the practice of recording the end-to-end execution of an LLM request as a structured trace. It gives a non-deterministic system a durable, inspectable history.
The two core primitives are the span and the trace, and the distinction matters:
- Span: one unit of work, such as a single model call, a retrieval, or a tool call.
- Trace: the tree of spans produced by one request, from entry point to final response.
A span is the leaf; a trace is the tree. Each span captures its own inputs and outputs, the model and parameters used, token counts, latency, and metadata such as user or session identifiers.
Walk the trace for one retrieval-augmented request. It typically contains:
- Root span: the incoming request and the response finally returned to the user.
- Retrieval span: the query sent to the vector store and the documents it returned.
- LLM span: the assembled prompt, the model, parameters, token counts, and completion.
- Tool span: any function the model called and the result that came back.
The tree structure is what makes a trace useful. Parent-child links preserve the order and nesting of work. A flat log loses that relationship.
You can see that a tool call happened inside a specific model turn, not somewhere else. That nesting is what lets you attribute a failure to one step.
Read top to bottom, the tree shows exactly which step produced the wrong balance. The retrieval span might reveal a stale document; the LLM span might reveal a truncated prompt. Either way, you move from a guess to a located fault.
To keep these attributes portable, lean on an open standard. OpenTelemetry gives you a vendor-neutral way to capture span attributes such as prompts, token usage, and tool calls and export them to the backend of your choice [2]. Your traces are not locked to one vendor.
How To Start Tracing Your LLM Application In Four Steps
Getting started rests on one distinction. Automatic and manual instrumentation solve different problems, and you use them together:
- Automatic instrumentation: prebuilt library hooks wrap your model SDK and framework to emit spans with little or no code.
- Manual instrumentation: spans you write yourself to capture custom logic and context the automatic layer never sees.
With that clear, follow four steps to trace your LLM application:
- Instrument with OpenTelemetry so your trace data stays portable across any backend.
- Enable auto-instrumentation for your model SDK and framework to capture LLM and tool spans.
- Add manual spans around custom logic, such as retrieval and business rules, for missing context.
- Route traces to a backend where you can search, visualize the trace tree, and alert.
The code below applies steps two and three. It turns on auto-instrumentation for the model SDK, then opens one manual span around retrieval.
from opentelemetry import trace
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import OpenAI
# Automatic instrumentation: every model call becomes a span, no call-site changes
OpenAIInstrumentor().instrument()
client = OpenAI()
tracer = trace.get_tracer("support-assistant")
# Manual instrumentation: retrieval logic the auto layer cannot see on its own
with tracer.start_as_current_span("retrieve_account_context") as span:
docs = vector_store.search(query, top_k=5)
span.set_attribute("retrieval.doc_count", len(docs))
span.set_attribute("retrieval.query", query)
# The model call below is captured automatically as a child span
response = client.chat.completions.create(
model="gpt-4o",
messages=build_messages(query, docs),
)Two details make step four pay off. First, keep raw payloads attached to spans in development so you can read the exact prompt and completion. Second, set alerts on span-level signals such as latency, error status, and token counts, not just on the top-level request.
Portability decides where those traces land. The Fiddler AI Observability and Security Platform is OpenTelemetry-first and framework, model, and cloud agnostic. It ingests these traces through the gateway your team already runs, with no agent rewrites.
How Do You Trace Multi-Agent Systems Across The Agentic Hierarchy?
A single LLM call is the simple case for LLM tracing. In a multi-agent system, one request becomes a decision tree of agent calls, sub-agent outputs, tool invocations, and MCP server calls.
That decision tree is the agentic hierarchy. Tracing it means more than logging each call in isolation.
Span-level telemetry has to roll up into aggregate insights across the agent's full timeline, so you can reason about the run as a whole. This roll-up is the core of Agentic Observability.
The mechanism that makes the roll-up possible is context propagation: a shared trace and parent-span identifier flowing across service and agent boundaries [3]. That propagation is largely automatic for tools and sub-agents your framework opens directly; for hosted or provider-managed integrations, carrying the trace context across that boundary becomes the responsibility of the hosting service rather than your framework [3].
Take a research agent that spawns three sub-agents to gather sources. If the parent context does not propagate into each sub-agent, their spans orphan. You see the top-level request and a scatter of unlinked calls, and root-cause analysis stalls before it starts.
Decision lineage is what you gain from a connected trace. You can trace an incorrect final answer back through the sub-agent that produced it, the tool it called, and the input that misled it. Without that lineage, you know the run failed but not where the reasoning went wrong.
The Fiddler AI Observability platform surfaces aggregate insights across the entire agentic hierarchy. That spans first-party, third-party, and coding agents, with full execution context and decision lineage. The result is one connected view of the run, not a pile of isolated calls.
What To Watch For When You Store Trace Data
Once traces reach production, treat them as governed data. AI governance is the primary lens here, and security is one part of that broader frame. A trace store you cannot account for is a governance problem before it is a security incident.
- Raw prompts and outputs often hold PII or PHI pulled in through tools and MCP calls, making an unprotected trace store a data-exposure risk, a concern documented alongside the broader MCP attack surface, including data exfiltration and tool poisoning, that this class of risk sits within [4].
- The non-obvious misconfiguration is exporting full-payload spans to a third-party trace backend, which silently ships sensitive content outside your environment.
- Redaction of sensitive fields before spans are exported keeps developer workflows running, unlike rejecting the whole request.
- Down-sampling to cut cost hides the rare failing traces you most need, so be deliberate about what you drop.
The Fiddler AI Observability platform enforces policy inline on the request and response path, redacting sensitive content before it leaves your environment. It reserves a full block for cases like prompt injection, where the entire request must be rejected.
What Changes Once Tracing Is In Place
With LLM tracing in place, the support assistant's intermittent bug stops being a mystery. You open the failing trace, read down the tree, and find the exact span that produced the wrong answer. Unexplained incidents become reproducible, fixable ones.
Traces also become the shared substrate for evaluation. The same production traces feed a continual loop:
- Evaluate candidate changes before you deploy them.
- Observe every decision once the application is live.
- Diagnose and annotate the failures you find.
- Feed what you learn into the next iteration.
Those evaluations run on generative metrics that fit LLM output, such as faithfulness and answer relevance, rather than traditional classifier scores [5]. Scoring the same traces you already store means you evaluate real production behavior, not a synthetic proxy.
Fiddler Centor Models (formerly Fiddler Trust Models) are batteries-included, in-environment evaluators. They run directly against your traces, with no external API calls, no per-evaluation cost, and under 100ms response time.
One problem stays open. Semantic conventions for agent reasoning and tool intent are still unsettled, so traces are not yet fully portable across tools. Until that settles, cross-tool trace portability remains incomplete.
From Unexplained Incident to Located Fault
Return to the support assistant. With a trace tree, the team no longer guesses. They open the failing trace and see the exact retrieval span that pulled a stale balance into the prompt.
The next step is small. Instrument one request path with OpenTelemetry this week, then expand coverage from there.
Every request you trace becomes a request you can reason about later. As agents take on more autonomy, that recorded execution history is what keeps their behavior explainable. If you are extending tracing to multi-agent systems, Fiddler's work on Agentic Observability is a useful next read.
References
[1] B. Atil et al., "Non-Determinism of "Deterministic" LLM Settings," arXiv, 2024. [Online]. Available: https://arxiv.org/abs/2408.04667
[2] Databricks, "Observability for Any Agent, Anywhere: Production-Ready Tracing with OpenTelemetry and Unity Catalog," Databricks Blog, 2026. [Online]. Available: https://www.databricks.com/blog/observability-any-agent-anywhere-production-ready-tracing-opentelemetry-unity-catalog
[3] Microsoft, "Observability," Microsoft Agent Framework Documentation, 2026. [Online]. Available: https://learn.microsoft.com/en-us/agent-framework/agents/observability
[4] H. Errico, J. Ngiam, and S. Sojan, "Securing the Model Context Protocol (MCP): Risks, Controls, and Governance," arXiv, 2025. [Online]. Available: https://arxiv.org/abs/2511.20920
[5] NVIDIA, "Mastering LLM Techniques: Evaluation," NVIDIA Technical Blog, 2025. [Online]. Available: https://developer.nvidia.com/blog/mastering-llm-techniques-evaluation/