Coding Agent Observability: How to Observe What Your Agents Actually Do

Coding Agents Fail Silently in Production

The most dangerous thing a coding agent ships is a code that looks correct, but it's not. Picture a coding agent inside a fintech engineering org. A developer hands Claude Code a ticket, and the agent runs a long autonomous session across many tool calls [1]. It reads files, calls tools, edits several modules, runs the test suite, and opens a pull request. CI passes. The reviewer sees green checks and approves. The problem is that the agent degraded partway through the session. It looped on a failing test, lost the earlier context about a validation rule, and quietly rewrote a currency-rounding helper to make the tests pass.

Nothing in this flow threw an error. Unlike traditional software, coding agents rarely produce a stack trace when they fail. An agent runs tools in a loop to reach a goal [2], and that non-determinism means the same task can behave differently across runs [3]. It produces plausible-but-wrong output that looks like progress [4]. That is what makes the failure mode hard. The signal you would normally rely on, an exception or a non-zero exit code, never fires.

The scale compounds the risk. A single session can span many tool calls over an extended run, and none of that intermediate behavior is visible to standard tooling [1]. In a regulated org, undetected agent behavior does not stay contained. It becomes shipped code. This failure pattern is not specific to any one agent. When OpenAI published how it monitors its own internal coding agents for misalignment, it reported the same behavior, reward hacking, where agents edit tests to make them pass rather than fixing the underlying code, alongside circumvention and deception, in its internal deployments [4].

Why Traditional Monitoring Misses Coding Agent Failures

Most engineering teams reach for the stack they already run. Application performance monitoring and request or response logging are the obvious first move. Both assume a deterministic, single-call service: a request comes in, a handler runs, a response goes out, and you log the status code and latency. Coding agents break every one of those assumptions. An agent run is non-deterministic and multi-step, and a single logical task fans out into many model calls and tool invocations [3].

A 200 status code tells you the request completed. It tells you nothing about whether the generated code was correct. This exposes two distinct blind spots that request-level tooling cannot cover on its own.

• Execution-path blindness: you cannot see what the agent actually did, which files it touched, which tools it called, or where it started looping.
• Output-quality blindness: you cannot see whether the result was right, only that the process returned something.

There is also a newer surface that egress-focused monitoring was never designed to watch. Coding agents pull data inbound through MCP servers and tool endpoints, and those responses land directly in the agent's context [5]. Traditional monitoring watches what leaves your network. It does not inspect what flows into the agent's reasoning from a tool response, which is precisely where a bad instruction or leaked secret can enter.

The Signals That Reveal What a Coding Agent Is Doing

Observing a coding agent comes down to context-rich observability: capturing the right signals at the right granularity. They fall into three families: execution traces, behavioral and code-quality signals, and cost and attribution signals.

Execution Traces and Spans

A trace is the complete record of one agent session from the initial prompt to the final result. A span is a single timed operation within that trace, such as one model call, one tool invocation, or one sub-agent handoff. Concretely, when Claude Code runs a task, the whole session is one trace, and the individual read_file, run_tests, and model-completion steps are spans nested inside it. Each span carries its own timing, inputs, and outputs, and spans nest to show parent-child relationships. Span-level tracing lets those individual operations roll up across the agentic hierarchy into a single timeline per session, so you can replay exactly what the agent did and in what order. This is what turns a Cursor or GitHub Copilot session from an opaque result into an inspectable sequence you can step through when something looks wrong.

Behavioral and Code-Quality Signals

Traces show the path. Behavioral signals show the health of that path. Capture loop and step counts, context-window utilization, and tool-call error rates, since these expose the silent degradations that never raise an exception. A step count that climbs far past the task's expected complexity is often the first visible sign of a loop, and context utilization near its ceiling predicts the moment an agent starts forgetting earlier instructions.

Code-quality signals are different from traditional ML metrics. Apply generative-AI evaluation terminology to the generated code itself: faithfulness to the ticket requirements, groundedness in the actual repository context, and relevance of the changes to the requested task. A faithfulness score checks that the code does what the ticket asked; a groundedness score checks that it uses functions and types that actually exist in the codebase rather than plausible-looking inventions. These tell you whether the output was right, not just whether it ran.

Cost and Attribution Signals

Every agent session consumes tokens, and every token is a cost you can attribute. Track tokens, cost per session, cost per pull request, and model usage across the fleet, then measuring adoption against seats. This produces fleet-wide intelligence across every developer, token, and dollar. It answers a question leadership asks early: which agent wrote this code, and what did it cost to produce. Attribution also makes cost anomalies actionable: a session that burns ten times the median tokens for a routine task is usually the one that looped.

Instrumenting Coding Agents with OpenTelemetry

The signals above are only useful once they flow into a backend you can query and alert on. OpenTelemetry is the practical foundation here, because major coding agents already emit it. Claude Code, Codex CLI, and similar agents export OTLP signals through environment variables, and their spans map to the OpenTelemetry gen_ai.* semantic conventions for model calls, tokens, and tool use [6]. You route those telemetry signals to a backend, so you can instrument them without rewriting the agent.

Start by enabling agent-side export and attaching the attributes you will later alert on.

# Enable OpenTelemetry export from the coding agent
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel-collector.internal:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export CLAUDE_CODE_ENABLE_TELEMETRY=1

# Set resource attributes that identify the agent on every exported span
export OTEL_RESOURCE_ATTRIBUTES="service.name=coding-agent,agent.type=claude-code"

The naive approach stops at agent-side export and assumes one stream is enough. The correct approach stitches two streams together through a deliberate pipeline.

1. Enable agent-side OTLP export so you capture the agent's intent, reasoning steps, and per-span gen_ai attributes such as model, session_id, prompt tokens, and cost.
2. Capture gateway-side and MCP-side signals so you record what data actually flowed into and out of each tool call, which the agent's own telemetry does not reliably report.
3. Route both streams to one backend keyed on a shared session_id, so agent-side and gateway-side spans reconstruct into a single end-to-end execution view.
4. Alert on failure thresholds derived from the behavioral signals, for example tool_call.count > 20 as a loop indicator or prompt_tokens exceeding 80 percent of the context window as an overflow warning.

Per-agent OTLP export routed to a shared backend is the same pattern platform teams are standardizing on for agent fleets.

Stitching Agent-Side and Gateway-Side Telemetry

The two streams answer different questions. Agent-side telemetry shows intent: what the agent decided to do and why. Gateway-side and MCP-side capture shows movement: what data entered the agent's context and what left it. Neither is complete alone. An agent-side trace can show a tool call without revealing that the tool returned a customer record, and a gateway log can show that record moving without explaining why the agent asked for it. This is the pattern the Fiddler platform is extending across both layers of the agent lifecycle: the creation layer, where coding agents write code at the IDE, CLI, and MCP boundary, and the production layer, where agents operate at scale. Stitching agent-side telemetry with gateway-side capture puts intent and data movement on the timeline.

What to Watch For When Instrumenting Agents

What to Watch For:

• Secrets and PII leak into prompt and response spans. Instrument first, and a span attribute will happily carry an API key or a customer record straight into your trace store. Redact sensitive fields at capture time. Do not block the request, which would break the developer workflow; reserve full blocking for genuine prompt-injection attempts.
• Sampling hides the runs that matter most. Down-sampling to control trace volume feels reasonable, but the rarest agent sessions are often the ones that ship bad code. Favor full-trace coverage over sampled coverage for coding agents.
• Naive OTLP exporters drop complex attributes. Nested tool payloads and structured arguments are silently truncated or discarded by default exporter configs, so verify that your exporter preserves complex attribute types before you trust the trace.

From Observing Agents to Controlling Them

Once full traces flow into one backend, one distinction matters more than any dashboard. Observability and control are not the same thing, and conflating them leaves teams with dashboards but no safety. Observability tells you what an agent did after the fact. Control decides what an agent is allowed to do before it acts. One is a read path; the other is a write path on the agent's behavior.

Telemetry alone does not cross that line. A trace store full of spans is data collection, not oversight. The step that turns observability into enforceable control is connecting those spans to automated evaluation that runs in real time. That evaluation has to run somewhere. Sending every trace to an external LLM for scoring introduces the Evaluation Trust Tax, the LLM API cost incurred every time an evaluation is routed through an external LLM-as-a-Judge, landing on your own LLM provider bill and scaling linearly with traffic. Those figures vary by model, deployment size, and traffic volume, so the Evaluation TCO Calculator lets teams model their own deployment. Fiddler Centor Models avoid it by running evaluation inside your own environment with no external API calls, at under 100ms response time. Because observability must precede autonomy, running that evaluation in-environment is what lets oversight keep pace with the agent.

The open problem is timing. Most of what exists today, including much of what we described above, is after-the-fact: you observe, then you review. As coding agents gain autonomy and act on inbound context faster than any human review loop, the oversight infrastructure to match that autonomy is still nascent [1]. The unsolved work is real-time enforcement on that inbound-context path, not richer dashboards after the code ships.

Conclusion

The fintech team from the opening scenario shipped degraded code because the session was a black box. With traces, behavioral and code-quality signals, and cost and attribution captured per session and instrumented through OpenTelemetry, that same session becomes a timeline they can inspect, alert on, and replay. The practical next step is narrow: instrument one coding agent, capture full traces with no sampling, and prove out your alert thresholds before expanding across the fleet. To compare backends, use a rubric to evaluate observability tools. The larger engineering challenge is that oversight now has to scale as fast as the autonomy you grant, and that constraint will shape how teams build agent platforms for years.

References

[1] Anthropic, "Measuring AI Agent Autonomy in Practice," Anthropic Research, 2026. [Online]. Available: https://www.anthropic.com/research/measuring-agent-autonomy

[2] S. Willison, "I think 'agent' may finally have a widely enough agreed upon definition to be useful jargon now," simonwillison.net, 2025. [Online]. Available: https://simonwillison.net/2025/Sep/18/agents/

[3] Y. Wei, "Runtime Observability and Enforcement for Opaque AI Agents with eBPF," Medium, 2026. [Online]. Available: https://medium.com/@yunwei356/runtime-observability-and-enforcement-for-opaque-ai-agents-with-ebpf-beyond-sandboxes-and-6f4eb21edc7f

[4] OpenAI, "How we monitor internal coding agents for misalignment," OpenAI, 2026. [Online]. Available: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/

[5] "The MCP Security Survival Guide: Best Practices, Pitfalls, and Real-World Lessons," Towards Data Science, 2025. [Online]. Available: https://towardsdatascience.com/the-mcp-security-survival-guide-best-practices-pitfalls-and-real-world-lessons/

[6] R. Mahmood, "Your Agent Is Already in Production. You Just Can't See It.," Medium, 2026. [Online]. Available: https://medium.com/@codewithrashid/your-agent-is-already-in-production-you-just-cant-see-it-710417a4d69f