Key Takeaways
- Score every vendor on the same eight criteria before any demo shapes your shortlist.
- Test whether a platform scores all production traffic or a sample, because sampling decides which rare failures you miss.
- Confirm that inline checks run through the existing gateway without agent-side changes.
- Model cost at fleet scale, including external judge spend and retention, before you sign.
- Run a two to three week proof of value on real agent traffic, not demo data.
An analysis of 150 production incidents in compound AI systems found that silent degradation caused 51% of them, with outputs that passed every health check and were still wrong. Silent failures took a mean of 4.2 days to detect, against 12 minutes for crash failures [1]. Compound AI systems and agents chain model calls, tool calls and retrieved context, so a bad intermediate step can surface as a plausible final answer while latency, error rate and HTTP status stay normal.
More organizations will face this soon. Gartner’s 2026 CIO and Technology Executive Survey found only 17% of organizations have deployed AI agents, while more than 60% expect to do so within two years [2]. The teams moving agents from pilot to production need monitoring that follows the whole chain of decisions. That raises a practical buying question: what criteria should you evaluate when choosing an AI agent monitoring solution?
This guide answers it with eight criteria: tracing depth, built-in evaluation, agent-specific metrics, cost and latency attribution, alerting and guardrails, OpenTelemetry integration, deployment and security controls, and pricing at scale. For each one, it compares specialized agent observability platforms with general APM suites that have added LLM extensions, then gives you a weighted checklist to score vendors.
Quick Reference: Agent Monitoring Options Compared
Four solution categories compete for the same budget, and each one prices and differentiates differently:
| Category | Best For | Pricing Model | Key Differentiator |
|---|---|---|---|
| Specialized agent observability platforms | Production multi-step agents that need tracing, scoring and inline checks together | Per trace or per span with volume tiers; negotiated enterprise contracts | Agent-native hierarchy with evaluation and enforcement on the same data |
| Open-source tracing frameworks | Teams that must keep prompts on their own servers and can run the infrastructure | License free to self-host; cost moves to infrastructure and staff; some enterprise features license-gated | Data ownership and inspectable code |
| APM suites with LLM extensions | Organizations standardized on one APM vendor | Per span, host or GB ingested, with retention add-ons | LLM spans correlated with infrastructure metrics and user sessions |
| Hyperscaler-native monitoring tools | Agents built and run inside a single cloud | Consumption billing for telemetry ingest, storage and queries | Native fit with that cloud’s identity, logging and model services |
Why Agents Need Monitoring Built for Multi-Step Behavior
Agent failures hide inside steps that each look successful. A request-response service either returns the right payload or throws an error, so monitoring that watches known paths for known failures works. An agent reasons across steps, picks tools, reads their results and adapts. A tool can return HTTP 200 with an empty payload, the agent can treat that as real data, and every span in the trace stays green.
Standard tracing alone misses many agent-level faults. In the AgentTelemetry benchmark, standard OpenTelemetry tracing, with or without its GenAI extensions, detected only 6 of 14 injected agent fault types [3]. For their own agents, teams should therefore assess signals that infrastructure tracing does not report on its own:
- Whether the agent chose the right tool.
- Whether its arguments made sense.
- Whether the final state matched the goal.
Non-determinism compounds the problem. The same prompt can take a different path on the next run, so a single passing test says little about the next thousand runs. Errors also propagate. An early wrong assumption gets inherited by later steps, and the visible failure often sits several steps after the decision that caused it.
APM alone is enough in narrower cases. A single model call behind a fixed workflow, with deterministic validation on the output and low business impact, can run on LLM spans inside an existing APM suite. Once an agent selects tools, passes context between sub-agents or acts on external systems, the team needs quality signals per step and enforcement that can allow, block or redact an action according to the policy for that use case, not only a record of latency.
Eight Criteria That Separate Agent Monitoring Solutions
Specialized agent platforms lead on most criteria that concern agent behavior, and APM suites lead where the question is correlation with the rest of the stack. Each subsection below covers one criterion and what to ask vendors about it.
Tracing Depth: Capture Spans, Sessions and Multi-Step Reasoning
Deep tracing lets an engineer go from an alert to the exact failing step. The structure that makes this possible is the agentic hierarchy, five nested levels: application, session, agent, trace and span. An application contains user sessions. A session contains the agents invoked. Each agent run produces traces, and each trace is a tree of spans for model calls, tool calls and retrieval.
The OpenTelemetry GenAI conventions describe the span tree for one agent run: a top-level invoke_agent span, a child chat span for each model call and an execute_tool span for each tool invocation [4]. A vendor that stores only flat LLM calls loses the parent-child links that explain why a tool ran. Ask each vendor to show:
- Session reconstruction across multiple turns and sub-agents.
- Tool spans with arguments and responses, not only model spans.
- Drill-down from an application-level alert to the specific span that failed.
- Time from instrumentation to visibility in the platform, measured on your traffic.
Prompt and response content needs an explicit decision. Under the OpenTelemetry conventions, message content is captured only when the team opts in [4]. Without content, an engineer can see that a step happened but not what the agent read or said.
Built-In Evaluation: Score Production Traffic Online and Offline
Evaluation shows whether an agent’s output was good, which traces alone cannot. Teams need offline evaluation on curated datasets before release and online scoring on production traffic after it.
Generative-AI checks fall into a few groups:
- Faithfulness and groundedness: Whether each claim in a response is supported by the retrieved context, which catches RAG answers that invent policy details.
- Toxicity: Whether output contains harmful or abusive content.
- Jailbreak and prompt injection: Whether input tries to override the system’s instructions.
- PII/PHI exposure: Whether prompts, tool returns or outputs carry personal or health data.
Where evaluation runs matters more than the scoring technique. An LLM-as-a-Judge setup sends each prompt and response to an external model provider, which adds per-call cost and moves data outside your network. Purpose-built evaluator models that run in-environment keep data inside the boundary and make cost depend on provisioned capacity rather than call count.
That cost model drives the sampling decision. OpenTelemetry’s own documentation states that head sampling alone cannot ensure every trace containing an error is kept [5]. Full-coverage scoring removes that blind spot, and the rare high-impact failure is the one that matters most.
Agent Metrics That Map to Real Failure Patterns
Useful agent metrics each correspond to a specific, nameable failure. Published benchmarks show why final-answer accuracy is not enough. In τ-bench, state-of-the-art function-calling agents succeeded on fewer than 50% of tasks, and pass^8, the chance that all eight repeated trials of a task succeed, fell below 25% in the retail domain [6]. An agent that passes once can still fail most of the time.
Map each metric to the failure it detects:
- Tool selection accuracy: Detects an agent choosing the wrong tool or skipping a required one.
- Tool-call correctness: Detects malformed or wrong-value arguments, and agents treating empty tool responses as valid data.
- Task completion: Detects runs that end in the wrong final state even when every step returned successfully.
- Consistency across repeated runs: Detects agents that succeed on one attempt and fail on the next, which pass^k measures [6].
- User feedback and human review: Detects failures automated checks miss, such as answers that are technically grounded but unhelpful.
- Failure clustering: Groups similar failing sessions so engineers fix a pattern instead of chasing individual traces.
Ask vendors which of these ship out of the box and which require custom evaluators.
Cost and Latency Tracking That Attributes Spend per Step and User
Agent cost grows with every step, so cost tracking has to attribute spend at the step level. Anthropic reports that, in its data, agents typically use about 4 times more tokens than chat interactions, and multi-agent systems about 15 times more [7]. Each turn re-reads prior context, retries add calls, and a looping agent can multiply spend while every call succeeds.
The OpenTelemetry GenAI spans record the model and the input and output token counts for each LLM call [4], which supplies the raw material. The monitoring tool has to roll those counts up along the agentic hierarchy and across organizational dimensions:
- Per request and per step, to find the tool loop or retry storm driving a spike.
- Per user and per session, to find outlier usage.
- Per team, application and model, to support showback.
The FinOps Foundation recommends reporting AI costs through showback by project and workload before moving to chargeback. It also lists agent loops and prompt changes that double token use as anomalies to watch, and recommends monitoring tokens per minute at the application level, not only daily spend [8]. Latency needs the same treatment, broken down per step, so engineers can separate slow tools from slow models.
Alerting and Guardrails That Act Before Bad Output Reaches Users
Alerts report that a problem happened, and guardrails decide whether it reaches a user or a downstream system, allowing, blocking or redacting it according to the policy for that use case. Agents need both, because on-call engineers review detected actions after execution, while only inline enforcement can block them beforehand.
Inline enforcement runs two kinds of check. Request checks run before the model is invoked, for example redacting a customer’s account number from a prompt. Response checks run before output reaches the user or a downstream system, for example blocking an answer that is not supported by retrieved context.
Enforcement sits in the request path, so response time is a hard constraint. Evaluate guardrails against a target of under 80ms response time, measured at your traffic volume and payload size, not on a vendor benchmark. Check also that enforcement decisions are logged with the session and agent that triggered them.
For alerting, confirm routing into the tools your on-call team already uses, such as PagerDuty, Slack and Microsoft Teams, with severity mapping. Alerts should fire on evaluation thresholds (faithfulness dropping on a specific agent) as well as on operational signals.
OpenTelemetry and Integration Methods: Keep Instrumentation Portable
OpenTelemetry provides a safeguard against lock-in. When agents emit standard traces, a team can switch or add backends without rewriting application code. Three integration methods exist, and each sees a different slice of agent behavior:
- Native SDK: Instruments the agent framework in process, capturing tool calls, sub-agent handoffs and prompt assembly. It requires code changes in each agent.
- Proxy: Sits in front of model providers and sees every model call with minimal code change, but misses tool calls and orchestration that never leave the process.
- Existing gateway: Attaches monitoring and enforcement to the LLM or MCP gateway the team already runs, which covers third-party agents the team cannot modify without adding a new hop.
Check framework coverage against your actual stack. LangGraph, Google ADK and Amazon Bedrock agents each support OpenTelemetry export differently, and coverage for TypeScript, Go and Java agents is often thinner than for Python.
The OpenTelemetry GenAI semantic conventions are still at Development stability. The project tracks stabilization of the core inference and agentic execution conventions in an open issue, and until that work completes, attribute names can still change [9]. Ask vendors how they handle attribute renames between releases.
Deployment, Security and Retention Controls for Stored Prompts
Stored prompts and responses can contain personal data, health data and credentials, so the deployment model decides who else holds that data. Common options are SaaS, a dedicated VPC in the buyer’s cloud account, on-premise, and AWS GovCloud for workloads with government compliance requirements. Vendors in financial services, healthcare and government deals are often filtered out on this criterion before any feature comparison.
Data exposure extends past model-provider traffic. Prompts, tool returns, file reads, outputs and coding-agent sessions all flow into the monitoring store. Evaluate:
- Detection and redaction: Whether PII, PHI and secrets are detected and redacted on both request and response paths before storage.
- Residency: Which regions store data, and whether evaluation calls send content to an external judge in another jurisdiction.
- Retention: Configurable retention per data type, and whether longer retention is a paid add-on.
- Access controls: SSO, role-based access and audit logs, and which plan tier includes them.
Health data raises the bar. HHS states that a cloud service provider that creates, receives, maintains or transmits electronic protected health information (ePHI) for a covered entity is a business associate, even if it stores the data encrypted without holding the key [10]. A monitoring vendor that stores prompts containing ePHI therefore needs a business associate agreement. A SOC 2 Type 2 report shows that an independent auditor tested the vendor’s controls over a period, but it does not cover the buyer’s own configuration or replace HIPAA obligations. Treat any certification as support for compliance evidence, not a guarantee of compliance.
Pricing at Scale: Forecast Cost From Pilot to Fleet
The pricing unit determines how the bill grows. Per-trace and per-span pricing tracks traffic volume, and one agent workflow can produce many spans. Per-seat pricing tracks headcount and compounds when more teams adopt. Per-agent pricing becomes hard to forecast as fleets grow. Retention add-ons and query charges sit on top of all three.
Evaluation adds its own cost layer, and evaluation TCO has three components:
- Evaluation Trust Tax: The per-call API cost of sending prompts and responses to an external LLM judge.
- Incident Risk Exposure: The cost of incidents that go undetected because evaluation down-samples traffic.
- Operational Overhead: The cost of building and maintaining custom evaluation infrastructure.
Model all three at your projected production volume, not pilot volume. Estimate evaluation TCO with fiddler.ai/evals-tco-calculator. Results vary by model, deployment, and traffic.
Specialized Platforms and APM Suites Compared on the Same Criteria
The following table compares how the three main commercial categories typically perform on each criterion:
| Criterion | Specialized Agent Platforms | APM Suites | Hyperscaler-Native Tools |
|---|---|---|---|
| Tracing depth | Agent-native hierarchy with session and tool spans | LLM spans correlated with infrastructure and user sessions | Strong for agents on the provider’s runtime; extra setup elsewhere |
| Built-in evaluation | Built-in evaluators; some score in-environment at full volume | Evaluation often added later, frequently via external judges | Built-in evaluators, often tied to the provider’s services |
| Agent metrics | Tool, task and session metrics out of the box | Agent metrics vary by suite | Tool-call and task metrics, some in preview |
| Cost attribution | Per step, session, user and team | Token and cost metrics on LLM spans | Cost through cloud billing and tagging |
| Alerting and guardrails | Inline request and response checks in some platforms | Mature alert routing; enforcement often upstream at the provider | Provider guardrails, scoped to the provider’s own services |
| OpenTelemetry | Usually OTel-native | OTLP ingestion common | OTel export, best inside the provider’s cloud |
| Deployment | SaaS to VPC, on-premise and government cloud, varies by vendor | Mostly SaaS | The provider’s cloud |
| Pricing at scale | Per trace or span; evaluation cost model varies | Per span, host or GB plus retention | Consumption billing across several services |
What Each Solution Category Delivers in Production
No single category wins every criterion, and the tables above show where each one leads. The weaknesses below are the ones that usually decide fit after those strengths are weighed.
Specialized Agent Observability Platforms
These platforms are built around agent traces, evaluation and, in some cases, inline enforcement. They model the agentic hierarchy directly and connect evaluation results to production alerts. Weaknesses include:
- A separate system from the APM suite that operations teams already watch.
- Uneven deployment options across vendors, with self-hosting often limited to enterprise tiers.
- Recent acquisitions that may change product roadmaps and pricing.
Open-Source Tracing Frameworks
Open-source frameworks provide tracing and basic evaluation under a permissive or source-available license, with the option to run everything on a team’s own servers. Weaknesses include:
- Self-hosting burden: A typical production stack combines a transactional database, an analytical columnar store, a queue or cache and object storage, each needing upgrades, backups and scaling.
- Staffing load: Staff must handle upgrades, version migrations and on-call.
- Gated features: Audit logs, retention policies and fine-grained access control can require a commercial license.
- Evaluation depth: Tracing is often strong while evaluation and runtime enforcement are thin or absent.
APM Suites With LLM Extensions
APM suites extend existing tracing and metrics products with LLM and agent spans, token and cost metrics, and evaluation features. Weaknesses include:
- Roadmap tied to APM: AI features compete with the vendor’s core monitoring priorities.
- Cost forecasting: Span, host and retention meters can make agent traffic expensive to predict.
Hyperscaler-Native Monitoring Tools
Cloud providers bundle agent tracing, evaluation and guardrails into their agent runtimes and model platforms. Weaknesses include:
- Single-cloud fit: Agents running elsewhere need extra instrumentation, and cross-cloud features are often in preview.
- Fragmented billing: Tracing, storage, evaluation and guardrails often bill as separate consumption lines.
How Fiddler Maps to Each Criterion
The eight criteria above all draw on the same production traffic, and the platform turns that traffic into traces, evaluation scores, enforcement decisions and cost data for a team. The Fiddler AI Observability and Security platform covers first-party agents teams build, third-party agents they deploy, and coding agents developers use. Each criterion maps to specific platform behavior:
- Tracing depth: Fiddler traces the full agentic hierarchy (application, session, agent, trace, span).
- Built-in evaluation: Batteries-included Fiddler Centor Models score every prompt and response in-environment. Scoring runs inside the existing environment, so data stays there and evaluation covers full production traffic instead of a sample. Centor Models do not sample, and their costs are a step function compared to LLM costs that grow linearly.
- Cost and latency attribution: Token usage, cost per session and fleet-wide cost analytics break down spend by user and team. For coding agents, fleet intelligence reports cost per pull request, adoption against purchased seats and model usage across the developer fleet.
- Alerting and guardrails: Inline enforcement runs request checks and response checks with under 80ms response time. Prompt injection and jailbreak detection run on the same path. Nielsen reports 98% accuracy on persona-play attacks against its Ask Nielsen multi-agent copilot. Alerts route to PagerDuty, Slack, Microsoft Teams, email and webhooks.
- Integration methods: Teams can deploy Fiddler directly or through the LLM or MCP gateway they already run, such as LiteLLM, AgentGateway or Kong Gateway, with no new infrastructure, agent-side integration or SDK to embed, including for third-party agents a team cannot modify. For in-process instrumentation,
fiddler-otelis the OpenTelemetry-native foundation, with framework SDKs for LangGraph, LangChain, AWS Strands Agents and Google ADK, plus JavaScript and TypeScript packages. - Deployment and security: Deployment options are SaaS, VPC (including the Amazon SageMaker Partner AI App), on-premise, and AWS GovCloud. Centor Models run inside the deployment with zero data egress, meaning evaluation and enforcement happen in-environment and prompt data is not sent out.
- Governance evidence: Every enforcement decision is recorded with who triggered it and what action was taken, and teams can keep track of every deployed agent in a registry built from the telemetry Fiddler already ingests. These records support compliance evidence; they do not guarantee compliance.
Weighted Scoring Checklist Teams Can Use to Rate Vendors
The suggested weights below assume production agents with moderate regulatory exposure. Adjust them to your environment, then score each vendor from 1 to 5 per criterion on evidence from your own proof of value:
| Criterion | Suggested Weight | What to Test in a Proof of Value |
|---|---|---|
| Tracing depth | 15% | Reconstruct a multi-turn, multi-agent session; drill from an alert to the failing span; measure time to visibility |
| Built-in evaluation | 15% | Confirm whether online scoring covers 100% of traffic; check where evaluation runs and whether content leaves your network |
| Agent metrics | 10% | Verify tool selection, tool-call correctness and task completion metrics on your agents without custom code |
| Cost and latency attribution | 10% | Attribute one day of spend to team, application, model and step; reproduce a cost spike |
| Alerting and guardrails | 15% | Measure enforcement response time at production load; test allow, block and redact paths; route an alert to on-call |
| OpenTelemetry and integration | 10% | Instrument one agent per framework you run; test gateway-based coverage for a third-party agent |
| Deployment, security and retention | 15% | Deploy in your target environment; confirm redaction before storage, residency, retention settings and the BAA position |
| Pricing at scale | 10% | Get a quote at 10 times pilot volume, including evaluation, retention and query costs |
How Platform Teams Should Choose by Profile
An organization’s constraints decide which criteria to weight highest.
Regulated enterprise. Deployment and security, guardrails and governance evidence carry the most weight. Shortlist vendors that run evaluation and enforcement inside your VPC, on-premise or in AWS GovCloud, redact PII and PHI before storage, and record every enforcement decision. Rule out any vendor whose evaluation sends prompt content to an external judge unless your data policy permits it.
APM-standardized organization. Start by testing whether your APM suite’s LLM extension handles your agents’ actual complexity. If agents are simple, it may be enough. If they call tools and hand off between sub-agents, add a specialized platform for evaluation and enforcement, and export its metrics into the APM console so on-call engineers keep one alert path.
Early-stage team. Weight tracing depth, OpenTelemetry portability and pricing at scale. Instrument with OpenTelemetry from the first agent so a later backend change costs configuration, not code. Choose a vendor whose pricing unit you can project to production volume, and avoid tools that will force a re-platform when governance requirements arrive.
Whatever the profile, run a two to three week proof of value on real production or shadow traffic. Measure enforcement response time, evaluation coverage, framework fit and projected cost against the checklist above, and compare vendors on the same traffic.
Request a demo to review how inline enforcement and in-environment evaluation would fit the existing agent architecture and deployment environment.
References
[1] R. K. Paul and S. Nandy, “Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents,” arXiv preprint arXiv:2610.02503v1, 2026. Accessed: Nov. 12, 2026. [Online]. Available: https://arxiv.org/html/2610.02503v1
[2] Gartner, “2026 Hype Cycle for Agentic AI,” reporting the Gartner 2026 CIO and Technology Executive Survey, Gartner, 2026. Accessed: Nov. 12, 2026. [Online]. Available: https://www.gartner.com/en/articles/hype-cycle-for-agentic-ai
[3] K. C. Balusu, “AgentTelemetry: A Fault Detection Benchmark and Toolkit for LLM Agent Observability,” in Proc. 3rd ACM Int. Conf. AI-Powered Software (AIware), Jul. 2026. Accessed: Nov. 13, 2026. [Online]. Available: https://dl.acm.org/doi/10.1145/3805760.3814931
[4] OpenTelemetry, “Inside the LLM Call: GenAI Observability with OpenTelemetry,” OpenTelemetry Blog, 2026. Accessed: Nov. 12, 2026. [Online]. Available: https://opentelemetry.io/blog/2026/genai-observability
[5] OpenTelemetry, “Sampling,” OpenTelemetry Documentation. Accessed: Nov. 12, 2026. [Online]. Available: https://opentelemetry.io/docs/concepts/sampling
[6] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,” arXiv:2406.12045, Jun. 2024. Accessed: Nov. 12, 2026. [Online]. Available: https://arxiv.org/abs/2406.12045
[7] Anthropic, “How we built our multi-agent research system,” Anthropic Engineering, Jun. 2025. Accessed: Nov. 12, 2026. [Online]. Available: https://www.anthropic.com/engineering/multi-agent-research-system
[8] FinOps Foundation, “FinOps for AI Overview,” FinOps Foundation. Accessed: Nov. 12, 2026. [Online]. Available: https://www.finops.org/wg/finops-for-ai-overview
[9] OpenTelemetry, “Stabilize inference and core agentic execution semantic conventions,” Issue #511, open-telemetry/semantic-conventions-genai, GitHub, updated Nov. 6, 2026. Accessed: Nov. 12, 2026. [Online]. Available: https://github.com/open-telemetry/semantic-conventions-genai/issues/511
[10] U.S. Department of Health and Human Services, “FAQs Category: Business Associates,” HHS.gov. Accessed: Nov. 12, 2026. [Online]. Available: https://www.hhs.gov/answers/business-associates/index.html
Frequently Asked Questions
Can APM Suites Handle Multi-Step Agents?
APM suites can record agent spans, token counts and latency alongside infrastructure metrics, which covers operational visibility well. They tend to fall short on quality evaluation and inline policy enforcement, which are where multi-step agents fail silently. If your agents select tools or act on external systems, pair the APM suite with agent-specific evaluation and guardrails.
Which Agent Metrics Are Essential?
Start with tool selection accuracy, tool-call correctness, task completion, faithfulness for retrieval-based answers, PII/PHI exposure and cost per session. Add consistency across repeated runs once there is enough traffic, because an agent that passes once can still fail most of the time on the same task.
Does OpenTelemetry Prevent Vendor Lock-In?
OpenTelemetry reduces lock-in by letting teams emit traces once and send them to any compatible backend. It does not remove lock-in entirely. The GenAI semantic conventions are still at Development stability, so attribute names can change between releases, and vendor-specific evaluation scores and policies do not transfer automatically.
Should Teams Choose Open Source, Self-Hosted or SaaS?
Open source gives data control and inspectable code, but the team runs the databases, upgrades and on-call. Self-hosted commercial platforms keep data inside the network with vendor support. SaaS is fastest to start but moves prompt data to the vendor. Regulated workloads usually narrow the choice to self-hosted or VPC deployment.
How Should Teams Think About Evaluation Cost and Sampling?
Evaluation cost depends on where scoring runs. External LLM judges bill per call, which pushes teams to sample, and sampling risks missing the rare sessions that cause incidents. In-environment evaluator models make cost depend on provisioned capacity, which makes full-coverage scoring practical. Model the full evaluation TCO at production volume before deciding.
