Key Takeaways
- High token costs come from three separate causes: inference waste, evaluation overhead, and incident exposure from unmonitored traces. Fixing one without the others just shifts the loss, it does not remove it.
- Enterprise LLM API spending hit $8.4 billion in the first half of 2025 [1], and in our experience teams routinely waste 40 to 60 percent of their token budgets on suboptimal implementations that erode ROI before anyone notices.
- Observability has to come before optimization. You cannot fix what you cannot attribute to a specific agent, workflow, or policy violation.
The Three Causes of High Token Costs
Most teams treat token costs as an API billing problem. That framing misses two-thirds of the cost. High token spend is the cumulative financial erosion that unchecked token consumption inflicts on AI ROI across production systems, and it comes from three distinct causes. Address one without the others and you simply move the loss somewhere else.
- Cause #1: Inference waste. This is the most visible category: tokens burned on over-provisioned models, bloated system prompts, missing caching, and uncontrolled output lengths. In our experience, teams commonly waste 40 to 60 percent of their token budgets here. Think of using a frontier reasoning model to make a binary yes/no classification, stuffing 80,000 tokens of context into every call when 5,000 would do, or running identical prompts thousands of times an hour with no caching. Each of these patterns consumes tokens that produce no incremental value.
- Cause #2: Evaluation overhead. Every AI output needs evaluation. The question is where that evaluation runs. When it calls an external LLM, every trace generates a per-call charge on your own LLM provider bill. This is the Evaluation Trust Tax, and it behaves differently from inference spend. It is driven by trace volume and evaluation count, not by model choice or context size, and it scales linearly with traffic. Teams can model their own exposure using the Evaluation TCO Calculator. At 100,000 traces per day with three evaluation metrics per trace, that is 300,000 external LLM calls a day added to your bill. At high volume, evaluation can cost more than the inference it was meant to monitor. Fiddler Centor Models (formerly Fiddler Trust Models) eliminate this cost by running evaluation in-environment, with no external API calls and no per-evaluation charge.
- Cause #3: Incident risk exposure. When evaluation costs become unsustainable, teams down-sample. They evaluate 10 percent of traces and hope the other 90 percent behave. Those unmonitored traces become the incident surface: hallucinated outputs reaching customers, compliance violations in regulated workflows, agent actions with real-world consequences. These incidents carry costs that dwarf the token savings from down-sampling. A single undetected hallucination in a financial advisory agent can create regulatory exposure that no amount of token optimization will offset.
These causes compound. Cut inference waste without addressing evaluation overhead and you still carry a linear cost curve on every trace. Eliminate evaluation costs by down-sampling and you expand your incident exposure. The only path to sustainable ROI addresses all three at once.
Why Token Spend Accelerates at Enterprise Scale
Token costs that look manageable in a pilot become ROI-destroying in production. The figures below describe aggregate inference and infrastructure spend, the Cause #1 problem. The Evaluation Trust Tax from Cause #2 is a separate cost that stacks on top of whatever you already pay for inference.
The Soo Group documented this pattern across enterprise deployments: a proof-of-concept that costs $50 in API calls can scale to $2.5 million per month at production volume [2]. In a separate case study from the same report, one deployment's total monthly AI operating costs, spanning API, infrastructure, monitoring, and compliance tooling, rose from $1,500 during the POC to just over $1 million in production, a 717x increase [2]. The LLM API line item alone within that deployment rose from $500 to $847,293, roughly a 1,700x increase on its own [2].
Multi-agent systems accelerate this non-linearly. Each agent call triggers sub-agent calls, tool invocations, and context window expansion. A single user request to an orchestration agent might fan out to five specialized agents, each consuming its own token budget across input, reasoning, and output. We have seen a single user interaction consume tens of thousands of tokens across the agentic hierarchy. Teams building agentic observability into their stack from the start avoid discovering these patterns after the bill arrives.
Inbound data exposure adds another dimension. Agents pulling data through MCP servers, tool endpoints, and WebFetch calls consume tokens on retrieval, not just generation. A RAG pipeline that over-retrieves documents can burn a significant share of its budget on context that never influences the output.
The broader market reflects the same pressure. Enterprise LLM API spending hit $8.4 billion in the first half of 2025 [1]. Microsoft reportedly began canceling its internal Claude Code licenses as the scale of employee usage prompted the company to reverse course [3]. GitHub shifted Copilot to usage-based billing tied directly to token consumption [4]. Even as per-token prices fall, aggregate costs rise, because agentic workloads consume orders of magnitude more tokens per task than simple completions.
The compounding effect is what catches teams off guard. As agent fleets grow, the cost surface expands across all three causes at once: more inference tokens, more traces to evaluate, and a larger unmonitored surface when teams inevitably down-sample. Enterprises building production agentic systems need cost attribution embedded in their observability layer from day one.
Getting Visibility into Token Spend in Agentic Systems
Finding where token burn comes from requires instrumentation that most teams lack. Aggregate daily spend dashboards hide the attribution data you need to act on. We recommend a four-step diagnostic approach.
- Instrument every call. Capture the model identifier, input and output token counts, the feature or agent that triggered the call, latency, and estimated cost per request. Without per-call instrumentation, cost attribution is guesswork.
- Attribute costs to agents and workflows. Per-agent and per-span attribution reveals which parts of the system burn the most. A daily total of $15,000 in token spend is not actionable. Knowing that the document-summarization agent accounts for 60 percent of that spend, and that 80 percent of its tokens go to context window stuffing, is.
- Separate inference costs from evaluation costs. Most teams lump all LLM API charges into a single line item, which masks evaluation overhead entirely. When evaluation runs against external LLMs, those calls land on the same provider bill as inference. Separating them requires tagging evaluation calls at the telemetry layer.
- Monitor for cost anomalies. Set alerts on token spend thresholds so spikes are caught before they compound. A cascading agent loop can burn through a week's budget in hours if nothing triggers an alert.
The following telemetry wrapper demonstrates the minimum instrumentation needed to capture per-call token metrics. For a deeper look at structuring telemetry signals across the agentic hierarchy.
import time
import functools
from dataclasses import dataclass, field
from typing import Any
@dataclass
class TokenMetric:
model: str
input_tokens: int
output_tokens: int
latency_ms: float
agent_id: str
call_type: str # "inference" or "evaluation"
estimated_cost_usd: float = 0.0
def __post_init__(self):
# Replace with your provider's actual rates
rates = {
"gpt-4o": {"input": 2.50 / 1_000_000, "output": 10.00 / 1_000_000},
"gpt-4o-mini": {"input": 0.15 / 1_000_000, "output": 0.60 / 1_000_000},
}
r = rates.get(self.model, {"input": 0.0, "output": 0.0})
self.estimated_cost_usd = (
self.input_tokens * r["input"] + self.output_tokens * r["output"]
)
def track_tokens(agent_id: str, call_type: str = "inference"):
"""Decorator that captures token metrics from any OpenAI-compatible response."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
response = func(*args, **kwargs)
elapsed = (time.perf_counter() - start) * 1000
usage = getattr(response, "usage", None)
if usage:
metric = TokenMetric(
model=response.model,
input_tokens=usage.prompt_tokens,
output_tokens=usage.completion_tokens,
latency_ms=round(elapsed, 2),
agent_id=agent_id,
call_type=call_type,
)
# Emit to your telemetry pipeline
emit_metric(metric)
return response
return wrapper
return decorator
def emit_metric(metric: TokenMetric):
"""Send to your observability backend. Replace with actual exporter."""
print(
f"[{metric.call_type}] {metric.agent_id} | {metric.model} | "
f"in={metric.input_tokens} out={metric.output_tokens} | "
f"${metric.estimated_cost_usd:.6f} | {metric.latency_ms}ms"
)This wrapper tags every call with the originating agent, distinguishes inference from evaluation, and computes estimated cost at the call level. The Fiddler AI Control Plane provides this attribution natively through Standardized Telemetry across the full agentic hierarchy, from application to session to agent to trace to span, with Continuous Monitoring that surfaces cost anomalies in real time.
Watch For Ways Token Costs Balloon
Even with instrumentation in place, several failure modes amplify token damage in ways that standard dashboards miss.
- Context window inflation in RAG pipelines. Poor data serialization and over-retrieval routinely consume a large share of token budgets. A retrieval step that returns 20 document chunks when 3 are relevant wastes tokens on context the model must process but gains nothing from. Serializing structured data as verbose JSON instead of a compact format compounds the waste.
- Evaluation sampling blind spots. Down-sampling to 10 percent of traces to control evaluation costs leaves 90 percent of agent actions unmonitored, and the incidents that matter most are often edge cases hiding in that unsampled majority. Centor Models remove this trade-off by enabling 100 percent trace coverage with no per-evaluation cost.
- Cascading agent loops. Multi-agent systems with retry logic or circular tool-call patterns can generate unbounded token consumption. An agent that retries a failed tool call five times, expanding its context window with each failure, can consume 10x its normal budget on a single request. Without spend thresholds, these loops run until the context window fills or a rate limit fires.
- Shadow token costs. Tokens consumed by evaluation, guardrail checks, and observability tooling never show up in the primary application's billing view, but they hit the same LLM provider bill. Teams optimizing their application's token consumption often overlook these adjacent costs entirely. When evaluation runs against external LLMs, the evaluation bill can approach or exceed the inference bill at high trace volumes.
Token Governance Is the Next Data Governance
High token spend is a structural problem, not an incidental one. It compounds across inference, evaluation, and unmonitored incidents in ways that aggregate dashboards and falling per-token prices cannot resolve. The fix starts with instrumentation and attribution, not with switching to cheaper models. A team that cannot attribute token spend to a specific agent, workflow, or policy violation cannot optimize. It can only react.
As agentic systems gain autonomy, token governance will become as critical as data governance. The organizations that build cost attribution and policy enforcement into their observability stack now will scale their AI investments. Those that optimize reactively will keep chasing bills that grow faster than the value they deliver.
Explore how Fiddler helps teams instrument, attribute, and control token spend across production AI systems. Request a demo to see per-agent cost attribution in your environment.
References
[1] Menlo Ventures, "2025 Mid-Year LLM Market Update: Foundation Model Landscape + Economics," Jul. 2025. [Online]. Available: https://menlovc.com/perspective/2025-mid-year-llm-market-update/
[2] The Soo Group, "Why Your Enterprise AI POC Will Die in Production (And How to Prevent It)," Jul. 2025. [Online]. Available: https://thesoogroup.com/blog/why-your-enterprise-ai-poc-will-die-in-production
[3] J. Angelo, "Microsoft reports expose AI's cost problem: Using the tech is more expensive than paying human employees," Fortune, May 22, 2026. [Online]. Available: https://fortune.com/2026/05/22/microsoft-ai-cost-problem-tokens-agents/
[4] GitHub, "GitHub Copilot is moving to usage-based billing," The GitHub Blog, Apr. 27, 2026. [Online]. Available: https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/