Key Takeaways
- AI overspend is an agent-behavior problem before it is a billing problem, so provider invoices and spreadsheets cannot explain it.
- Tracing overspend requires request-level and span-level telemetry that ties every token to a model, prompt, agent, workflow, and owner.
- Visibility alone does not stop spend; controlling overspend means enforcing budgets and policies inline on the request path, not after the invoice arrives.
Why AI Overspend Hides From Your Invoice
An AI bill can jump 40% while traffic stays flat, and the invoice will never tell you why. A financial services team ships a customer-support agent into production. Traffic holds steady across the month. The model bill still jumps 40%. Nobody changed the pricing, and nobody added users. The invoice reports a bigger number and nothing about why.
This is the moment most teams discover that AI spend does not behave like the rest of their software budget. Traditional cloud and SaaS costs are predictable. You pay per seat or per instance, and the line item barely moves.
AI spend is variable. It is driven by how many tokens each request consumes [1]. Token consumption is an emergent property of runtime behavior, not a fixed rate you can forecast in a spreadsheet.
Token-based pricing is the root of the volatility. The per-token price a provider lists does not determine actual cost, because hidden token consumption varies from run to run and is hard to predict [2]. Per-token pricing also varies widely across models, and leading providers bill per input and output token costs [3] [4].
The behavior compounds fast. Research on LLM agents finds that token usage for the same task is highly variable, ranging up to roughly 30 times across runs [5]. The agent reasons, calls tools, and reprocesses context on every step.
Provider dashboards do not close the distance. They report account totals or project totals. They tell you the aggregate went up. They do not tell you which workflow, which agent, or which decision drove the spike. That is the core problem this article addresses.
AI cost tracking is the practice of attributing spend to the real units of an AI system: the request, prompt, agent, workflow, and user. A monthly total is not the finest resolution available. You cannot control spend you cannot attribute.
Why Billing Dashboards and Spreadsheets Miss the Real Cost Drivers
The spreadsheet model of AI cost assumes a clean formula: tokens multiplied by rate multiplied by number of calls. Production reality diverges from that formula on the first day. Both the number of calls and the tokens per call are determined by agent behavior you did not model.
Two terms get conflated here, and the distinction decides whether your tracking is useful. Cost visibility is seeing where spend appears, the totals and trends a dashboard already shows you. Cost attribution is tying spend to the specific request, owner, and outcome that produced it.
Visibility tells you the bill grew. Attribution tells you the retry loop in the refund workflow grew it. You need the second to act.
Manual per-vendor tracking is the common fallback, and it breaks down as the stack grows. Stitching together separate provider dashboards produces only static snapshots. That creates a lag between when spend happens and when anyone notices.
By the time the monthly export lands, the expensive behavior has run thousands of times. The real cost drivers sit underneath the total, in runtime behavior the invoice never itemizes.
The Hidden Drivers: Retries, Context Growth, and Agent Loops
Overspend rarely comes from one obvious source. It accumulates from runtime behaviors that each look small per request and multiply across production volume.
- Context accumulation: Each tool result is appended to the running conversation, so every subsequent model call carries a larger payload. An agent with several tools can carry thousands of tokens of accumulated context before it generates a final response [5].
- Retry amplification: A malformed tool call triggers a retry, and the retry resends the full accumulated context. If roughly one in five calls retries, you pay at least 1.2 times the expected token cost, and more when retries land late in a session with a large accumulated context.
- Agent loops and tool invocations: A single agent run can trigger many model calls. When a loop lacks a hard bound, cost escalates quickly, and the run that should have cost cents costs dollars.
- Invisible evaluations: Automated quality checks that call a judge model consume tokens too. Left unwatched, evaluation traffic can rival primary inference cost on your bill.
These behaviors are invisible to any tool that stops at the request boundary. Seeing them requires span-level tracing that follows the work inside each run.
Trace Every Dollar to the Request, Span, and Decision That Caused It
The unit of AI cost is not the invoice. It is the span. To trace overspend, cost telemetry has to attach at the level where the behavior happens. It then rolls up so an aggregate anomaly can drill down to the exact step that caused it.
That structure is the agentic hierarchy: the full decision tree of agent calls, tool invocations, and sub-agent outputs. A single span is one model call or one tool invocation. Spans roll up into a trace for one agent run, traces roll up into a session, and sessions roll up into an application view.
Cost attached to each span means a 40% spike at the application level resolves downward. It drills to the trace, then to the span, then to the retry loop or prompt change that produced it. Span-level telemetry that rolls up to aggregate insights across the agent's timeline is what turns a bigger bill into a named root cause.
The Fiddler AI Observability and Security platform captures full execution context and decision lineage across this hierarchy, with 100% trace coverage and no sampling. Every run is recorded, so the expensive outlier is present in the data rather than discarded before you go looking for it.
Attribution is only trustworthy if it reconciles. Compare your calculated cost against the provider bill on a regular cadence. A drift between the two exposes stale price tables, unapplied cache discounts, or untagged usage that reconciliation would otherwise catch.
What to Capture on Every AI Request
Attribution is only as good as the metadata on each span. Capture too little and the totals reconcile but explain nothing. At minimum, record the following on every request:
- Provider and model: which vendor and model version served the call.
- Input and output tokens: the raw consumption that drives cost.
- run_id and step_name: which agent run and which step inside it.
- agent_name and workflow: the logical owner of the behavior.
- retry_count: how many times the step re-ran.
- team, tenant, and environment: the business owner and deployment context.
- estimated_cost: the calculated spend for the span.
A generic, OpenTelemetry-style span attribution looks like this:
from opentelemetry import trace
tracer = trace.get_tracer("ai.cost")
def record_llm_span(provider, model, usage, run_id, step_name):
with tracer.start_as_current_span("llm.call") as span:
# Identity and lineage
span.set_attribute("gen_ai.system", provider)
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("ai.run_id", run_id)
span.set_attribute("ai.step_name", step_name)
# Cost attribution tags
span.set_attribute("ai.customer", usage["tenant"])
span.set_attribute("ai.team", usage["team"])
span.set_attribute("ai.workflow", usage["workflow"])
span.set_attribute("ai.retry_count", usage["retry_count"])
# Token accounting
span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"])
span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"])
# Server-side pricing maps model -> current per-token rate
span.set_attribute("ai.estimated_cost", price(model, usage))
Keep the pricing map server-side. When a provider changes its per-token rate, you update one table and every span stays accurate. Hard-coding a rate lets it silently go stale.
From Observing Spend to Enforcing It
Observability and control are not the same thing, and the difference is where overspend actually stops. Observability tells you what spend happened. Control acts on the request before the spend, or the data, leaves your network. A tool that only observes can raise an alarm; it cannot prevent the next expensive call.
Flagging is not enforcing. An alert that fires after a runaway loop has already billed is an incident report, not governance. It documents the damage. It does not prevent it.
Treat overspend controls as part of AI governance. Governance is the primary lens, enterprise-wide visibility and control over every live, testing, and retired AI application, and policy enforcement sits within that frame. A budget you cannot enforce at runtime is a preference, not a control.
This is where inline enforcement matters. Fiddler applies allow, block, and redact verdicts on the agent request and response path in under 80ms. It runs at the gateway layer, inside your environment and before data or spend leaves your network. This is the Enforceable Policy capability: a budget or safety rule that intervenes on the request itself rather than reconciling the damage after the invoice arrives.
What to Watch For When Instrumenting AI Cost
Instrumenting cost telemetry has its own failure modes. These are the misconfigurations that quietly undermine attribution or inflate the very bill you are trying to control.
- Evaluation cost can double your bill. When quality evaluation runs against an external LLM, every evaluated trace adds a per-call charge to your own provider invoice. This is the Evaluation Trust Tax: the per-call cost enterprises incur when external LLMs are used for evaluation. It lands on your LLM provider bill, not the observability vendor's. Fiddler Centor Models eliminate it by running evaluation in-environment, with no external API calls and no per-evaluation cost. Model your own numbers with the Evaluation TCO Calculator at fiddler.ai/evals-tco-calculator, since figures vary by model, deployment size, and traffic volume.
- The observability layer has its own cost. Platforms that bill per token of trace content, or that enable playground evaluations by default, can charge you for monitoring you never consciously turned on. A managed AI service can produce exactly this surprise on the invoice.
- Sampling creates blind spots. Down-sampling traces to cut evaluation cost hides the expensive outliers you most need to see, because the runaway run is statistically rare. Full trace coverage keeps those outliers in the data.
- Untagged usage cannot be attributed. One API key serving many use cases collapses attribution into a single undifferentiated total. Separate usage at the gateway, key, or workspace boundary so each cost has an owner.
What Changes When Cost Becomes an Operational Signal
Once cost is a first-class signal attached to every span, the work shifts from reactive audit to proactive control. Teams stop waiting for the monthly export and start acting on spend the moment it deviates.
Three things become possible in daily operation:
- Real-time control: alert on abnormal spend as it happens, and enforce budgets at the routing layer before a loop runs away.
- Cost-aware model selection: choose models on a measured cost-to-quality basis instead of a guess, because you can see both signals per workflow.
- True unit economics: compute cost per inference, cost per workflow, and cost-to-serve per customer. For coding agents, that extends to cost per pull request and cost per developer across the fleet.
One problem remains unsolved. There is still no shared schema for cost telemetry across model providers and agent frameworks. Each vendor names its token fields differently, so cross-provider attribution still requires manual mapping. Until a standard emerges, the reconciliation step stays partly a human job, and that is the frontier worth watching.
Conclusion
Return to the financial services team and their 40% spike. With request-level and span-level telemetry in place, that number is no longer a mystery on an invoice. They can name the exact agent step, the retry loop, or the prompt change that caused it. They can see it days before the bill arrives.
The practical next move is narrow on purpose. Instrument one workflow end to end, capture cost on every span, and reconcile it against the provider bill until the two agree. Then expand attribution outward to shared infrastructure and the rest of the fleet. Cost becomes something you steer, not something you explain after the fact.
References
[1] Alpha Iterations, "LLM Cost Estimation Guide: From Token Usage to Total Spend," Medium, 2025. [Online]. Available: https://medium.com/@alphaiterations/llm-cost-estimation-guide-from-token-usage-to-total-spend-fba348d62824
[2] "The Price Reversal Phenomenon: When Cheaper Reasoning Models End Up Costing More," arXiv, 2026. [Online]. Available: https://arxiv.org/html/2603.23971v1
[3] C. Li, "LLM Token Prices Are All Over the Map: Formula for Unit Margin per Million Tokens," Medium, 2025. [Online]. Available: https://medium.com/@cli_87015/llm-token-prices-are-all-over-the-map-formula-for-unit-margin-per-million-tokens-11fe611b2d79
[4] "API Pricing," OpenAI, 2026. [Online]. Available: https://platform.openai.com/pricing
[5] "Token Economics for LLM Agents: A Dual-View Study from Computing and Economics," arXiv, 2026. [Online]. Available: https://arxiv.org/html/2605.09104v1