Why Cheaper Models Cost More to Complete Coding Tasks

Choosing a coding model often starts with comparing token prices. But coding agents work through multiple model responses, tool calls, and test results, so a lower token price does not always mean a lower cost per completed task.

We tested this by running a set of predefined coding tasks through the same agent loop. We began with two Anthropic models at different price tiers: Claude Haiku 4.5 and Claude Sonnet 5. Although Haiku's input and output token rates were half of Sonnet's, it cost significantly more per successful task in our study [1]. We then expanded the study to four models across two vendors.

The clearest sign something was wrong showed up in one Haiku response: 21,492 tokens describing 60 shell commands, when the agent loop only allows one command per turn. Haiku had invented the outcome of 59 commands that never ran, then declared the task complete based on a state that never existed. That single response is most of what follows: why a cheaper model can cost more, and why the tests you can see aren't always the tests that matter.

In the first evaluation, we tested five Python debugging tasks with Haiku and Sonnet, making three attempts per task and running three sweeps of it. This produced 45 runs per model and 90 runs in total. Across those runs, Sonnet cost $0.45 and Haiku cost $4.00. Figure 1 shows the cost per verified success for each sweep.

Haiku cost more per verified successful task in all three sweeps, with the difference ranging from 8.4x to 14.1x. That range is wide, and a later experiment showed why: across three repeats holding every setting constant, the same ratio came out at 3.4x, 6.2x, and 11.6x. A single unusually long Haiku response can move a sweep's total, so the size of the gap should be read as roughly an order of magnitude rather than as a precise multiple. What was consistent was the direction. Because the same pattern appeared each time, it was unlikely that a single expensive run was driving the result. Something in Haiku's behavior inside the agent loop was increasing the cost in every sweep we ran.

Databricks reported the same disconnect after benchmarking coding agents against real engineering tasks on their own multi-million line codebase, concluding that per-token price is a poor proxy for the cost of completing a task [2]. Our results matched that pattern on a smaller scale. To understand why, we examined what happened during each turn of the agent loop.

How the Study Was Organized

Every model used the same agent loop. On each turn, which we call a step, the model could issue one shell command to inspect files, edit code, or run tests. Commands had to follow a fixed format that our harness could parse. The harness executed the command and returned the result.

Each task included tests the model could inspect and additional withheld tests. We considered a run successful only if the final solution passed those withheld tests. The run ended when the model declared completion or used eight steps. We used the same loop and retry policy for every model.

The study contained three evaluations with different objectives.

Across the three evaluations, we ran 678 trials. Additional repeats and two targeted experiments brought the total above 800. Because each evaluation addressed a different question, we report the results separately. First, we examine why Haiku cost more than Sonnet despite its lower token price. Next, we show how visible tests overstated model performance. Finally, we test whether the failure patterns and cost rankings held when the coding tasks changed. We then separate the effect of the model from the effect of our own test harness, and describe what made the root cause difficult to identify.

Why the Cheaper Model Costs More

The first evaluation compared Haiku and Sonnet on five Python debugging tasks using the same agent loop. Its purpose was to understand why Haiku had a higher cost per verified success despite its lower token price.

The first question was whether Haiku cost more because it took more turns or succeeded less often. It used 1.74x as many turns, but even before accounting for success, each Haiku attempt cost 8.9x more despite its lower token price. The cost difference therefore came primarily from token usage within each turn, not from the number of turns.

Figure 3 shows the difference in response length. Sonnet's median response was 16 output tokens, compared with 361 for Haiku. At the 90th percentile, their responses were 276 and 2,980 tokens. Their longest responses were 541 and 22,721 tokens.

Haiku therefore produced much more text before the agent loop could execute a command and return the result.

A long agent response affects cost more than once. It incurs output token charges when generated, then increases input token charges on every later turn because it remains in the conversation history. Figure 4 shows this effect. Haiku's input context grew 13.7x between turns one and two, then remained between 9,000 and 10,000 tokens per request. Sonnet's context grew more gradually and reached about 1,200 tokens by turn five.

Figure 5 isolates this effect. When Haiku kept its responses concise, each step cost $0.00235, slightly less than Sonnet's $0.00244. During verbose runs, Haiku cost $0.01693 per step, which was 7.2x more than its concise runs.

To understand what made Haiku's verbose runs so expensive, we examined its longest responses. One contained 21,492 output tokens and described 60 shell commands.

In a normal turn, the model should request one command and wait for the harness to execute it and return the result. Haiku did something different. It selected a plausible first command, but instead of stopping, it predicted what the command would return, wrote that invented output into its response, and continued to the next command. It repeated this process until it had produced what looked like a complete agent session and declared the task complete.

Haiku did not appear to lack a plan. Many of the commands it selected were reasonable. The problem was that it treated its first response as an entire agent session rather than one turn within a live loop. The harness executed only the first valid command and returned the actual output. That output contradicted the result Haiku had invented, which meant its remaining commands, reasoning, and completion claim were based on a state that never existed. By then, however, Haiku had already generated the entire response.

This exposed a gap in our monitoring. The model described 60 commands, the runtime executed one, and the model gateway recorded one response containing 21,492 output tokens. Each record was accurate, but none showed that 59 of the described actions never occurred. Figure 6 places the three records side by side.

This behavior was not limited to one run. Figure 7 compares how often it appeared in Haiku and Sonnet runs using the same agent loop.

In one repeat, 83.8% of Haiku's output tokens appeared in fabricated session responses, compared with none of Sonnet's. In another repeat, 63% of Haiku's turns contained both an earlier command and a later completion signal. The parser, which reads the response and decides what the harness should do, executed the command and did not process the completion signal. Sonnet showed neither pattern.

One question this raises is whether the fabrication is really just a symptom of long responses, and whether trimming those responses alone would fix the cost problem. We tested that directly.

Because Haiku's longer responses were a major contributor to the difference, we tested whether changing the harness configuration would affect the results. We capped Haiku's responses at Sonnet's 90th-percentile response length. In two later repeats, Haiku cost 7.7x and 4.6x as much as Sonnet, with roughly one fewer verified success out of 15 runs in each repeat. Both figures fall inside the 3.4x to 11.6x range we later measured with no intervention at all, so at this sample size the effect of the cap cannot be separated from ordinary run-to-run variance.

After identifying the main source of the cost difference, we turned to whether the grading system had correctly identified successful runs. A model can pass every test it sees and still miss requirements those tests do not cover. In the next section, we examine this through the second evaluation.

Why Withheld Tests Matter

For the second evaluation, we ran four models on nine Python debugging tasks that we created and audited. Each model attempted every task three times, producing 27 runs per model and 108 runs in total.

One task asked the model to fix code for detecting overlapping calendar events. The instructions stated that if one meeting ends at 5:00 and another begins at 5:00, the meetings do not overlap. The visible tests covered only clear overlaps and gaps, so the buggy code passed them. The withheld tests included the touching boundary case and revealed whether the model had implemented the stated requirement. The expected behavior was not hidden. Only the test was withheld.

Figure 8 grades the same 108 runs twice. Using only the visible tests, every model passed all 27 runs. When the withheld tests were included, Gemini passed 26 and Haiku passed 22, while Sonnet and Opus still passed all 27.

We call a run that passes the visible tests but fails the withheld tests a quiet failure. Every failure in this evaluation was a quiet failure.

In production, this is similar to an agent passing every evaluation during testing but failing after deployment. Evaluations confirm performance only on the cases they cover, not on every situation the agent may encounter. To see whether this failure pattern extended beyond our authored tasks, we repeated the analysis on a different set of coding problems.

Why Task Selection Matters

We ran the same four models on forty HumanEval+ problems [3]. The prompt format and agent loop remained the same. Each model attempted every problem three times, producing 120 runs per model and 480 runs in total.

Verified successes and quiet failures both varied across the two task sets.

Haiku recorded no quiet failures on HumanEval+, but this did not mean it performed better. It passed 97 of its 120 runs, the lowest of the four models, and all 23 of its failures were visible: the final code failed tests the model could already inspect. In 15 of those 23, Haiku declared the task complete anyway. It had no quiet failures because its failures were loud. The same pattern from the first evaluation was present here: the harness discarded 24,709 lines of unexecuted output from Haiku's runs, against none from Sonnet's.

The task set affected both the failure patterns and the ranking by cost per verified success. Figure 9 compares the rankings for the two task sets.

Sonnet had the lowest cost per verified success on the nine tasks we created, while Haiku had the highest. On HumanEval+, Gemini ranked first, Sonnet second, Haiku third, and Opus last. The ranking changed with the coding tasks, showing why model cost should be measured on work that resembles your own.

What the Signals Miss on Their Own

Similar results have been observed elsewhere. Databricks reported roughly a twofold cost difference for the same model across different agent harnesses, largely because one harness sent smaller inputs back to the model [2].

Our monitoring captured model responses, tool actions, evaluation results, and costs as separate signals. Each signal was accurate, but no single signal explained the cost behavior. Connecting them across the complete run revealed whether model responses led to executed actions, how the evaluator judged the results, and how the full sequence affected cost. This connected view across systems is necessary to identify behaviors that contribute to inefficiency.

What It Takes to Catch Failures Before They Become Costly

This study is a small-scale version of a problem enterprises hit at scale; individual signals can each be accurate and still miss what actually happened.

At enterprise scale, the same challenges compound into significant spend and increase agent failures. Token prices alone don't show whether an agent completed a task correctly, repeated unnecessary steps, or carried a growing context into later turns. And as this study showed, visible tests alone don't show whether a run actually succeeded either.

The Fiddler Control Plane connects these signals: model responses, tool calls, evaluation results, and token usage across the complete run. That's what makes it possible to see where costs are actually coming from, compare models and agent configurations based on completed work instead of price sheets, and tell whether an issue traces back to the model or to the workflow around it.

Fiddler can also evaluate responses and enforce policy inline, at the point where the agent acts, so a fabricated session or a failure gets caught before it reaches production, not discovered after the fact. That's how Fiddler helps enterprises scale coding agents while keeping performance, risk, and spending manageable.

What This Study Cannot Tell Us

We tested two models on five debugging tasks, then four models on nine authored tasks and forty HumanEval+ problems, all using one agent setup. Results from a study like this depend on the harness, task set, and models being evaluated. The study was not designed to establish model economics across all coding workloads. Prior exposure to benchmark data may also affect performance. For example, some models may have encountered HumanEval+ during training. These findings should therefore be read as evidence from this evaluation, not as a universal ranking of model cost or capability.

See what your coding agents are actually costing you. Request a demo.

References

[1] Anthropic, "Pricing," Anthropic. Available: https://www.anthropic.com/pricing. Accessed: September 2026.

[2] Databricks, "Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase," Databricks Blog. Available: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase

[3] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation," in Advances in Neural Information Processing Systems (NeurIPS), 2023. Available: https://arxiv.org/abs/2305.01210