What Is Agent Evaluation?

Single-Turn Scores Miss Multi-Step Agent Failures

Picture an insurance claims agent that returns a polished explanation of a denied claim. The final message is coherent, polite, and on-brand, and offline text checks score it highly for relevance and tone. But in that same run, the agent called the wrong policy lookup API, invented a coverage code, and retried the same tool three times before it ever drafted that reply.

Single-turn LLM evaluation grades the last message. Agent evaluation has to grade the path that produced it, because agents plan, select tools, form arguments, update state, and decide when to stop, and failure can sit anywhere in that chain even when the closing text looks correct [1]. A correct-looking answer built on a bad tool path can still leave the workflow wrong in production. Score both the outcome and the ordered reasoning, tool, and observation path that produced it.

Agent Evaluation Means Scoring Outcomes and Paths

Agent evaluation is the systematic measurement of how well an agent plans, uses tools, and completes tasks against explicit success criteria. It goes beyond single-response quality [1].

That broader scope exists because generative AI agents run multi-step reasoning, call tools, and change external systems, so evaluation has to cover behavior and task success, not just surface text  [2].

Use Three Scopes to Isolate Failures Faster

Three scopes answer different debugging questions [3]:

1. End-to-end outcome: Did the final state or user-visible result meet the goal?
2. Trajectory: Was the ordered path acceptable on required steps, forbidden actions, and efficiency?
3. Component: Did a single span choose the right tool, arguments, or retrieval hop?

Outcome-only scoring can hide inefficient or unsafe paths, and trajectory-only scoring can miss incomplete goals when the path looks tidy [3]. Component scores are what localize root cause once an end-to-end check has already failed. In practice, the three scopes work together: gate ship decisions on end-to-end outcomes, then debug with trajectory and component scores [1] [3]. Black-box outcome scoring can stand alone for a simple goal check, but trajectory and component scoring both require ordered traces or spans to work at all [3].

Define Tasks, Trials, and Graders Clearly

A clear vocabulary keeps the harness honest [4]:

1. Task: One test case with inputs, environment setup, and success criteria.
2. Trial: One attempt at a task; run multiple trials because agents are non-deterministic.
3. Grader: Logic that scores part of the transcript or the outcome.
4. Transcript: Full trial record of messages, tool calls, and intermediate results.
5. Outcome: The final environment state after the trial ends.

Success criteria need to be explicit and checkable: binary gates for safety and policy, weighted or hybrid thresholds for multi-part quality [4]. Two related metrics matter here. Pass@k measures whether at least one of k trials succeeds; pass^k measures whether all k trials succeed. For customer-facing agents, Anthropic emphasizes consistency metrics like pass^k over best-of-k peak scores, since a customer only sees one attempt  [4]. This structure, a defined claim, repeatable trials, and graders you can defend in review, is the core of reliable evaluations across the agent lifecycle.

Build Tasks and Success Criteria You Can Grade

Write Tasks That Mirror Real Workflows

Source tasks from production failures and high-risk user journeys, not only synthetic prompts [4]. Each one should include:

• The tools available in the environment and the expected side effects
• Hard constraints such as PII handling, allowed actions, and policy limits [1]

Anthropic separates two suite types early [4]:

1. Capability evals: Hard tasks with a low initial pass rate that expose what still fails.
2. Regression evals: Near-100% keepers that protect known behaviors after every change.

Capability suites are the hill to climb. Regression suites are the backslide guard [4].

A worked example makes this concrete. Take a fintech support task called Check refund eligibility:

• Task: Check refund eligibility
• Inputs: customer_id, order_id, partial chat history
• Tools available: get_order, get_payment, create_refund_case
• Constraint: no refund above policy without a human handoff
• Expected side effect: a correctly typed case record, not a free-form apology

Turn Goals Into Checkable Success Criteria

Split criteria into outcome checks and path checks.

1. Outcome criteria: Final state is correct, required artifacts exist, and safety constraints hold.
2. Path criteria: Required tools called, forbidden tools avoided, step budget held, no empty loops.

Use partial credit on multi-component tasks when a partial solve still has operational value, and avoid brittle exact-string matches when several valid answers exist; state assertions plus rubrics handle open-ended quality better [4]. Carry the same criteria from pre-production gates into continuous monitoring on live traces. Production and offline definitions of success must match. Otherwise offline green scores stop predicting live behavior [1].

task = {
    "id": "refund_eligibility_v3",
    "input": {
        "customer_id": "C-10482",
        "order_id": "O-77821",
        "user_message": "I was charged twice for order 77821. Can I get a refund?"
    },
    "environment": {
        "tools": ["get_order", "get_payment", "create_refund_case"],
        "initial_state": {"refund_cases": []}
    },
    "success_criteria": {
        "outcome": [
            "refund_case.exists",
            "refund_case.amount == duplicate_charge_amount",
            "refund_case.status in {\"pending_review\", \"approved\"}",
            "no_pii_in_user_visible_text"
        ],
        "path": [
            "called.get_order",
            "called.get_payment",
            "not_called.issue_refund_direct",
            "tool_calls <= 6",
            "no_identical_retry_loop"
        ]
    },
    "graders": [
        {"type": "code", "name": "state_assertions"},
        {"type": "code", "name": "tool_trace_checks"},
        {"type": "model", "name": "tone_and_policy_rubric"},
    ],
    "trials": 5,
    "aggregate": "pass_at_k_and_mean_score"
}

def grade_trial(trial, task):
    results = []
    for grader in task["graders"]:
        results.append(run_grader(grader, trial.transcript, trial.outcome))
    return aggregate(results, task["aggregate"])

Choose Graders That Match What You Need to Prove

Combine Three Grader Types for Speed and Coverage

Anthropic recommends mixing three grader families in agent suites [4]:

1. Code-based: Schema, tool-call, state, and budget checks that stay fast and objective.
2. Model-based: Rubrics for plan quality, groundedness, tone, and open-ended synthesis.
3. Human: Subject-matter review for edge cases and gold labels that calibrate judges.

Combine graders per task, requiring all of them to pass for safety and policy checks, and weighting quality dimensions when a partial solve still has operational value [4]. LLM-as-a-judge is a valid technique here, as long as it's paired with tight rubrics and periodic human calibration. The harder operational question is where that judge runs and how its cost scales with trace volume.

Score Tool Use and Efficiency Explicitly

Score tool use on its own merits: right tool, right arguments, sensible order, and correct interpretation of tool output [5]. Documented tool-path failure modes include hallucinated tools, missing required parameters, wrong types, and invented fields [1], and function-call checks catch the same wrong names, missing parameters, and bad types [2]. Efficiency belongs on the scorecard too: track cost per run, latency, iteration count, and token usage [1]. Treat safety and policy adherence the same way, as explicit criteria rather than afterthoughts checked only on the final message [2].

High evaluation volume raises what we call the Evaluation Trust Tax: the per-call cost that shows up on your LLM provider's bill when external models judge your traces. Fiddler Centor Models (formerly Fiddler Trust Models) avoid that entirely:

• Batteries-included and in-environment, so evaluations run inside your own environment
• No external LLM call required for the Out of the Box and Customizable paths, and no Evaluation Trust Tax on either one
• Results in under 80ms
• 100% of traces covered, with no sampling by default

The Fiddler platform is model- and cloud-agnostic across providers such as Azure OpenAI, Amazon Bedrock, and LangGraph.

# Same groundedness grader id in pre-prod gates and production traces
eval_def = {
    "name": "refund_groundedness_v1",
    "judge": "centor_out_of_the_box",
    "metric": "groundedness",
    "threshold": 0.85,
    "coverage": "all_traces",  # prefer full coverage over sampling
}

def score_trace(trace, eval_def=eval_def):
    # In-environment path: no external LLM call for this judge family
    return run_centor_evaluator(
        model=eval_def["judge"],
        metric=eval_def["metric"],
        trace=trace,
        threshold=eval_def["threshold"],
    )

preprod_pass = all(score_trace(t)["pass"] for t in offline_suite_traces)
prod_flags = [score_trace(t) for t in live_traces_last_24h]  # e.g. 100K traces/day

What to Watch For When Agent Evals Mislead

A harness can look healthy for months and still miss the failure that reaches your customers. Most of the time, it comes down to one of six recurring design mistakes.

Six eval design mistakes that create false confidence:

1. Graders reject valid solutions when exact-match strings or fixed tool sequences are too rigid [4].
2. Contaminated environments let the agent read prior-trial artifacts and inflate scores [4].
3. High pass@k hides weak first-try consistency that customer-facing flows require [4].
4. Offline golden sets drift from production tool schemas, prompts, or policies [1].
5. Sampled production traces miss rare failures that never re-enter the suite [1].
6. Judge drift accumulates when model-based graders run without periodic human calibration [4].

These failure modes trace back to brittle graders, non-isolated trials, and the tradeoffs that come with non-determinism [4]. A green dashboard and a red production incident aren't actually in conflict when the harness is measuring the wrong claim or the suite has drifted from production, since stale task distributions and uncalibrated graders simply stop matching live behavior [4] [1]. The fix is to score production traffic continuously and feed new failures back into the offline datasets, so the suite stays aligned with what's actually happening in production [1]. With Centor Models, full-trace coverage stays practical for this because in-environment evaluation doesn't bill a per-trace external API call and doesn't sample by default.

Run the Same Criteria From Pre-Production Into Production

Treat offline suites as quality gates, using capability and regression thresholds as ship criteria before promotion [4]. When production traces fail, feed them back into the reference dataset so new failure modes stay covered as the suite evolves [6]. Keep judge definitions stable across stages, since pre-production and production need the same groundedness and policy definitions for offline green scores to actually map to live behavior [1].

The Fiddler AI Observability and Security Platform supports that continual loop. See the product overview for how span-level telemetry rolls up into aggregate insights across the agent's timeline and broader agentic hierarchy. In practice, the loop looks like this:

1. Evaluate before you deploy.
2. Observe decisions in production.
3. Annotate failures.
4. Feed failures back into the next iteration with the same evaluation definitions.

Multi-agent handoffs and coding-agent fleets need success criteria that roll up across the agentic hierarchy, not only the leaf span, because leaf-span scores alone can miss failures that cross sub-agents, shared tools, and long-running developer workflows. Design agent measurability before you scale autonomy.

Close the Loop Before You Scale Autonomy

Agent evaluation is not a last-message quality spot check. It's a harness of tasks, success criteria, graders, and trials that scores both outcomes and paths. Start with a small set of realistic workflows, write state-checkable criteria, and mix code, model, and human graders. Then close the loop: offline gates decide what ships, and production traces keep the suite honest.

1. Pick one high-risk workflow.
2. Write five tasks with explicit outcome and path checks.
3. Run multi-trial baselines.
4. Promote nothing that fails the bar you just defined.

When you are ready to operationalize the harness, review eval-driven development patterns for agentic applications.

References

[1] Databricks, "What is AI Agent Evaluation?," Databricks Blog, Mar. 6, 2026. [Online]. Available: https://www.databricks.com/blog/what-is-agent-evaluation

[2] IBM, "What is AI Agent Evaluation?," IBM Think, Nov. 17, 2025. [Online]. Available: https://www.ibm.com/think/topics/ai-agent-evaluation

[3] Confident AI, "AI Agent Evaluation," DeepEval Docs, 2026. [Online]. Available: https://deepeval.com/guides/guides-ai-agent-evaluation

[4] Anthropic, "Demystifying evals for AI agents," Anthropic Engineering, Jan. 9, 2026. [Online]. Available: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

[5] OpenAI, "Evaluation best practices," OpenAI Developers, 2025. [Online]. Available: https://developers.openai.com/api/docs/guides/evaluation-best-practices

[6] H. Selbie, "A methodical approach to agent evaluation: Building a robust quality gate," Google Cloud Blog, Nov. 18, 2025. [Online]. Available: https://cloud.google.com/blog/topics/developers-practitioners/a-methodical-approach-to-agent-evaluation