How to Run Human-in-the-Loop Evaluation for Large Language Model (LLM) Apps

When Automated Scores Still Miss Production Failures

Consider a hypothetical insurance claims assistant in a property and casualty (P&C) workflow. Offline faithfulness and relevance scores look green.

Production users still escalate. The answer is grounded in retrieved text. It is still wrong for policy intent, tone, or escalation rules.

Automated metrics and large language model as a judge (LLM-as-a-Judge) scoring scale well. They also share model blind spots.

Those include verbosity bias, weak domain judgment, and subtle policy misses. OpenAI evaluation guidance documents position and verbosity bias in LLM judges and recommends human feedback to calibrate automated scoring [1].

Survey work on LLM-as-a-Judge catalogs length, position, and self-enhancement biases. Gu et al. still treat human annotation as necessary in many scenarios [2].

Human-in-the-loop evaluation (HITL evaluation) means trained reviewers score model outputs against a fixed rubric. Prefer full traces for multi-step apps.

This process turns human judgment into structured ground truth. That is different from HITL runtime control.

In runtime control, a person must approve an agent action before it executes. This article covers evaluation only.

On specialized expert tasks, subject matter expert (SME) and LLM-judge agreement can be only moderate [3]. Szymanski et al. report that in one pairwise preference study, SMEs agreed with LLM judges 68% of the time in dietetics on overall preference [3].

Szymanski et al. also report agreement was 64% in mental health in that same study [3]. Bavaresco et al. find human-LLM agreement varies widely by dataset, property, and annotator expertise [4].

They recommend validating judges on task-specific human labels before deployment [4].

The operational goal is simple. Humans set the standard and surface failure modes.

Automated evaluators then apply that standard across production traffic (for example, 10K to 100K traces per day).

Design Human-in-the-Loop Evaluation Rubrics Before You Open a Review Queue

Noisy labels start with vague scorecards. Design the rubric before you open a queue.

Pick Dimensions That Match User and Policy Risk

Choose three to six dimensions. Common sets include task completion, factual groundedness, policy or safety, and tone or helpfulness.

For agents, add tool-use correctness.

1. Prefer binary pass or fail for safety and compliance.
2. Use graded 1-5 scales for quality trends.
3. Keep a short free-text field for root-cause notes.
4. Map each dimension to a decision: ship gate, monitor-only, or escalate-to-SME.

Multi-dimension rubrics appear in Anthropic and Hashemi et al. evaluation guidance [5], [6].

Write Anchor Examples and Run a Calibration Round

For each score level, write two or three anchor examples: good, borderline, and fail. Add a one-sentence rationale.

Have at least two reviewers score the same 20-30 traces independently. Then resolve disagreements in a short calibration session.

Track inter-rater agreement with a chance-corrected metric such as Cohen's kappa, not raw percent agreement alone. Gu et al. treat kappa as a standard agreement measure alongside percent agreement [2].

On imbalanced labels, report both. Version the rubric like code.

Store a rubric_id with every annotation. Hashemi et al. report that humans often disagree with one another on multidimensional rubrics [6].

Raw LLM score predictions often fail to match human judges without calibration [6]. In practice, we treat residual disagreement as a cue to refine anchors and re-run calibration.

Build a Trace-Backed Human-in-the-Loop Evaluation Workflow

Answer-only review collapses multi-step apps into a single bad answer label. Anthropic defines a transcript as the complete record of a trial.

That record includes outputs, tool calls, reasoning, intermediate results, and other interactions. Anthropic treats human graders as the quality reference used to calibrate model-based graders on those transcripts [5].

Step 1: Capture Full Execution Context

Log the prompt, system instructions, retrieved context, tool calls, intermediate spans, final output, latency, and token cost.

For agents, capture the agentic hierarchy. Reviewers should see the full decision tree of agent calls, tool invocations, and sub-agent outputs.

Span-level telemetry should roll up to aggregate insights across the hierarchy and the agent timeline. Without traces, review cannot separate bad retrieval from bad planning or bad generation.

{
  "trace_id": "tr_01HZX...",
  "prompt": "...",
  "system": "...",
  "retrieved_context": ["..."],
  "tool_calls": [{"name": "policy_lookup", "args": {}, "result": "..."}],
  "spans": [{"name": "planner"}, {"name": "generator"}],
  "final_output": "...",
  "latency_ms": 1840,
  "token_cost_usd": 0.021
}

Step 2: Seed and Version a Golden Set

Start with a modest, coverage-focused expert-labeled set. Many teams begin near 100-300 cases as a practical range, not a standard.

Treat the golden set as a release gate. New prompts and models must not regress on it.

Grow it from production failures as permanent regression cases. Version datasets with prompt and model versions.

OpenAI recommends mining logs for eval cases and treating evaluation as continuous [1]. Fiddler docs describe building a golden dataset from production traffic.

# Practical gate pattern after each prompt or model change
pytest tests/eval_golden.py --dataset golden_v17 --max-fail-rate 0.02

Step 3: Sample Production With Intent, Not Only Random Draws

Humans cannot score every request. Use a sustainable review budget.

Keep a small random baseline for calibration health. Stratify the rest on model signals.

Model signals include low-confidence scores, multi-judge disagreement, and novel topics. Business-risk signals include user thumbs-down, policy-sensitive intents, and high-cost or long traces.

Anthropic pairs automated evals with production monitoring and periodic human review. Methods include spot-check sampling and inter-annotator agreement checks [5].

# Practical sampling router (illustrative, not a standard rate)
def select_for_hitl(trace, auto_scores, rng):
    routes = []
    if rng.random() < 0.02:  # small random baseline; tune to review capacity
        routes.append("random_baseline")
    if auto_scores.get("disagreement"):
        routes.append("judge_disagreement")
    if trace.user_feedback == "thumbs_down":
        routes.append("user_negative")
    if trace.intent in POLICY_SENSITIVE:
        routes.append("policy_intent")
    if trace.token_cost > COST_P95 or trace.n_tool_calls > 8:
        routes.append("long_or_costly")
    return routes

Step 4: Route Review Queues to the Right Experts

Separate first-pass QA from domain SME and safety escalations. Show the full trace plus rubric form, and batch similar cases.

When building calibration labels, run randomized, blinded human review as OpenAI recommends [1].

Keep auto-scores out of the reviewer UI in those rounds. Capture structured scores plus a short rationale on disagreements.

{
  "trace_id": "tr_01HZX...",
  "rubric_id": "claims_v3",
  "reviewer_role": "domain_sme",
  "blinded_to_auto_scores": true,
  "scores": {
    "task_completion": 4,
    "groundedness": 5,
    "policy_safety": "pass",
    "tone": 3
  },
  "rationale": "Correct statute cite; tone too casual for denial letter."
}

Step 5: Close the Loop Into Automated Judges and Releases

Compare human labels with automated judges on the same traces.

Refine judge prompts, anchors, and thresholds. Scale automated scoring only after the judge consistently agrees with human annotations for that task [1].

Also require task-specific validation first [4]. Promote stable failures into continuous integration (CI) or pre-deploy regression suites.

Re-run the fixed golden set after every material change. Schedule recurring transcript audits and periodic SME calibration after model or prompt updates.

def judge_ready(human_labels, judge_labels, min_kappa):
    """min_kappa comes from your task-specific calibration study; re-check after changes."""
    kappa = cohens_kappa(human_labels, judge_labels)
    return kappa >= min_kappa

What to Watch For in HITL Programs

1. Reviewer fatigue and rubric drift: Extended review sessions and turnover between reviewers can erode label quality if you do not recalibrate [5].
2. Anchoring on the judge: Prefer randomized, blinded calibration rounds; do not treat visible auto-scores as ground truth [1].
3. Random-only sampling: Misses tail failures; mix stratified and event-driven routes.
4. Answer-only review: Hides tool errors, bad retrieval, and planning loops that full transcripts would expose [5].
5. One-and-done golden sets: Production traffic shifts; mine logs and treat evaluation as continuous so sets stay representative [1].
6. Judge drift after changes: Re-validate judges after model, prompt, or tool changes; agreement is task-dependent and can shift [4].
7. Permanent full-coverage HITL: Cost explodes; use humans to teach automated evaluators.

Turn Human Labels Into Continuous Production Evaluation

HITL is not a one-time labeling project. It is a continual loop across the lifecycle.

In pre-production, HITL builds golden sets and ships gates. In production, automated evaluators score high daily volume (for example, 10K to 100K traces) while humans handle stratified review.

Feed production annotations back into the next offline experiment. That keeps lab and live measures aligned.

Anthropic combines automated evals with production monitoring and periodic human calibration [5].

For higher-stakes generative AI, NIST's voluntary Generative AI Profile recommends combining human oversight with automated evaluation methods [7].

It also supports structured human feedback and TEVV documentation (test, evaluation, validation, and verification) [7]. Retain audit-ready trails: who scored what, on which rubric version, against which trace.

When automated evaluation runs on every production trace, external LLM judges add per-call API cost. That cost grows with evaluated traces and metrics.

Model Evaluation TCO with the Evaluation TCO calculator. Figures vary by model, deployment size, and traffic.

Evaluation TCO has three separate components.

1. Evaluation Trust Tax: the per-call API cost on the customer's LLM provider bill when external models score outputs.
2. Incident Risk Exposure: cost from undetected incidents on unsampled traces when teams down-sample to control eval costs.
3. Operational Overhead: engineering cost of building and maintaining custom evaluation infrastructure to avoid external evaluation costs.

This cost category excludes data exposure and latency. Those are separate concerns.

Fiddler Centor Models (formerly Fiddler Trust Models) are batteries-included, in-environment evaluators.

Out of the Box and Customizable paths run in-environment. They require no external API calls.

They incur no per-evaluation cost and avoid the Trust Tax. They target under 100ms response time.

They are fully framework, model, and cloud agnostic.

Bring Your Own Judge uses external providers. That path incurs external evaluation cost.

# Prefer in-environment evaluators; external judges only when needed
evaluator = "centor_customizable"  # or centor_ootb
if requires_external_reasoning:
    evaluator = "bring_your_own_judge"

These in-environment evaluators eliminate all three Evaluation TCO components on Out of the Box or Customizable paths. See the Centor glossary for cost details.

Shared rubric dimensions between HITL and live monitors keep offline and online scores comparable. The Fiddler AI Observability and Security Platform supports that pattern.

Pair HITL labels with LLM monitoring and Continuous Monitoring so score drift can enqueue another sample.

<code class='language-python'>if production_metric_delta("groundedness", window="7d") < -0.05:
    enqueue_hitl_sample(strategy="drift_groundedness", n=50)

In our production evaluation work, reusing the same three to six HITL dimensions in live monitors catches policy-intent misses faster than disconnected scorecards.

Human Judgment Is Still the Anchor for What Good Means

Return to the claims assistant. With a clear rubric, trace-backed review, and calibrated scoring, that team can catch policy-intent failures before users escalate.

HITL done well shrinks human load over time while increasing trust in automated gates.

Next step:

1. Pick one production slice and write a three-dimension rubric.
2. Label about 50 traces this week as a starting pattern.
3. Measure agreement with your current automated scorer.

As multi-agent systems grow, review must climb the agentic hierarchy. Span-level judgment should roll up to aggregate insights across the full hierarchy and the agent timeline.

Human judgment remains the anchor for what good means. See lifecycle evaluations.

References

[1] OpenAI, "Evaluation best practices," OpenAI Developer Documentation. [Online]. Available: https://developers.openai.com/api/docs/guides/evaluation-best-practices

[2] J. Gu et al., "A Survey on LLM-as-a-Judge," arXiv:2411.15594, 2024. [Online]. Available: https://arxiv.org/abs/2411.15594

[3] A. Szymanski et al., "Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks," arXiv:2410.20266, Oct. 2024. [Online]. Available: https://arxiv.org/abs/2410.20266

[4] A. Bavaresco et al., "LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks," arXiv:2406.18403, 2025. [Online]. Available: https://arxiv.org/abs/2406.18403

[5] Anthropic, "Demystifying evals for AI agents," Anthropic Engineering, Jan. 2026. [Online]. Available: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

[6] H. Hashemi et al., "LLM-Rubric," ACL 2024. [Online]. Available: https://aclanthology.org/2024.acl-long.745/

[7] NIST, "AI RMF Generative AI Profile," NIST AI 600-1, July 2024. [Online]. Available: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=958388