Measuring LLM Quality, Safety and Performance: Which Metrics to Track

When retrieval returns off-topic chunks and the model answers from memory, you need to identify the failing span. A golden dataset that passed in continuous integration (CI) does not show which span failed in live traffic. With LLM metrics, a pre-release score can drive production alerts, then support investigation of retrieval or tool-call failures and checks on whether a prompt change caused them.

What LLM Metrics Measure

LLM metrics are numeric scores attached to a prompt, a response or a whole session, helping teams check whether an LLM application did what its builders intended, safely and within budget. Most of the ones that matter in production are reference-free, because live traffic has no gold answer to compare against. Most are also computed per event, so an alert can fire on a single bad trace rather than a weekly average.

Quality and safety metrics assess application behavior:

• Quality: Faithfulness, hallucination, answer relevance, context relevance, coherence, correctness, and fluency measure whether the output is right and grounded. They also cover readability.
• Safety: Toxicity, personally identifiable information (PII) and protected health information (PHI) leakage, jailbreak, and prompt injection measure whether an input or output violates policy.

Operational metrics cover latency, time-to-first-token, throughput, token usage, and cost per session. They measure what the application costs to run and how it feels to the user.

Quality scores identify which component to debug, safety scores determine what to block, and operational scores track spend. The methods that produce these scores (golden datasets, A/B comparisons, red teaming, human review) are covered in the evaluation methods overview.

How LLM Metrics Are Scored

Three scoring mechanisms produce LLM metrics, and the first split is whether a metric needs a reference answer.

• Reference-based: BLEU (Bilingual Evaluation Understudy), a machine-translation metric, and ROUGE (Recall-Oriented Understudy for Gisting Evaluation), a summarization metric, score n-gram overlap between a response and a gold answer. On expert-rated CNN/DailyMail summaries, ROUGE-L’s Kendall’s tau with human judgment was 0.0735 for coherence and 0.1471 for consistency [1]. BERTScore and embedding similarity sit in the middle ground: they compare contextual embeddings, so paraphrases score near the reference. Perplexity is comparable only across models that share a tokenizer, and some closed APIs return no token log probabilities.
• LLM-as-a-judge: A second model scores the response against a rubric. GPT-4 judgments matched expert human judgments over 80% of the time on MT-Bench, matching human-human agreement, but judges show position bias [2].
• Purpose-built evaluators: These are fine-tuned small language models built for specific evaluation tasks and hosted in-environment. Because they run there instead of calling an external LLM, they avoid the per-call charge that external judge calls add to the team’s LLM bill, the Evaluation Trust Tax (results vary by model, deployment, and traffic). Prompts, tool returns, and outputs also stay inside the team’s environment.

Quality Metrics

Quality metrics check grounding and relevance, then assess whether the response was well-formed:

• Faithfulness: Faithfulness is the share of claims in a response that the retrieved context supports, which RAGAS scores as |V|/|S|, with an LLM extracting and verifying each statement [3]. Teams gate on the golden-set cutoff and alert below it.
• Hallucination: Hallucination is the inverse of faithfulness when context exists, plus claims that contradict world knowledge when it does not. In retrieval-augmented generation (RAG), a low faithfulness score signals hallucination, so the faithfulness alert triggers the investigation.
• Answer relevance: Answer relevance checks whether the response addresses the question, with the score averaging the cosine similarity between the original question and questions generated from the response [3]. Evasive or padded answers score low even when every sentence is true.
• Context relevance: Context relevance is relevant sentences over total sentences in the retrieved context, selected by a judge or purpose-built evaluator [3]. A retrieval-span alert can fire on a sustained drop.
• Coherence and fluency: Coherence and fluency are judge-scored against a rubric for one consistent line of argument in grammatical language, with the pass share tracked per release.
• Correctness: Correctness is scored by comparing the answer to a gold answer through exact match or token overlap on golden datasets, or through a judge on open-ended answers. In production, faithfulness serves as a proxy.

Retrieval and Generation

Splitting retrieval metrics from generation metrics in RAG identifies the component behind a bad score. Context precision checks whether relevant chunks ranked above irrelevant ones (rank-weighted, typically against a reference or expected output). Context recall checks whether the chunks cover the claims in the reference answer. Faithfulness then isolates the generator.

Agent Tasks and Tools

Tool selection checks that the agent called the right function, and argument correctness checks the parameters against the schema and the user’s intent. Task completion is scored by comparing the final environment state with the expected outcome.

Anthropic describes pass^k as the chance that all k trials of the same task succeed, which is the reliability production users experience. A 75% per-trial pass rate gives roughly 42% odds that three of three trials pass [4].

An agent that passes once in testing can still fail often in production, so release gates use pass^k. Scoring runs across the full task completion process, with each failure attributed at the span level.

Safety Metrics

Safety metrics score inputs and outputs against policy with a classifier or a judge rather than a reference:

• Toxicity: A classifier returns a 0–1 probability that a reader would perceive the text as toxic. The score measures classifier confidence, so 0.9 means the model is more certain the text is toxic; severity is a separate question. For moderation, Perspective API’s documentation suggests holding content scored at 0.9 or above for review [5].
• PII/PHI leakage: A detector scans prompts, tool returns, file reads, and responses for entities such as names, Social Security numbers (SSNs), credit card numbers, and medical conditions. It is scored as entity count or maximum confidence per event. The Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor method lists 18 identifier categories, some of which, like full-face photos, a text detector cannot see [6].
• Jailbreak: A classifier scores the probability that an attacker is using a prompt to override the system prompt or safety training through persona play or direct harmful instructions. Pre-release red teaming measures attack success rate, the share of adversarial prompts that elicit prohibited behavior.
• Prompt injection: Injection arrives directly in the user turn or indirectly through retrieved documents, tool results, or files the agent reads. The Open Worldwide Application Security Project (OWASP) ranks prompt injection first in its 2026 Top 10 for LLM Applications, as it did in the 2025 edition [7]. A classifier scores every input the model sees for injected instructions.

In production, these four scores run twice. An asynchronous monitoring pass raises alerts. A second, inline pass runs through inline guardrails that intercept requests and responses to allow, block, or redact. It checks the request before the model is invoked and the response before it reaches the user or a downstream system.

Fiddler Guardrails run those inline checks with under 80ms response time, using purpose-built scoring models to score faithfulness, safety and PII/PHI, with thresholds typically more sensitive than monitoring thresholds.

Injection scores are most informative alongside tool-call spans, since the complex agent behaviors that go wrong can begin with an instruction read from a tool result.

Operational Metrics

Operational metrics come from the trace, not a scorer, and they are the LLM performance numbers users feel first:

• Latency: Latency measures end-to-end time per request and per session. Time-to-first-token (TTFT) covers queueing, prefill, and network time until the first streamed token. The median (p50) and 99th percentile (p99) are tracked separately, and reasoning models belong on their own chart because their thinking time counts toward the first answer token.
• Throughput: Throughput measures output tokens per second from trace timestamps, for one user and across concurrent requests; an alert fires when it drops below the golden-run baseline.
• Token usage: Token usage counts input, cached input, output, and reasoning tokens per LLM span, summed per trace and per session. Providers generally bill thinking tokens at the output rate. Multi-turn conversations also resend prior turns as input, so session cost grows faster than turn count.
• Cost per session: Cost per session sums (input tokens ÷ 1M × input price) + (output tokens ÷ 1M × output price) across every LLM span, using cached and batch rates where they apply, and attributes it to the agent that spent it.

A runaway loop can show up here first. Token usage climbs on one agent span while quality scores stay flat, and a per-request cost alert can miss it until the invoice arrives. The golden-run 95th percentile (p95) latency and median cost per session serve as the baseline, with an alert when production exceeds it by a chosen margin.

How to Choose Metrics for Your Use Case

Metric choice starts with what the application can get wrong, then adds the operational set. The table pairs four application types with metrics to gate on before release and alert on in production:

Judge and classifier scores need calibration against human labels before any threshold is trusted. Two annotators label a sample of production traffic. Cohen’s kappa compares their labels with each other and with the evaluator’s labels.

Kappa, κ = (p_o − p_e) / (1 − p_e), corrects raw agreement for chance. Landis and Koch place 0.61–0.80 at substantial agreement and 0.81–1.00 at almost perfect [8].

Raw agreement can look high while kappa stays low when one label dominates. Recalibration is needed whenever the judge model or evaluator version changes, since scores are not comparable across versions.

Tracking LLM Performance: From Pre-Release Threshold to Production Alert

Using the same metric for the release gate and the production alert lets LLM monitoring double as LLM accuracy tracking. For faithfulness, the path runs in four steps:

1. In development, run an LLM-as-a-judge evaluator or the in-environment faithfulness evaluator over a golden dataset of questions and their contexts paired with answers. Use Tests & Experiments for custom evaluators with domain-specific criteria and to compare prompts and models under different configurations. The faithfulness threshold defaults to 0.5 and is data-dependent.
2. Before release, set a quality gate on the RAG triad (faithfulness with context and answer relevance) at the threshold derived from the golden-set distribution.
3. In production, that threshold can become an evaluator rule with a warning level and a critical level, routed to PagerDuty or your APM tool. Judge-based RAG triad metrics may route through an external LLM.
4. On an alert, drill from the application score to the session, agent, trace, and span, and re-score the offending cases to confirm what changed.

Fiddler Centor Models are batteries-included and score faithfulness, safety, and PII/PHI in-environment on 100% of prompts and responses, so the threshold set before release can become the alert that runs in production. Their costs are a step function, compared to external LLM judge costs that grow linearly with every evaluation.

Because every request is scored rather than a sampled subset, continuous monitoring can run at application scale. Bring Your Own Judge routes scoring to an external LLM provider and is the only Centor Models option with an external per-call charge.

The Fiddler platform records each enforcement decision with the policy that applied, giving governance teams an evidence trail.

Request a demo to review how this evaluation-to-alert path would fit an existing agent architecture and deployment environment.

References

[1] A. R. Fabbri et al., “SummEval: Re-evaluating Summarization Evaluation,” Transactions of the Association for Computational Linguistics, vol. 9, 2021. Accessed: Oct. 19, 2026. [Online]. Available: https://aclanthology.org/2021.tacl-1.24/

[2] L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS Datasets and Benchmarks Track, arXiv:2306.05685, 2023. Accessed: Oct. 19, 2026. [Online]. Available: https://arxiv.org/html/2306.05685

[3] S. Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” Proc. EACL 2024 System Demonstrations, 2024. Accessed: Oct. 19, 2026. [Online]. Available: https://aclanthology.org/2024.eacl-demo.16.pdf

[4] Anthropic, “Demystifying evals for AI agents,” Anthropic Engineering, Jan. 9, 2026. Accessed: Oct. 19, 2026. [Online]. Available: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

[5] Jigsaw, “About the API: Score,” Perspective API Developer Documentation. Accessed: Oct. 19, 2026. [Online]. Available: https://developers.perspectiveapi.com/s/about-the-api-score

[6] U.S. Department of Health and Human Services, Office for Civil Rights, “Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule.” Accessed: Oct. 19, 2026. [Online]. Available: https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html

[7] OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2026,” Version 2026 (publication date to be set). Accessed: Oct. 19, 2026. [Online]. Available: https://genai.owasp.org/download/56857/

[8] J. R. Landis and G. G. Koch, “The Measurement of Observer Agreement for Categorical Data,” Biometrics, vol. 33, no. 1, 1977. Accessed: Oct. 19, 2026. [Online]. Available: https://jjcurtin.github.io/book_iaml/pdfs/landis_1977_kappa.pdf