Evaluating LLM Outputs
Build practical evaluations for correctness, relevance, safety, and consistency.
Define success before grading
An evaluation needs a clear task, input distribution, expected properties, and decision threshold. “Sounds good” is not a stable requirement. Specify whether the output must be factually correct, grounded in sources, complete, concise, polite, schema-valid, or safe to refuse. Different tasks require different rubrics, and one score should not hide a failure in a critical dimension.
Build representative cases
Collect ordinary requests, edge cases, ambiguous requests, adversarial inputs, and examples that caused real user corrections. Remove or replace private data while preserving the difficulty. Separate development, validation, and held-out sets. Include multilingual and long-context examples when the product supports them. A small but representative set is more useful than a large collection of nearly identical prompts.
def passes(result):
return (
result["schema_valid"]
and result["grounded"]
and result["safety_review"] == "pass"
)
print(passes({"schema_valid": True, "grounded": True, "safety_review": "pass"}))Deterministic checks
Use code for properties that code can judge: valid JSON, required fields, allowed values, citation identifiers, numeric ranges, prohibited secrets, and latency limits. Deterministic checks are fast and repeatable, so they should run on every candidate release. They cannot assess every nuance, but they prevent obvious regressions from being hidden by a favorable subjective score.
Reference and rubric grading
Reference answers help when there is a known factual target, while rubrics work better for open-ended quality. State what earns full, partial, and failing credit with concrete examples. Model-assisted graders can scale review but need calibration against humans and should not be trusted for high-impact decisions without oversight. Preserve grader prompts and versions as evaluation dependencies.
Safety and abstention
A good answer is not always an answer. Test whether the system refuses unsafe requests, asks for missing information, acknowledges uncertainty, and avoids inventing evidence. Include indirect instructions in retrieved documents and attempts to extract secrets. Measure false refusals as well as unsafe completions, because an over-aggressive system can block legitimate work and encourage users to bypass controls.
Regression analysis
When a candidate loses, inspect the examples rather than only reading the aggregate score. Group failures by retrieval, reasoning, formatting, language, safety, and product logic. A model change may improve average quality while damaging one important segment. Keep a failure corpus and turn confirmed regressions into permanent tests so the same lesson is not forgotten.
Using evaluations in release decisions
Set thresholds for must-pass checks and acceptable movement for softer metrics. Compare quality with cost, latency, and operational reliability under production-like settings. Document known weaknesses and the mitigation that covers them. Evaluation is a decision instrument: it should tell the team whether to ship, revise, limit exposure, or add human review—not merely produce a leaderboard.
Worked example: grading grounded answers
For a policy assistant, an answer can be fluent yet unsupported. Evaluate at least four dimensions separately: schema validity, factual support in the supplied sources, safety policy compliance, and usefulness to the user. A human rubric should define what counts as sufficient evidence and how to score partial answers. Keep evaluator instructions independent from the answer generator to reduce correlated mistakes.
Code walkthrough
The `passes` function combines three gates and returns a boolean, which is easy to integrate but too coarse for diagnosis. In a real evaluator, return a result object containing each gate, evidence IDs, rubric scores, and an overall decision. Validate missing keys before indexing them and distinguish a failed evaluation from an unavailable evaluator. Do not automatically publish an answer merely because the JSON wrapper is valid.
Trade-offs to measure
Human review is richer but slower and expensive; model-based grading scales better but can favor fluent answers or share the generator’s blind spots. Exact-match metrics are reproducible but narrow, while rubric scores capture nuance and introduce disagreement. Use automated checks for hard constraints, sampled human review for meaning and harm, and calibration cases to detect evaluator drift.
Practical exercise
Prepare ten answers with known supported, unsupported, incomplete, and unsafe cases. Implement separate checks for schema, citations, groundedness, and safety. Ask two reviewers to score usefulness and support, calculate agreement, and compare their judgments with an automated grader. Create a release rule that blocks publication on one critical safety failure even when the aggregate score is high.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.