Evaluating Generative AI

The Assistant Passed and the Customer Still Lost

A benefits assistant earns strong average ratings for clarity and tone. During a pilot, a customer asks whether a dependent remains eligible after a recent move. The assistant gives a polished answer in the requested format and cites a real policy page. It also invents an eligibility condition that the page does not contain.

One reviewer calls the answer “mostly helpful.” Another fails it because a customer could act on the unsupported condition. Both judgments fit the same output; they answer different questions.

Generative AI evaluation needs a portfolio of evidence. Define the task, keep unlike qualities separate, probe consequential failures directly, and make every score answer to a release decision and a real user outcome.

Turn “Helpful” into an Evaluated Task

Before scoring the disputed answer, the team must say what the assistant is for. It drafts benefits guidance from approved policy sources so that a caseworker can verify, correct, and send it. It does not decide eligibility. When the available evidence is incomplete or conflicting, it should identify the missing fact and route the case to a person.

That statement supplies observable success conditions. A draft must preserve the relevant facts, support material claims with applicable evidence, obey policy and format constraints, communicate uncertainty, and leave the caseworker able to complete the job. “Good answer” supplies none of those boundaries.

Version the object being evaluated as a system:

model + system instructions + task prompt + tools
+ retrieval corpus and index + policies + post-processing + interface

A new model is only one possible change. A parser can detach a qualification from a rule; a prompt can alter refusal behavior; a policy update can make yesterday’s reference answer stale; an interface can hide the citation a caseworker needs. Record the input, allowed context, expected and prohibited behavior, evaluator guidance, severity, and full system version for each case.

Build the Set Around Work and Consequence

The first task set should resemble the queue the assistant will actually encounter. Include routine questions, common exceptions, ambiguous requests, missing evidence, conflicting policies, and cases where the assistant must escalate. Then add cases chosen for consequence rather than frequency: privacy boundaries, fabricated conditions, misleading citations, prompt injection, and attempts to obtain guidance outside the assistant’s authority.

Language and accessibility belong in the set, not in a promise attached after evaluation. A translated question may expose different retrieval coverage or refusal behavior. A response that is technically correct but unreadable with the intended assistive technology has not completed the same task. Test important languages, reading needs, document formats, and issue types as declared segments, with qualified reviewers where meaning or cultural context requires them.

Keep the uses of the data distinct. Developers need a visible set for iteration. Every repaired defect belongs in a regression set. A protected release set must remain unseen until the gate is fixed. Adversarial cases exercise misuse and controls. Appropriately handled production samples reveal changes in real work that constructed cases missed.

Semantic near-duplicates do not become independent evidence because their wording differs. Record provenance, deduplicate paraphrases, and refresh protected cases once developers have learned their shape. The useful question is not how many prompts exist, but which parts of the task and risk distribution they permit the team to judge.

Make Each Dimension Keep Its Own Failure

Return to the invented eligibility condition. The answer may score well for style and instruction following while failing correctness and groundedness. If the evaluator averages those judgments, polish compensates numerically for a material falsehood.

Separate the qualities the task can fail on:

• correctness: are the material claims and proposed actions true for this case?
• groundedness and citation accuracy: does allowed evidence support each material claim, and does each citation point to that support rather than merely to a real page?
• completeness: are the required facts, qualifications, warnings, and next steps present?
• instruction following: did the response respect format, policy, role, and authority constraints?
• safety and privacy: did it avoid harmful guidance and improper disclosure?
• calibration: did certainty, abstention, and escalation match the available evidence and consequence?
• usefulness: can the intended person complete the task correctly and efficiently?
• style and accessibility: can the intended audience understand and use the response?

The chapter’s opening answer needs no debate about its average. An unsupported condition in consequential benefits guidance is a critical defect. Define such gates before seeing results. Then report failures by criterion, severity, segment, and count alongside distributions or averages. The Generative AI Evaluation Rubric provides anchored scoring fields; the release team must still adapt its examples and critical defects to this use case.

Give Deterministic Requirements to Deterministic Checks

Some requirements do not need an opinion. Code can verify that output matches a schema, required fields exist, forbidden fields do not, identifiers resolve, numeric results agree with a trusted calculation, citations name available sources, and latency and cost stay within stated bounds. Tool calls and permissions can be checked against an explicit contract.

These tests are cheap, repeatable, and sharp. They are not semantic judgment. A citation identifier can resolve while the cited passage says nothing about the claim beside it. A privacy pattern can catch a known identifier format without proving that no sensitive fact was inferred. Use deterministic checks to remove mechanical uncertainty, then send the remaining meaning to an appropriate judge.

Word-overlap measures have the same limited role. They can help with narrow transformations whose wording should remain stable. For open-ended guidance, a correct answer may use different language, while a high-overlap answer may preserve every noun and reverse one decisive condition.

Human Review Is a Measurement Process

The benefits team recruits policy specialists because correctness depends on domain meaning, and caseworkers because usefulness depends on the receiving workflow. Reviewers get criterion-specific instructions, anchored examples, and practice cases. Responses are blinded and randomized; reviewers score independently before adjudication.

Pairwise comparison can reveal which of two drafts better satisfies a criterion, especially when an absolute scale feels vague. It cannot show that either draft is safe enough to release. Absolute ratings can enforce a minimum standard, but only when adjacent scores describe observable differences. The team may use pairwise judgments during iteration and criterion floors for release.

Agreement is evidence about the measurement, not a ceremonial statistic. Examine it by criterion. If reviewers disagree about groundedness, the evidence may be ambiguous, the rubric may bundle several claims, or one reviewer may lack the needed policy expertise. Preserve the initial scores and reasons during adjudication. Erasing disagreement also erases a clue about the system or the task.

A Model Grader Must Earn a Narrow Job

The team wants a model grader because qualified policy review is scarce. It first draws a reference sample judged by people, then compares grader decisions with those judgments by criterion and severity. It reverses response order, varies formatting and verbosity, and includes outputs that attempt to persuade or instruct the grader. High-impact cases and uncertain grader results remain with people.

This calibration may show that the grader is useful for triaging style deviations and familiar omission patterns but unreliable on policy applicability. That is still valuable. The mistake would be promoting it from a demonstrated instrument to universal ground truth because one call can return every score.

Version the grader model, prompt, rubric, and calibration results. Revalidate after changes and against fresh human judgments. A grader can prefer verbosity, share blind spots with the system under test, miss domain nuance, respond to answer position, and change when its provider changes the underlying model. Its evidence must therefore name both its useful range and its limits.

Test the Boundary Between Answering and Not Answering

The assistant’s hardest success may be declining to invent. Give it paired cases: one with enough evidence for a benefits answer and one missing the decisive residency fact. Add disallowed requests, indirect paraphrases of those requests, retrieval failure, tool timeout, and a malformed required format. Observe whether it answers, abstains, refuses, redirects, escalates, and recovers appropriately.

False refusals are failures too. A system that blocks harmless, answerable requests can look safe while making the service unusable. Measure refusal precision and coverage by request class, and inspect what happens after the refusal: whether the explanation leaks sensitive policy detail, whether redirection is actionable, and whether an authorized person can recover the task.

Perturb the Case Without Moving the Truth

Rewrite the dependent-and-move question with spelling errors, reordered facts, extra irrelevant text, concise and verbose phrasing, and important language variants. Those meaning-preserving changes should not alter the eligibility evidence or escalation decision. Then change the move date or jurisdiction so the correct result should change. The pair tests stability and sensitivity; a system that always repeats the same answer can pass the first and fail the second.

Generative variance requires repeated trials when a rare failure could change the release decision. Freeze generation settings for comparisons and report pass probability or failure frequency rather than selecting the best sample. Robustness evidence should also cover degraded retrieval and tools, long contexts, conflicting instructions, and attempts to manipulate either the assistant or its evaluator.

Every production change returns to the same suites. Model, prompt, retrieval, policy, parser, post-processing, and interface changes can each revive an old failure. Regression cases protect known repairs; fresh protected cases test whether the team has learned only the test set.

Write a Gate That Can Say More Than Yes

The support-drafting copilot is not ready because it achieved a single win rate. It may enter a limited queue when required-fact recall and unsupported-claim rates beat the current template baseline; every severe policy and privacy case passes adjudication; citations support their associated claims; refusal and escalation meet per-class thresholds; and performance floors hold across important languages and issue types.

System evidence belongs in the same decision. Qualified caseworkers must complete work faster without more severe corrections. Latency and cost must fit the operating budget. Rollback, feedback capture, and production sampling must already have owners. The release record names the task-set and rubric versions, reviewer agreement, automated checks, grader calibration, error clusters, accepted limitations, approver, monitoring triggers, and containment action.

The resulting decision need not be “launch everywhere” or “stop.” It can limit the assistant to drafting for specified queues, require human review for named case classes, or send the team back to retrieval, policy, or interface design. Production monitoring then tests the claim that offline success survives contact with real work: completion time, correction severity, escalation, complaint and appeal signals, cost, latency, and the distribution of cases all matter.

Now score three candidate answers to the original dependent-and-move question: a fluent unsupported answer, a terse refusal despite sufficient evidence, and a qualified draft that names the missing fact and escalation route. Define the critical defect before assigning any number. Then introduce one paraphrase that should preserve the judgment and one fact change that should reverse it. If the rubric cannot explain those outcomes, it is not yet a release instrument.

Generative evaluation becomes credible when each form of evidence is allowed to see what it can see—and no score is permitted to hide the failure the system was built to prevent.

Source Notes

• NIST AI RMF Generative AI Profile, voluntary cross-sector guidance on generative-AI measurement, human review, content provenance, testing, and risk management; verified 2026-07-19.
• NIST AI Resource Center, official resources for testing, evaluation, verification, and validation of AI systems; verified 2026-07-19.
• See Guardrails, Policies, and Control Layers for layered controls, Evaluation Mindset for sample and threshold design, and Evaluating RAG and Knowledge Systems for tracing retrieval and evidence failures to their responsible layer.