Evaluation Mindset: From Demo to Evidence

Three Good Summaries

A claims team watches an assistant summarize three files. The summaries are fast, fluent, and accurate. Someone asks the assistant to revise a sentence; it does. Someone else tries a longer file; that works too. By the end of the demonstration, the room is discussing a pilot.

The three files were clean, short, and selected by the team that built the assistant. Production files are not. They include scans, contradictory notes, several claims in one document, and languages absent from the demonstration. A missing denial reason is rare, easy to overlook in polished prose, and costly when a reviewer accepts the draft without reopening the source.

The demonstration established a useful fact: this configuration can produce an impressive result. It did not establish how often the result survives the intended workload, whether the assistant improves on the current process, which failures concentrate in which cases, or what evidence would justify exposing claimants to the system.

An evaluation begins where the applause ends: with a decision that could still go either way.

Write the Decision Before the Score

The claims team replaces “evaluate the model” with a sentence that can govern action:

Decide whether configuration S may assist trained reviewers
for claim types C1 and C2 during a four-week pilot,
provided it reduces time to a verified summary,
improves severe omissions over the manual baseline,
and every high-risk segment meets its safety floor.

This is narrower than “Does the assistant work?” because useful evidence is always evidence for a bounded claim. The sentence names the system, users, workload, comparison, safeguards, and permitted release. It also leaves room for several honest results: launch the bounded pilot, remain in shadow mode, narrow the eligible files, gather more evidence, redesign, or stop.

Writing those outcomes in advance changes the evaluation. The team can no longer let a strong average authorize an unrelated use, such as automatic customer communication. It must say who owns the decision and who may accept residual risk. Thresholds chosen after the final scores arrive are especially vulnerable to wishful reinterpretation; set them before opening the release set whenever practical.

The same discipline serves other questions. A component test may ask whether the parser caused missing denial reasons. An ablation may ask whether retrieval adds enough value to justify its failure surface. A regression test may ask whether a prompt change revived an old error. An online pilot may ask whether reviewer behavior preserves an offline quality gain. Each question needs its own evidence and decision rule.

Freeze What Is Being Evaluated

The team initially writes “claims assistant v2” in the plan. That label is too loose. The result can change when the provider silently updates a model, when a prompt changes, when the parser handles scans differently, when the retrieval index is rebuilt, or when the interface places the source link farther from the draft.

The evaluated object therefore includes the full configuration: model and provider version; prompts and policies; parsers, retrieval corpus, index, tools, and orchestration; thresholds, fallbacks, and permissions; interface and reviewer workflow; deployment dependencies; and the versions of the dataset, evaluation code, rubric, and automated graders. A model score cannot stand in for this system score.

This boundary also separates component evidence from end-to-end evidence. Parser recall may reveal why text vanished. Summary review may measure whether required facts survived generation. Only a workflow test can show whether reviewers detect omissions, correct them, and finish faster. The layers answer different questions; a team needs enough of each to locate failure without mistaking a healthy component for a safe product.

Build a Sample That Can Resist the Claim

A random sample of recent files describes frequency, but it may contain almost none of the rare cases the release decision most depends on. The team creates two connected sets. The first represents the expected mix of claim type, language, scan quality, length, complexity, and outcome. The second deliberately over-samples historical severe omissions, ambiguous files, missing-data conditions, unanswerable requests, and adversarial cases.

The sets must not be blended into one comforting percentage. The production-like set estimates ordinary performance. The challenge set tests named boundaries and controls. Results are reported for both, with counts for relevant segments and with the sampling method, time window, sources, exclusions, deduplication, contamination risk, and known gaps. Representation includes consequence as well as frequency.

The sample supports only the population it can reasonably represent. Files from two claim types in one region do not establish performance for every product or language. A confident evaluation report makes that boundary easy to see.

Reference Answers Have a Production Process

For these summaries, “ground truth” is not a file of indisputable labels. Two qualified reviewers may select different supporting details while agreeing about every fact required for a decision. Forcing early consensus would conceal this useful disagreement.

The team gives two trained reviewers the same source evidence and explicit instructions. They independently identify required facts, denial reasons, unsupported statements, and severity. Their initial agreement is measured. A senior reviewer adjudicates consequential differences, while genuinely ambiguous cases remain marked as ambiguous rather than being converted into false certainty. Periodic quality checks test the reference process itself.

This procedure is part of the result. Reviewer qualifications, evidence access, instructions, independence, adjudication, escalation, and treatment of ambiguity must be recorded. Persistent disagreement may indicate a weak rubric, an unclear policy, or a task that should remain judgment-heavy. It is evidence about the system’s requirements, not noise to erase.

Human evaluation needs the same care elsewhere: anchored rating criteria, training, blinding when practical, agreement analysis, and a path for hard cases. Automated graders can increase coverage, but first require evidence of agreement with qualified human judgment, known sensitivities to prompt and position, and version stability. A grader is another measured component, not an oracle.

Make the Existing Work Earn Its Place as the Baseline

The comparison is not an imaginary world with no summaries. Reviewers already read files, draft summaries, correct errors, and route uncertain cases. The team measures that workflow: required-fact recall, severe omissions, completion time, downstream correction, and the workload displaced to other people.

Against it, the proposed system must improve an outcome rather than merely accelerate one step. Generation time is irrelevant if verification and correction take longer. A simpler template, deterministic extraction rule, search interface, smaller model, or previous release may also be a credible baseline. If a simpler intervention achieves similar value with less risk and cost, the evaluation should be capable of discovering that.

Ablations help locate contribution. Remove retrieval, swap the parser, disable a tool, or compare a simpler prompt while holding the rest of the configuration fixed. Their purpose is diagnosis: to learn which component creates value or failure. They do not become release evidence until the assembled system and human workflow are tested end to end.

Measure the Errors the Decision Cannot Absorb

The claims team chooses measures from consequences. Broad summary quality matters, but it can average away the omission that should stop release. The gate therefore includes required-fact recall, severe-omission and unsupported-statement rates, correction severity, time to a verified summary, abstention behavior, and floors for high-risk segments. Privacy-control violations are tested separately because an average quality score says nothing about them.

An error budget makes the permitted loss explicit: which errors may occur, how often, at what severity, and within which segment. It is not a pool that allows a team to trade several catastrophic failures for thousands of harmless successes. Some controls remain release vetoes; others can have bounded rates tied to monitoring and rollback.

Latency, availability, and cost belong when they constrain the workflow. So do appeals, rework, reliance, and incidents when they reveal effects after the output leaves the model. Chapter 26 develops metric and threshold selection in detail; the essential habit here is to connect every measure to a user outcome, system constraint, or error the release owner has agreed to bear.

Every rate needs its numerator and denominator. On a finite sample, it is an estimate, so the report includes an uncertainty interval suited to the sampling design. Repeated or clustered cases may not be independent, and a small set can easily observe no rare failures without demonstrating that the failure rate is zero. “Zero observed privacy violations” is meaningful only beside the number, variety, and strength of the tests.

Statistical uncertainty is not the whole uncertainty. The team also records label disagreement, evaluator bias, missing segments, dataset shift, benchmark contamination, and differences between the test and production workflow. A precise interval around an unrepresentative number does not make the claim representative.

Let the Errors Rewrite the Design

The first evaluation misses the gate. Average summary quality is high, but denial reasons disappear disproportionately in low-quality scans. Reviewers correct most of these omissions when the source panel is open; when the interface defaults to the polished draft, they miss more. The parser, generation step, and reviewer behavior are entangled in the aggregate score.

The team traces each important error through the system:

requirement -> source data -> parsing -> retrieval -> generation
-> interface -> reviewer action -> policy -> operation

It groups errors by origin and consequence, then examines representative cases, affected segments, severity, detectability, and plausible cause. The response is not “improve the prompt.” Low-confidence scans are now rejected or routed to manual work; denial reasons receive an explicit extraction check; the interface keeps source evidence beside the generated statement; and the eligible pilot population narrows.

This is what error analysis is for. A score supports a decision, but a failure cluster produces the next design. Sometimes the right response is a better component. Sometimes it is a control, a narrower scope, a stronger reference process, or an admission that more evidence is needed.

Protect the Test From the Development Loop

Once the team has inspected these cases, they are no longer unseen. Useful failures enter a regression suite, where they can prevent recurrence, but repeatedly tuning against them cannot demonstrate general improvement.

Development examples, validation data, and the final release set therefore have distinct jobs. Access to protected cases is controlled and recorded. Where feasible, the team retains a holdout and refreshes it with independently handled cases. It checks for training overlap and benchmark contamination, blinds candidate identity during subjective review when practical, and avoids copying production incidents into tests without preserving provenance and access controls.

Leakage is broader than a duplicated row. It occurs whenever knowledge of the answer shapes development or scoring in a way that will not exist for a new case. Regression tests prove that named failures stayed fixed. A fresh representative sample is still needed to show that the fix travels.

Move From Offline Evidence to a Bounded Pilot

The revised configuration passes the offline gate on 1,200 adjudicated, production-like files and 150 challenge cases. Severe omissions improve on the manual baseline; each predeclared high-risk segment meets its floor; verified summary time falls; and no privacy-control violation is observed in the adversarial set. Those results authorize only the four-week assistive pilot described in the decision—not automatic decisions, new claim types, or customer-facing text.

Online evaluation now asks what offline files could not: whether live inputs shift, whether reviewers actually reopen evidence, whether queue pressure changes reliance, and whether corrections, appeals, latency, or incidents reveal a new cost. The pilot samples reviewed outputs weekly, preserves a manual fallback, names stop conditions and an owner, and keeps the system in shadow mode where evidence remains weak.

Before any model, prompt, parser, corpus, policy, grader, or interface release, the regression suite runs again. Material changes also require fresh representative evidence. Evaluation is not a certificate attached permanently to a product name. It is a versioned argument that must remain true as the system and its world change.

The Evaluation Plan

The final plan is compact because its parts have already done work in the investigation. It records:

1. the use case, affected people, decision, and allowed release outcomes;
2. the evaluation owner, independent reviewers, and residual-risk authority;
3. the versioned model, data, software, controls, interface, and workflow under test;
4. dataset sources, sampling method, segments, exclusions, version, and known gaps;
5. the reference process, reviewer protocol, disagreement handling, and adjudication;
6. component, end-to-end, human, operational, and risk-specific measures;
7. the current workflow, simpler alternatives, and diagnostic ablations;
8. acceptance thresholds, segment floors, severe-error rules, and uncertainty reporting;
9. error-review categories, escalation, adversarial tests, and degraded-mode tests;
10. leakage controls, protected data, and the regression suite;
11. the launch rule, monitoring signals, stop conditions, and re-evaluation cadence.

To test whether the plan is real, return to the original demonstration transcript. For every successful exchange, write down the implied claim. Then name the sample that could challenge it, the baseline it must beat, the error that could veto it, and the decision that the evidence is allowed to support. If those cannot be named, the transcript is still a demonstration.

Evaluation does not remove uncertainty. It gives uncertainty a boundary, an owner, and a consequence.

Source Notes

• NIST AI Resource Center, current NIST resources for AI test, evaluation, verification, and validation and operationalizing the AI RMF; verified 2026-07-19.
• NIST AI Risk Management Framework, voluntary framework for managing risks to individuals, organizations, and society across AI design, development, use, and evaluation; verified 2026-07-19. NIST states that AI RMF 1.0 is being revised.
• NIST AI RMF Generative AI Profile, risk-specific actions for generative AI measurement and management; verified 2026-07-19.
• See AI Product Requirements for testable behavior contracts and Metrics for Predictive Models for metric and threshold design.