Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 21

Retrieval-Augmented Generation Systems

Design retrieval-augmented generation as an evidence system with separate retrieval, generation, access, freshness, and abstention controls.

The Answer Was Polished and Wrong

An operations team launches an assistant over equipment manuals. During a demo, it answers every prepared question and cites plausible-looking sources. A technician later asks how to reset a particular controller revision. The assistant retrieves a generic reset procedure from an older manual, blends it with a newer model’s warning, and produces one confident sequence. Both citations exist. Neither supports the sequence for the installed controller.

The answer looks like a generation failure. It is really the last visible event in a longer failure. The corpus did not represent equipment revisions cleanly. Parsing separated a warning from the procedure it governed. Retrieval favored a generic manual, and context assembly placed incompatible passages side by side without their scope. Evaluation rewarded the finished answer’s style without asking whether the right evidence had arrived.

Retrieval-augmented generation (RAG) is an evidence-delivery system before it is a writing system. Treat retrieval and generation as separate components with separate acceptance gates.

A RAG pipeline separates retrieval from generation: a query passes access control, retrieval, reranking, and context assembly before generation and groundedness verification; missing or unauthorized evidence leads to abstention.
Evaluate the evidence path before the prose path. Retrieval recall at k tests whether usable evidence arrived; groundedness tests whether the generated output stayed within it.

Begin with the Claim, Not the Index

Do not begin with an embedding model. Begin with the claim the system is allowed to make.

For the maintenance assistant, “How do I reset this controller?” is not yet an answerable question. The evidence contract requires a controller family, hardware revision, firmware range, site configuration, and effective manual version. A procedure is usable only when those fields agree. If the installed revision is unknown, the assistant must ask for it. If two current sources conflict, it must show the conflict and route the technician to an owner. If no applicable source survives, it must decline.

That contract determines which sources may enter the corpus, who may see them, how applicability travels with them, how quickly changes and deletions must propagate, and what counts as sufficient evidence. It also defines the boundary of the answer: the assistant may explain an approved procedure, but it may not construct a new one from fragments belonging to different machines.

Corpus membership is therefore a product and governance decision. Indexing every file that happens to be available gives the system more text, not necessarily more knowledge. Archived drafts, superseded manuals, regional variants, and field notes can be valuable, but only if their status is explicit and the retrieval policy knows when they are admissible.

Follow One Passage into the Answer

Ingest without erasing structure

The correct reset procedure begins as a boxed sequence beneath the heading “Revision C, firmware 4.2–4.6.” A warning immediately above it says to isolate the backup circuit before holding the reset control. A parser that extracts the steps but drops the heading and warning has produced clean text and damaged evidence.

Parsing must preserve the structures that determine meaning: headings, table headers, lists, captions, document identity, page or section anchors, and relationships between warnings and procedures. The test is not whether the parser returned text. It is whether the retrieved unit still supports the same claim a careful reader could make from the source.

Store provenance beside every retrievable unit:

source_id, source_version, effective_at, ingested_at
section_path, page_anchor, owner, classification
access_policy, retention_rule, deletion_state, parser_version

Test representative PDFs, scans, spreadsheets, nested pages, and malformed documents. Include the awkward pages: repeated headers, multi-column layouts, scanned annotations, continued tables, and warnings that govern the next page. These are where a nominally successful ingestion pipeline changes meaning.

Chunk by meaning and answer shape

Fixed token windows are a baseline, not a universal design. In this manual they can put the warning in one chunk, the numbered steps in another, and the revision heading in a third. A larger window may reunite them, but it can also pull in the Revision B procedure from the previous section.

Prefer boundaries that keep a rule with its conditions, a table with its headers, a procedure with its warnings, and a definition with its scope. Attach parent-document and section metadata even when a chunk is otherwise self-contained. Use overlap only when it preserves necessary continuity; excessive overlap crowds the context with near-duplicates and can make repeated evidence look like independent support.

Evaluate chunking with real questions. For the reset question, inspect whether one retrievable unit contains enough evidence to act, whether the split removed a qualifier, whether metadata distinguishes similar revisions, and whether the unit leads a technician back to the exact source location. Average chunk size cannot answer any of those questions.

Retrieve broadly, then rank precisely

The technician calls the operation a “hard restart”; the manual calls it a “protected controller reset.” Semantic retrieval can bridge that paraphrase. The controller code AX-410C and firmware 4.4.1 demand exact matching. Lexical retrieval protects identifiers and rare terms; semantic retrieval broadens recall; hybrid search combines their candidate sets. A reranker can then judge the query-passage relationship more precisely than the first-stage index.

Similarity does not confer authority. An obsolete manual may be the closest linguistic match precisely because it contains the user’s wording. Apply identity, permission, document status, version, and scope constraints before content becomes model context. Security trimming after generation is too late, and filtering by applicability only after ranking may discard the few candidates that were actually eligible.

Assemble context as a controlled artifact

Suppose retrieval returns the Revision C procedure first, its warning fourth, and a highly similar Revision B procedure second. Context assembly now makes a substantive decision. It can restore the warning’s surrounding section, exclude the inapplicable procedure, preserve the source boundaries, and state that all three passages are evidence rather than instructions. Or it can concatenate them by score and invite the generator to reconcile a contradiction that metadata had already resolved.

Treat the assembled context as a controlled, inspectable artifact. Decide which passages survive, how much surrounding text returns with them, how conflicts appear, and how much room remains for the response. Cap redundant passages. Mark retrieved text as untrusted data so a sentence inside a manual cannot masquerade as a system instruction. For evaluation and incident review, record passage identifiers, versions, ranking scores, filters, and assembly rules; avoid logging full documents when those records would create unnecessary exposure.

Test Where the Evidence Path Can Break

Retrieval measures

Begin with a judged set of questions and the passages that count as relevant. Include exact lookups, paraphrases, synthesis, ambiguous revisions, obsolete sources, unauthorized documents, conflicting sources, and questions the corpus cannot answer.

Retrieval recall at k asks how much of the judged relevant evidence appears among the top k results. For one question:

recall@k = relevant passages in top k / all judged relevant passages

Aggregate the per-question values in a stated way, and pair recall with a success rate for questions that need at least one usable passage. The denominator matters: a question requiring three independent policy clauses differs from one requiring a single lookup. Unanswerable questions belong in the suite, but they test empty-result and abstention behavior rather than retrieval recall. A team should also choose k deliberately: recall at 50 can hide a context window that admits only five passages.

Also measure:

  • ranking quality and the position of the first usable passage;
  • corpus and question-class coverage;
  • permission-filter correctness;
  • freshness and deletion propagation time;
  • conflicting-source retrieval;
  • empty-result behavior;
  • retrieval latency and cost.

For the controller question, the retrieval gate is simple to state: the current Revision C procedure and its warning must survive into the available context, while Revision B and unauthorized field notes must not. If that gate fails, a beautifully written response is irrelevant.

Generation measures

Only after relevant evidence is present should generation be judged. Test:

  • citation precision: cited passages actually support the associated claims;
  • groundedness or faithfulness: material claims stay within supplied evidence;
  • completeness: the response covers the required evidence without inventing gaps;
  • abstention quality: the system declines or qualifies when evidence is insufficient;
  • instruction following, clarity, and task utility;
  • total latency and cost per completed answer.

Citation precision is not the fraction of answers that contain citations. It is the fraction of cited claims whose cited passages actually support them. A citation may be real, current, and still fail to entail the sentence beside it. Groundedness asks the complementary question: do the response’s material claims stay within the supplied evidence? “Faithfulness” is often used for the same family of checks; define the term and grading unit your team uses rather than reporting two opaque scores as if their names guaranteed different measurements.

Automated graders can accelerate regression testing, but calibrate them against human judgments and inspect disagreements. A model-based score is another measurement component, not ground truth. The severe cases—unsafe procedures, access violations, invented exceptions—deserve explicit review even when their average contribution is small.

End-to-end measures

Finally test whether the technician identifies the equipment correctly, can inspect the supporting warning and procedure, and either completes the task safely or reaches the right escalation path. Measure answer latency and cost here as system properties, alongside task success and severe-error rate. Component success does not guarantee workflow success; nor can an end-to-end score tell the team which component to repair.

Diagnose the Polished Failure

Return to the original answer. Before reading the diagnosis below, choose the first artifact you would inspect. The final prompt is tempting because it is close to the visible failure. But the investigation should follow the evidence in the order the system transformed it:

  1. Corpus: Was the applicable source present, current, and permitted?
  2. Parsing: Were its tables, qualifiers, and headings preserved?
  3. Retrieval: Did the relevant passage appear in the candidate set?
  4. Ranking: Did it reach the context window?
  5. Assembly: Was it truncated, duplicated, or mixed with a conflicting source?
  6. Generation: Did each material claim follow from the assembled context?
  7. Presentation: Could the user inspect the cited passage and scope?
  8. Policy: Should the system have abstained or escalated?

In this incident, the applicable source existed and parsing recovered its text, but chunking severed the warning from the procedure. Retrieval then admitted both revisions because the query lacked a verified controller identifier. Assembly removed neither conflict. The generator joined the fragments into a sequence that appeared coherent.

The first repair is therefore not a stronger instruction to “use only relevant sources.” The interface must obtain or resolve the controller revision; the index must carry applicability metadata; the chunk must retain the warning; and assembly must reject incompatible passages. A generation-level groundedness check is still useful, but it cannot recover a condition that never reached the context.

A Correct Index Can Become Wrong Tomorrow

The manufacturer withdraws Revision C manual version 3 after discovering an unsafe step and publishes version 4. The RAG system now has a runtime obligation: version 3 must stop influencing answers within a defined interval. A nightly ingestion job is not a freshness policy unless that interval is acceptable and failures are visible.

Define a service level for additions, revisions, withdrawals, and deletions. Trace every derivative: parsed copy, index entry, embedding, cache, evaluation set, prompt log, and generated artifact if retained. Deletion is not complete merely because the source vanished from search; the old chunk may remain in a vector index or cached context. Test tombstones, partial failures, and rebuilds, not just insertion.

A practical freshness record includes source owner, authoritative location, synchronization frequency, last successful ingest, next expected update, failure alert, and maximum tolerated staleness. The product should expose evidence version and time when age affects use. A failed synchronization should change system behavior: warn, narrow the answerable scope, fall back to direct search, or stop answering the affected question class.

Design the controller corpus’s freshness policy now. Which event triggers re-ingestion? How quickly must a safety withdrawal propagate? Which cache and index copies require deletion? How will the team prove that version 3 no longer appears? What should the assistant do while version 4 is being parsed and evaluated? If the policy has no answer for that interval, it has described a schedule rather than a safe state transition.

The Design Record

A checklist is useful here because the evidence path crosses product, data, security, evaluation, and operations. Complete it for a named question class and corpus; a generic checkmark says very little.

Evidence and access

  • Authoritative sources and exclusions are named.
  • Permissions are enforced before retrieval results enter context.
  • Provenance, version, applicability, and classification travel with chunks.
  • Conflicts, missing evidence, and unauthorized queries have defined behavior.

Retrieval and assembly

  • Parsing preserves decision-critical structure.
  • Chunking is tested against realistic questions and exceptions.
  • Lexical, semantic, hybrid, and reranking choices have measured value.
  • Context assembly separates instructions from untrusted source content.

Evidence and operations

  • Retrieval and generation use separate evaluation sets and metrics.
  • Citation support is checked at claim level.
  • Freshness, deletion, latency, cost, and access violations are monitored.
  • Incidents can be reproduced from versions and identifiers.
  • Abstention is a successful outcome when the evidence contract is unmet.

Release the Evidence Path, Not the Demo

The maintenance assistant is ready only when the team can explain what happened to a question at every boundary. The test set proves that applicable passages are retrieved and inapplicable or unauthorized passages are excluded. Claim-level review proves that citations support the answer and that material statements remain within the assembled evidence. Freshness tests withdraw a manual through every derivative store. Abstention tests show that missing revisions, conflicts, and empty results reach a useful destination. Latency and cost remain acceptable for the real workflow, not merely for one model call.

The design will still have limits. Some procedures depend on a physical inspection the corpus cannot perform. Source owners may publish ambiguous instructions. A current manual can be wrong. RAG does not turn documents into truth; it makes a particular evidence path observable enough to test, govern, and challenge.

When the controller revision is known and the approved warning and procedure arrive together, the generator has a bounded writing task. When they do not, the system has a different job: say what is missing and stop. That refusal is not a degraded version of the answer. It is evidence that the contract held.

Source Notes