Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 28

Evaluating RAG and Knowledge Systems

Trace RAG failures through corpus, retrieval, context, generation, policy, and user experience so fixes target the responsible layer.

The Missing Answer Was Never a Model Problem

An internal procurement assistant repeatedly gives an outdated contract-cancellation period. Engineers change the prompt, add a stronger model, and tighten the citation instruction. The answer remains wrong. The current policy PDF failed during parsing, so the index contains only the superseded version.

Every attempted fix targeted the last component in the chain. The defect lived at the beginning.

Evaluate a knowledge system as an evidence supply chain. A useful result identifies not only whether the answer failed, but where the applicable evidence disappeared and which owner must restore it.

A six-layer RAG evaluation inspection line shows corpus coverage, retrieval, context assembly, generation, policy enforcement, and user display, with a distinct test and failure exit at each layer.
Trace evidence in order. If the applicable source never entered the corpus or candidate set, changing generation cannot repair the answer.

Give the Test an Evidence Trail

A reference answer can tell the team that the assistant was wrong. It cannot tell them whether the current policy was absent, filtered, ranked too low, truncated, contradicted by the generator, or hidden from the user. The test fixture must describe the path that a correct answer could have taken:

question and intent
applicable source identifiers, versions, and passages
required claims and prohibited claims
user identity, permissions, region, and effective date
expected outcome: answer, qualify, abstain, or escalate

For the cancellation question, the fixture names the active contract-policy version, the clause that governs this supplier class, the superseded clause that must not be used, and the employee group permitted to see both contract and amendment. It also states what to do if the amendment is missing: do not guess a number; route the request to procurement operations.

Build the set from more than clean, answerable questions. Include partial evidence, conflicting versions, stale documents, unauthorized sources, ambiguous language, and genuinely unknown answers. A fixture with only reference prose can judge the ending of the pipeline. A fixture with evidence and applicability can inspect the whole journey.

Start Where Evidence Enters the System

Before calculating retrieval metrics, ask whether the corpus can support the task. The current cancellation policy must be present, parsed faithfully, marked as authoritative, associated with its effective date and supplier scope, and connected to the amendment that replaced the old period.

A coverage audit should follow the source families that matter to the product: their owners, formats, languages, jurisdictions, access classes, and update expectations. It should also exercise ingestion. Headings must remain attached to their content; table cells must retain headers and row meaning; warnings and exceptions must stay with the procedures they qualify; anchors must resolve; replacements and deletions must propagate; partial ingestion must be visible as failure rather than silent success.

This is where the procurement team finds the first defect. The current PDF exists in the source repository, but the parser rejected one page and the ingestion job retained the previous index without raising an alert. There is no relevant current passage for the retriever to find. Reporting poor retrieval recall would assign blame to a component that never received the evidence.

The repair belongs to ingestion: fail the update visibly, preserve last-known-good status without representing it as current, reprocess the policy, and add this PDF shape to the parser regression set.

Observe Retrieval Before Judging the Answer

Once the applicable passage exists in the corpus, inspect whether it reaches the candidate set and survives ranking. With passages or documents graded for relevance, the team can use several familiar measures. Recall@k asks whether the usable evidence appeared among the first k candidates. Precision@k asks how much of that set was relevant. Mean reciprocal rank rewards putting the first relevant result early. NDCG is useful when relevance has degrees and the order of several results matters.

The cutoff must come from the real system. Retrieval at 100 is not success if reranking and the context budget admit only five passages. Results should be sliced by exact identifiers, paraphrases, long and ambiguous queries, rare topics, language, document type, freshness, and permissions. Lexical search, semantic search, filters, and reranking should expose their own source identifiers so the first disappearance can be located.

After reingestion, the cancellation clause appears at candidate rank twelve. Five near-duplicate supplier guides crowd the top of the list because they repeat the query’s language, although none contains the governing amendment. The team groups duplicates and adds authority, effective date, and document relationships to reranking. The clause now reaches the set that context assembly can use.

No model has generated a word yet. The evaluation has already produced two actionable repairs.

Inspect What the Generator Actually Received

Candidate retrieval is not the same as usable context. A reranker can retain the governing clause while a context builder drops its exception, separates a table row from its header, or spends the budget on duplicate text. Capture the exact assembled context, with provenance and component versions, as a regression fixture.

For the procurement question, the assembly step includes the active clause but truncates the amendment sentence that limits the period to a particular supplier class. The remaining text looks decisive and is wrong for this contract. A broad “context relevance” score may miss that loss if it rewards topical similarity. The judgment must ask whether the context is sufficient and applicable, whether decisive qualifiers survived, whether conflicting sources remain distinguishable, and whether retrieved content is separated from system instructions.

The team changes assembly to keep linked clause-and-exception chunks together and to prefer diverse, authoritative evidence over repetition. It also preserves source version and applicability metadata for generation and display. The new fixture fails if the qualifier disappears again.

Judge Claims Against Applicable Evidence

Only now does answer evaluation become diagnostic. Break the response into material claims and ask whether an allowed passage supports, contradicts, or says nothing about each one. Claim-level groundedness, citation precision, citation completeness, required-claim coverage, unsupported-claim rate, and abstention quality describe different failures; one average should not let a polished citation compensate for a false cancellation period.

A citation can resolve and still fail. It may point to a real document that never states the adjacent claim. It may faithfully quote a superseded policy. It may support the general rule while omitting the exception that governs this supplier. Groundedness therefore depends on the earlier fixture’s version, authority, scope, and date—not merely on textual resemblance between answer and source.

With the repaired context, the generator states the right period and cites the amendment. On a conflicting-version test, however, it silently chooses the newer document without explaining why it applies. The release requirement is sharper: when applicability cannot be established from permitted evidence, qualify or escalate instead of presenting one version as settled.

Permissions Change the Correct Evidence

For a public procurement guide, a broadly visible answer may be appropriate. For a negotiated supplier amendment, the same query from another employee may have no permitted evidence. Permission-aware evaluation keeps the question fixed while changing identity, group, tenant, region, or document classification. Unauthorized material must not appear in candidates, assembled context, snippets, citations, caches, logs, or generated claims.

Test inference leakage as well as direct quotation. An answer can reveal a protected cancellation term without naming its source. Attempts to summarize an inaccessible amendment, compare confidential versions, follow instructions embedded in retrieved text, or exploit stale group membership belong in the suite.

The expected response for an unauthorized employee is not “a confidential amendment exists.” It is a useful statement that no applicable evidence is available to that user, followed by the approved route for help. Access invariants are security controls with named owners and hard gates. They should not be diluted into average answer quality.

The User Must Be Able to Verify the Result

The authorized employee finally receives a correct, grounded answer. The test is still incomplete. Can the employee see which clause supports the period, open the exact section, understand its date and scope, notice unresolved conflict, and escalate without reconstructing the search by hand?

Observe the work, not only a “helpful” rating. Measure task completion, verification time, correction severity, calibrated reliance, and escalation success. In this case, a citation to the policy home page forces the employee to search a long PDF and compare versions. Deep-linking the clause, displaying the effective date, and naming the supplier scope reduce verification work while making an incorrect answer easier to challenge.

The interface is part of the knowledge system. Evidence hidden behind a generic link has not fully reached the user.

Locate a Failure with Controlled Substitutions

When a case fails, replace one stage at a time. Give the generator the known-good passage. If the answer still fails, investigate generation or the judging rule. Place the known-good passage directly into first-stage candidates; if reranking loses it, the ranker owns the defect. Insert it after reranking; if context assembly damages it, inspect truncation, grouping, and metadata. Run the source under an authorized identity to distinguish missing evidence from an access filter. Swap in a known corpus fixture to isolate ingestion and freshness.

The order matters because each substitution establishes what the next component actually received. “RAG quality declined” becomes a trace: the authoritative amendment was absent; after ingestion repair it ranked twelfth; after ranking repair its qualifier was truncated; after assembly repair an unauthorized user could infer the term. Each result has a different owner and a different regression test.

Release the Chain, Not a Composite Score

The team versions the corpus, parser, chunker, embeddings, index, filters, ranker, context builder, model, prompts, access policy, and interface. A result without that lineage cannot gate a later release: the measured system no longer exists as soon as one unrecorded component changes.

Release evidence follows the same chain. Critical source families must be present and parsed without silent partial updates. Retrieval must meet floors for important question classes within the context budget. No decisive qualifier may disappear during assembly. Material claims must be supported by applicable evidence, and severe unsupported claims must remain below the product’s explicit tolerance. Paired permission tests must show no access violation. Intended users must be able to verify, correct, and escalate within the workflow’s target time and severity bounds.

These conditions do not need equal weight. A permission breach or unsafe invented contract term can block release even when common-question recall improves. A corpus gap may justify limiting the assistant to source families with demonstrated coverage. A display problem may permit an internal pilot with mandatory specialist review but prevent direct user delivery. The evaluation should support such bounded decisions instead of compressing them into one RAG score.

Return to the original cancellation failure. Trace it from source repository to user display and mark the first point at which the governing amendment disappears. Then change one fact: the employee no longer has access to the negotiated contract. The evidence path and the correct response should both change. If the test expects the same answer, it is measuring prose rather than a knowledge system.

RAG evaluation is complete when the team can say where applicable evidence was lost, what consequence followed, who owns the repair, and which test will catch its return.

Source Notes