Case Study 1: Internal Knowledge Assistant

The Answer That Revealed Too Much

A finance analyst asks an internal assistant how the company handles vendor disputes. The answer is accurate, concise, and supported by a confidential legal memo the analyst cannot open. When she selects the citation, the document system refuses access.

The model did not invent the answer. The system still disclosed the substance of a restricted document. Its citation made the failure easier to discover, not safer.

The team’s original goal—“let employees ask questions across company knowledge”—cannot survive that discovery. The deployable use case is narrower: help an authenticated employee find and synthesize evidence they are already allowed to access, while making uncertainty and source boundaries visible. The restriction changes ingestion, retrieval, generation, interface design, evaluation, incident response, and the order in which repositories can join the service.

Start With the Existing Search Path

Today, employees search one repository at a time, open documents, compare dates, and assemble an answer. This is slow, but it has useful properties: repository permissions decide which results appear, the employee can inspect the source, and no generated sentence disguises a conflict between documents. The team records time to a supportable answer, abandoned searches, source openings, and corrections on representative tasks before introducing the assistant.

The proposed system should shorten that work without weakening its evidence boundary. A product owner owns task completion and adoption. A knowledge owner decides which collections are eligible and who resolves stale or conflicting guidance. Security owns identity, authorization, leakage tests, and incident response. An evaluation owner maintains the retained test set. No single blended “answer quality” score can discharge those responsibilities.

The team refuses three shortcuts:

• no cross-repository retrieval until document-level permissions propagate into the index;
• no answer when usable evidence is missing, contradictory, stale beyond policy, or inaccessible to the requester;
• no autonomous action based on an answer.

These are system requirements, not prompt suggestions.

Authorization Must Survive Indexing

The first prototype copied text into a shared vector index and attached the repository name as metadata. That was enough to produce the legal-memo failure. Repository membership was not the true authorization rule: some folders inherited exceptions, some documents named individual readers, and some permissions changed after indexing. Once restricted text entered the model’s context, asking the prompt to respect permissions was already too late.

The replacement index treats authorization as part of each source’s lifecycle. Every chunk retains its document identity, owner, version, effective date, classification, and access-control reference. An ingestion worker refuses records whose permissions cannot be represented. Permission changes and deletions invalidate indexed material, and a reconciliation job looks for drift between the source repository and the index.

At query time, the application resolves the employee’s current identity and groups. Authorization filters reduce the candidate set before semantic retrieval and reranking. The system then verifies that the surviving documents are still accessible before it constructs the evidence packet. Routine logs keep versions and document identifiers rather than raw sensitive passages.

This design deliberately accepts lower recall. If the assistant cannot find enough permitted evidence, it must produce no answer even when a restricted collection contains an excellent one. The useful question is not whether the model “knows” the policy. It is whether this employee can inspect the evidence for this answer now.

Make the Answer Show Its Work

The generator receives a small evidence packet, not an undifferentiated dump of nearby chunks. Each passage carries its source location and date. The response format distinguishes directly supported claims, synthesis across sources, conflicts, and missing evidence. Citations open beside the answer, where the employee can compare wording and effective dates without losing the question.

Suppose one policy says that procurement owns vendor disputes while a newer regional addendum assigns certain cases to finance. A fluent merger would hide the operational choice. The assistant instead names the conflict, shows both permitted passages, and asks the employee to choose the relevant region or contact the knowledge owner. When evidence is absent, stale beyond policy, or inaccessible, it abstains and offers the ordinary search path.

This is also where overreliance becomes observable. The interface records whether people open evidence, report a stale source, revise a synthesis, or abandon it. Source opening is not proof of understanding, but a consequential workflow in which nobody ever inspects evidence deserves scrutiny.

Evaluate Components and Workflow Separately

The evaluation set starts with real question families from the pilot collection: common policies, difficult terminology, recent changes, conflicting sources, and plausible questions with no approved answer. Each question has an expected evidence set, permitted roles, freshness condition, and acceptable abstention behavior.

Authorization is tested first because later quality scores cannot compensate for a leak. For every role pair, forbidden documents must remain outside retrieved context—not merely absent from the final citation list. The suite changes group membership, removes access, deletes documents, uses indirect requests for restricted facts, asks what an inaccessible collection contains, and places malicious instructions inside an otherwise authorized document. Any known cross-role retrieval blocks release.

Among the permitted candidates, retrieval is measured by question family. A useful policy source appearing near the top for common vocabulary does not excuse failure on an internal acronym or a recently revised rule. Generation review then traces material claims to passages and verifies that citations resolve. A severe unsupported claim or a concealed conflict also blocks release.

Only after those gates hold does the team compare the complete workflow with search. Employees must reach correct, supportable answers faster without greater correction burden. The comparison includes unanswered questions; forcing a plausible response would improve completion while making the service less trustworthy.

Launch as a Bounded Service

The first pilot uses one current, well-owned policy collection and a small employee group. It excludes the legal repository that exposed the original defect. New repositories require separate evidence about permissions, ownership, freshness, and question families; connection to the ingestion pipeline is not approval to serve their contents.

The operating view keeps unlike failures apart: retrieval miss, unsupported synthesis, authorization denial, stale-source report, latency, cost, source opening, and user correction. Each signal has an owner and response. A suspected access leak disables generation or the affected collection immediately while preserving a search-only fallback.

Three weeks into the pilot, an employee changes departments at noon and retains answers from the old department until the next morning. The source repository revoked access correctly; the assistant’s identity cache did not. Security pauses the collection, preserves the query, identity, index, prompt, model, and policy versions, and investigates without copying answer text into a broad incident channel. The team shortens the cache lifetime, adds revocation tests across every authorization layer, and reruns the retained access suite before restoration.

The incident changes the design review. “Filter before retrieval” was necessary but incomplete: the filter must use current identity state, and revocation latency must have an explicit bound. Expansion now requires retained evaluation results, healthy source ownership, acceptable unanswered-query behavior, evidence that people inspect citations when consequences are material, and a tested path from permission change to index and cache invalidation.

Design Review Record

• Use case: authorized knowledge synthesis, not unrestricted company question answering.
• Primary harm: disclosure across access boundaries; secondary harms include stale guidance and overreliance.
• Key control: current identity-aware filtering before retrieval, followed by source revalidation, evidence, and abstention gates.
• Acceptance evidence: revocation and access-control suite, retrieval results by question family, supported-claim review, and workflow comparison.
• Operating owner: product and knowledge owners jointly; security owns leakage response.
• Pause trigger: any confirmed unauthorized retrieval or repeated high-severity unsupported claim.

The design record is intentionally brief because it points to versioned tests, source policy, evaluation results, and the incident playbook rather than repeating them. Product and security sign it before the pilot and after any material change to identity, indexing, retrieval, generation, or repository scope.

The legal-memo answer was fluent, grounded, and wrong to provide. That combination is the lesson. RAG creates a potentially inspectable evidence path; only current permissions, provenance, abstention, an evidence-bearing interface, and operating discipline preserve that path from source to employee.

Source Notes and Next Step

See RAG Systems, Privacy by Design, and Evaluating RAG. The next case shifts from evidence retrieval to a copilot whose output can reach a customer only after meaningful human review.