Case Study: AI/RAG Knowledge Base with Vector Search

The Confident Answer That Should Not Have Shipped

The internal assistant is popular within a month. Support uses it to summarize product behavior. Engineers ask for deployment runbooks. Sales asks it to explain security controls. New employees ask it policy questions that used to interrupt managers.

Then the trust failures arrive together. A support lead asks, “Can a customer export deleted conversation attachments during discovery?” The assistant answers yes, cites an old support note, and links to a page the support lead cannot open. The answer is wrong in three different ways. The policy changed last quarter. The cited note belonged to a restricted customer escalation. The current legal runbook says to route the request through a deletion exception review, but that page never made it into the index because the connector skipped a failed crawl.

This is the pressure that turns a RAG feature into a production data-system problem. Retrieval-augmented generation is not only prompt design around a vector database. It is a derived-data path from authoritative systems into chunks, embeddings, indexes, filters, retrieved evidence, generated answers, logs, evaluation fixtures, and deletion workflows. If the team cannot name the source of truth, prove freshness, enforce permissions before generation, migrate embedding versions, and explain why an answer used a particular document, the assistant is operating outside the discipline expected of the rest of the data platform.

The durable design principle is simple: the vector index is never the authority. It is a read model. It earns production trust the same way every derived view earns it: through contracts, lineage, versioning, observability, recovery, and tests that reflect the failures the business cannot tolerate.

Start With the Workload, Not the Model

The corpus is not one clean pile of documents. It is a mixture of policy pages, engineering design docs, deployment runbooks, incident retrospectives, customer tickets, support macros, source-code documentation, PDFs, spreadsheets, wiki pages, and product manuals. Each class has a different owner, permission model, update cadence, retention policy, and tolerance for stale answers.

The questions are just as mixed. Some require exact wording: “What is the current refund policy for annual contracts?” Some require synonym matching across informal language: “Why does the import job stop halfway through?” Some require identifiers and lexical precision: error codes, feature flag names, ticket IDs, API paths, customer account names, or policy section numbers. Some require the system to say no because the user lacks access or because the corpus does not support an answer.

Those pressures give the design its shape. Changing policies and runbooks require visible ownership, freshness, and source priority. Restricted notes living beside general documentation require permission filtering before generation. Deletion must reach chunks, vectors, caches, logs, exports, and evaluation fixtures rather than stop at the source. Exact identifiers require lexical and metadata signals alongside semantic similarity. Changes to chunking or embeddings require a versioned migration with evaluation and rollback. Because users may trust a fluent answer, citation and abstention are part of correctness rather than interface polish.

A production RAG system is therefore designed around answerability. For each question class, the team should know which sources are authoritative, how fresh they must be, which users may see them, which retrieval methods are allowed, what evidence must be cited, and when the assistant must refuse to answer.

Make Source Truth and Integration Controls Explicit

The first design repair is to restore authority to the source systems. The wiki remains authoritative for handbook pages. The policy store remains authoritative for legal and HR policy. The repository remains authoritative for code-adjacent runbooks. The ticket system remains authoritative for customer escalations. The vector index is an optimized retrieval surface built from those systems.

That distinction changes the ingestion contract. Each connector should emit a record with document identity, source system, source owner, source version or revision, event time, crawl time, content hash, extraction status, permission metadata, retention class, deletion marker, and retry state. Failed extraction should be visible as a failed source record, not hidden by an absent chunk. A missing policy document can be as dangerous as a stale one.

Stable source IDs prevent a rename or URL change from leaving duplicate chunks behind. Revision and content hash make a bad answer reconstructable after a source changes. Owner and authority tier prevent an informal support note from outranking legal policy. Extraction status and parse warnings expose the failed crawl that hid the current runbook. Permission snapshots retain the access meaning of a chunk, while a pointer to the current permission version lets sensitive reads reject an obsolete grant. Retention class and deletion state keep derived copies answerable to the source lifecycle.

Keep the extracted text and normalized metadata long enough to debug the pipeline. When the assistant gives a bad answer, the team should be able to reconstruct whether the source changed, the connector skipped it, the extractor damaged it, the chunking policy separated a rule from its exception, the embedding version reduced recall, the ranker preferred an inferior source, the ACL check failed, or the answer generator ignored retrieved evidence.

The connector should be boring in the best sense: idempotent, observable, retryable, and conservative. It should not silently “clean up” documents in ways that remove section headings, table labels, policy exceptions, code fences, or permission markers. Those details often decide whether retrieval finds the right chunk and whether the answer can be trusted.

Chunk for Evidence, Not Just Similarity

Chunking is where many RAG systems quietly lose correctness. A refund-policy page might contain a general rule, a regional exception, a renewal exception, and an escalation path. If the chunks split the exception from the rule, the assistant can retrieve a plausible but incomplete answer. If the chunk is too large, the relevant sentence may be diluted by unrelated text and waste context budget. If overlap is used blindly, sensitive or stale material is copied into more vectors than necessary.

Use document structure as the starting point. Headings, section paths, tables, list items, code blocks, decision records, and runbook steps should influence boundaries. Chunks should carry enough surrounding context to be cited honestly: document title, source system, owner, section path, revision, effective date when present, indexed time, permission scope, content hash, language, and deletion state.

For policy and runbook material, preserve decision units. Keep a policy rule with its exceptions and escalation pointer when possible. Keep ordered runbook steps connected to rollback notes. Repeat table headers with any separated row group so values retain their meaning. Carry the explanation, version, and environment with code or configuration. Split a long design document by section path, but retain its status, owner, summary, and decision date.

Applied to the discovery question, this rule keeps the prohibition on ordinary export beside the exception-review path. Retrieving only the word export is insufficient; the evidence unit must tell the support lead which rule applies and where the exceptional decision belongs.

Overlap is a tool, not a default virtue. It can protect context across boundaries, but it also increases index size, duplicates sensitive text, complicates deletion proof, and may make stale fragments appear more often. The only durable answer is to test chunking policy against the evaluation set and the incident history.

Retrieval Needs Versioned Indexes and Hybrid Ranking

Embeddings are derived data. Changing the embedding model, text preprocessing, tokenizer assumptions, chunking policy, metadata schema, ranker weights, or access-filter representation creates a new retrieval product. Treat it as a migration, not a parameter tweak.

A safe migration builds the new index beside the old one. The release plan should name the corpus snapshot, connector versions, chunking policy, embedding model identifier, embedding dimensions, metadata schema, ranker configuration, ACL-filter method, and evaluation run. Then the team can compare old and new retrieval on the same question set before routing users to the new path.

The cutover should have stages:

1. Build index_v_next from a declared corpus snapshot and current deletion ledger.
2. Run quality evaluation against expected-source and forbidden-source fixtures.
3. Run security evaluation with users from different teams, tenants, regions, and roles.
4. Replay recent production queries without showing new answers to users.
5. Compare recall, citation coverage, stale-source rate, forbidden-source rate, latency, and cost.
6. Shift a small percentage of eligible traffic while logging both old and new retrieval traces.
7. Keep rollback available until support, security, and the source owners accept the new behavior.

Every answer trace should include index version, embedding version, chunking policy, query transformation, filters applied, ranker configuration, retrieved chunk IDs, source revisions, and citation IDs. This is not excessive bookkeeping. It is the difference between “the assistant got worse” and a debuggable production incident.

Hybrid search is part of the same discipline. Vector similarity helps when users describe a concept in language different from the source. It is weak when exact identifiers matter. A production knowledge base usually needs semantic retrieval combined with lexical search, metadata filters, recency, source authority, document type, and sometimes source-specific rules. An error code, policy title, API path, or customer ID should not disappear because its embedding is not semantically rich.

Ranking should make authority visible. The current policy page should beat a stale support note for a policy question. A checked-in runbook should beat a chat transcript for incident procedure. A restricted customer escalation should never be retrieved for a user who cannot see it, even if it is the most semantically similar document in the corpus.

Put the Access-Control Gate Before the Model

Access control is not a citation problem. It is not enough to remove restricted links from the final answer if the model already saw restricted chunks. The retrieval path must intersect candidates with the user’s effective permissions before answer generation.

For low-risk content, the index can carry permission metadata and filter by tenant, group, role, region, document classification, and source-specific ACL identifiers. For high-risk content, retrieve candidate document IDs first, check the current permission service or source system, then fetch allowed snippets. That adds latency, but it avoids relying on stale permission snapshots for sensitive material.

The access decision should be logged without leaking content. A useful audit event records user identity or service identity, permission context, query class, source classes searched, filters applied, candidate counts before and after filtering, blocked document IDs or hashes where safe, allowed citation IDs, index version, and decision timestamp. The logs should help security prove that restricted documents were excluded without turning the log store into another copy of the restricted corpus.

The assistant also needs answer-time behavior for partial access. If the best source is blocked, the system should not summarize it indirectly. It can say that available sources do not support an answer, or that the user may need access to a named source class through the normal request path. It should not infer restricted facts from neighboring chunks.

Treat Freshness and Deletion as Release-Critical

Freshness should be visible to users and operators. A source can have a crawl time, an indexed time, and an effective date, and those are not the same. For critical collections such as legal policy, HR policy, customer commitments, incident runbooks, or security controls, define freshness SLOs. A collection that has not crawled successfully inside its SLO should be marked degraded, excluded from certain answer classes, or called out in the answer.

Deletion requires more than removing a row from the source connector. The purge path must cover raw extracted text, normalized documents, chunks, embeddings, vector index entries, lexical index entries, caches, answer traces, exported logs, evaluation fixtures, training or tuning datasets if any exist, and backup or retention stores according to policy. Some logs may be retained for security or audit, but their retention and redaction rules must be explicit. Do not let the vector index become the forgotten copy.

Maintain a deletion ledger. It should record source document ID, source system, deletion event time, received time, affected chunk IDs, affected vector IDs, affected cache keys, affected evaluation fixtures, purge completion time, exceptions, and verifier result. The verifier should query by document ID and by representative content fingerprints. A deletion workflow that only checks the source table can miss derived stores.

The three failures in the discovery answer now lead to different controls. The superseded policy gains an authority tier and effective date, and its retrieval becomes a stale-source evaluation case. The restricted support note becomes a forbidden-source fixture for the support role and must be removed by the pre-generation access gate. The missing legal runbook turns a failed crawl into visible freshness debt that blocks this answer class until ingestion recovers.

These controls make RAG less magical, but more trustworthy. The assistant becomes another consumer of governed data, not an exception to governance because it has a conversational interface.

Evaluation Is the Operating Loop

Evaluation is where RAG stops being a demo and becomes an engineering system. The test set should be assembled from real question classes, incident reports, source-owner expectations, and security concerns. It should include questions with expected sources, questions with forbidden sources, questions that require exact terms, questions that require current policy, questions that should abstain, and questions that exercise different permission contexts.

The opening question becomes one inspectable evaluation fixture:

The dashboard should break results down by source class and permission class. Aggregate recall can look healthy while HR policy is stale, support notes leak, or exact identifier queries fail. Track expected-source recall, citation coverage, stale-source rate, no-answer rate, forbidden-source violations, deletion propagation age, retrieval and answer latency, index freshness, evaluation drift, and cost per answered query. Precision belongs beside recall when the fixture defines a relevant set rather than one expected source.

Release gates should be blunt where the risk is blunt. Zero known unauthorized retrievals in the evaluation set. No release if critical source collections are outside freshness SLO. No release if deletion verification fails. Quality gains from a new embedding version do not justify weaker access control or a missing purge path.

AI Data Index Contract

Before releasing or materially changing a RAG index, complete this contract in the architecture review:

The contract should be short enough to use and serious enough to block a release. Its purpose is not to document that a vector database exists. Its purpose is to prove that the assistant’s knowledge path is governed with the same care as any other production read model.

Case Review Drill

Use the opening incident as a review exercise. The assistant answered a legal-support question from a superseded policy, cited a restricted customer note, and missed the current runbook because an ingestion job failed.

Trace the failure through the system:

1. Identify the authoritative source for each cited or missing document.
2. Check connector logs, extraction status, source revision, crawl time, and content hash.
3. Inspect chunk boundaries around the rule, exception, and escalation path.
4. Compare retrieval under the old and proposed embedding/index versions.
5. Verify lexical matching for policy title, discovery terms, and deletion language.
6. Replay the query under the support lead’s permission context and a restricted-user context.
7. Prove the old document is stale by effective date and authority tier.
8. Add evaluation fixtures for the expected source, forbidden source, abstention behavior, and deletion path.
9. Plan an index migration that keeps the old index available until quality, access-control, freshness, deletion, latency, and cost gates pass.

The drill is deliberately unglamorous. It asks the same questions this book has asked about ledgers, catalogs, notifications, observability stores, warehouses, tenant placement, and every other system that carries facts under pressure: where did the fact come from, who may see it, how fresh is it, what derived copies exist, how do we recover or delete it, and what evidence proves the system behaved correctly?

This final case returns to the book’s governing judgment. Production data engineering is not a preference for one storage engine over another. It is the habit of making facts accountable as they move through models, indexes, queues, caches, replicas, analytical stores, search systems, and now AI-facing retrieval paths. When the interface becomes conversational, the obligation does not get lighter. The system still owes the user source truth, boundaries, recovery, and proof.