Vector Stores, Feature Stores, and AI-Adjacent Data Systems

The Answer Outlived Its Permission

Consider an internal documentation assistant. At 09:00, it quotes a recovery runbook to an engineer who is allowed to use it. At 09:10, the runbook is moved into a restricted collection because it contains a privileged command. The source platform records the new permission immediately. At 09:12, another employee asks the assistant how to perform that recovery. It returns the old passage with a confident explanation and a valid-looking citation.

Every service can be healthy while this happens. The vector index answers quickly. The embedding service is available. The generation request succeeds. CPU, memory, and error-rate dashboards remain quiet. The system is wrong because a copied fact has outlived a change to its meaning and audience.

That failure gives AI-adjacent storage its proper shape. A vector index, retrieval corpus, feature store, ranking table, or model-memory system is a derived data system. It copies source facts, transforms them, and serves a specialized decision. The transformation may be model-shaped, but the production obligations are familiar: authority, freshness, provenance, access control, deletion, evaluation, repair, and ownership.

The design question is therefore larger than “Where should we store embeddings?” It is: Which decision will this derived path serve, which source facts may influence it, and how will the team know when the path no longer deserves to answer?

Follow One Fact Through Retrieval

The restricted runbook begins as a source record with an identifier, content version, owner, sensitivity class, and permission policy. An ingestion process reads it. A parser extracts text. A chunker creates overlapping passages. An embedding model converts those passages into vectors. An index stores the vectors beside metadata. At query time, lexical and vector retrieval may produce candidates; filters remove ineligible records; a reranker changes their order; a prompt assembler selects excerpts; a model writes an answer; caches, logs, traces, citations, and feedback systems preserve parts of the interaction.

Each step can change what the employee sees. A parser can drop a warning from a table. A chunk boundary can separate a command from the condition that makes it safe. Approximate nearest-neighbor search can miss a relevant passage while returning a semantically similar one. A metadata filter can use yesterday’s group membership. A reranker can prefer an obsolete runbook. A cache can preserve an answer after the underlying record changes.

This is why a vector workload should be described as a retrieval workload. “We have five million embeddings” says something about capacity, but little about the behavior the product owes its users. A useful workload description names:

• the corpus, its growth and churn, its languages, tenant boundaries, and sensitivity classes;
• the request path and rate, including whether queries come from users, software, batch jobs, or another model;
• the exact filters that enforce tenant, permission, region, lifecycle, or product-state boundaries;
• the freshness required for additions, edits, revocations, and deletions;
• the result the caller needs: candidates, diverse recommendations, exact excerpts, cited evidence, or a justified refusal;
• the latency and cost budget, together with the consequences of a partial or empty result.

Vector similarity is only one access path inside this design. Exact identifiers, error codes, API names, and statutory phrases often favor lexical search. Structured filters may be non-negotiable for isolation. A reranker may improve ordering after several retrieval methods produce candidates. In many systems, hybrid retrieval is the honest architecture because semantic resemblance, exact matching, and policy filtering solve different parts of the request.

Operators must be able to inspect those parts separately. When the assistant gives the wrong recovery instruction, “AI search failed” is not a diagnosis. They need to see the source and query versions, candidates before and after filtering, reranker output, excerpts sent to the model, citations returned to the user, cache state, and the policy decision that admitted each passage.

Correctness Means Evidence You May Use

A vector index rarely promises that every nearest neighbor is exact. Its application still needs a correctness standard. For the runbook assistant, retrieved evidence must be relevant, current, permitted, traceable, and sufficient for the answer. If one of those properties is missing, the safe result may be a document link, a request for clarification, or no answer at all.

Return to the permission change. Updating the source platform is only the first transition. Existing chunks must become ineligible before unauthorized users can retrieve them. Cached candidate lists and generated answers may need invalidation. Snippets, citations, prompt traces, feedback queues, and evaluation cases need retention and access policies of their own. If the runbook is deleted rather than restricted, the system needs to know which derived artifacts should be erased, tombstoned, recomputed, retained under a documented policy, or reviewed as part of an incident.

The necessary lineage can fit beside each indexed unit. It should identify the source record and version, chunk and chunking version, parser and metadata-schema versions, embedding model, indexed time, source-modified time, permission or tenant scope, and deletion state. Not every field must participate in every query. All of them must be available when an operator asks why this passage was eligible.

Chunking then becomes a data-model decision rather than preprocessing trivia. Small chunks can improve pinpoint retrieval but remove context. Large chunks preserve context but dilute similarity, increase prompt cost, and make citations and deletion boundaries less precise. Overlap may rescue continuity while multiplying storage and invalidation work. The useful choice emerges from real questions, source structure, citation needs, permission boundaries, and evaluation—not from a universal token count.

An embedding-model change is a migration for the same reason. It changes neighborhood relationships and may change relevance, language coverage, latency, cost, and failure behavior. Mixing old and new vectors without intent makes results difficult to reproduce. Re-embedding in place removes the old behavior before the new behavior has earned traffic. A versioned replacement preserves comparison and retreat.

Feature Stores Preserve a Past That No Longer Exists

Feature stores share the derived-data shape, but their sharpest correctness problem is time.

Suppose a fraud model uses failed_payment_attempts_7d. For a payment requested at 10:05 on Friday, the training row must contain only failures that were available to the decision at 10:05. A chargeback reported on Monday may be useful as the training label. It cannot be smuggled into Friday’s input feature. Joining historical decisions to a table as it exists today can leak the future and make a weak model look excellent offline.

The online path has a different burden. When a new payment arrives, it must produce the same feature definition with current events and within the scoring latency budget. The offline job may use event time while the online service uses processing time. One may incorporate late events and the other may not. One may treat a missing value as zero and the other as unknown. One may resolve an account merge while the other continues using the old identifier. These differences create training-serving skew even when the model artifact never changes.

A feature definition therefore needs more than a name in a registry. It needs an entity key and identity rules; authoritative source events and owners; event-time, prediction-time, and processing-time semantics; window boundaries and late-data policy; computation and defaulting rules; offline and online implementations; freshness and latency targets; and a version that connects training data, serving data, and the model that consumed them.

The failed-payment feature also needs a repair story. If late events reveal that yesterday’s training extract was incomplete, the team should know whether to regenerate examples, retrain a model, annotate the evaluation, or accept the discrepancy within a declared limit. If a window changes from seven days to thirty, old and new definitions should coexist long enough to backfill, compare distributions, evaluate model consequences, move serving deliberately, and reverse the change if necessary.

A feature store can help enforce these relationships, especially when many models reuse features or when online and offline paths must agree. A registry that merely catalogs SQL expressions does not solve point-in-time joins, lineage, skew, deletion, or monitoring. The store earns its place by making those obligations easier to satisfy than well-owned pipelines would be on their own.

Evaluation Is the Control Loop

Infrastructure health cannot tell whether retrieval or features remain useful. The assistant can respond in 200 milliseconds with the wrong runbook. The fraud feature can arrive on time while encoding future information. Evaluation is the feedback mechanism that connects system behavior to the decision it serves.

For retrieval, keep a durable set of representative queries and expected source behavior. Include ordinary questions, near matches, obsolete and conflicting documents, exact-term queries, permission changes, deleted sources, weak-evidence cases, and questions that should receive no answer. Recall and precision help describe candidate quality, but averages alone can hide the failures that matter most. Citation support, permission correctness, refusal behavior, latency, and cost per query belong in the same release decision.

For features, monitor freshness, null rate, value distribution, entity coverage, late-event rate, offline-online skew, and drift from training baselines. Then connect those signals to model behavior. A distribution change may reflect a real change in the world, a broken pipeline, an identity migration, or a feature definition that no longer means what its name suggests.

Feedback can improve either path, but it is another derived dataset. Clicks favor what was already shown. Thumbs-up signals can reward fluent answers rather than supported ones. Human labels can encode inconsistent policy or overrepresent a small group. Feedback needs named sources, sampling and reviewer rules, retention and consent boundaries, and evaluation before it changes retrieval, ranking, training, or prompts.

The control loop must be able to stop a rollout. Better average recall does not excuse a failed permission probe. Higher offline model accuracy does not excuse future leakage. More clicks do not excuse weaker citation support for a safety-critical answer. Quality, security, latency, and cost are simultaneous constraints, not a scoreboard from which one winning number can be selected.

Build the Replacement Before Removing the Retreat

Parser, chunking, embedding, metadata, feature, and reranking changes can alter behavior as much as an application release. The safer migration builds a new derived version alongside the serving version when resources permit. It backfills from authoritative sources, catches up changes that arrive during the build, evaluates old and new versions against the same cases, and sends shadow or limited traffic where the product risk justifies it. Traffic moves only after the new version has passed the relevant quality and policy gates.

Rollback must restore behavior, not merely process availability. If a new retrieval index admits restricted documents, retreat includes suppressing the index, invalidating caches, reviewing exposed logs and prompts, and restoring a permitted answer path. If a feature backfill used future data, retreat may require removing affected examples, regenerating training data, retraining, and recording which model versions were exposed.

Full duplicate indexes may be too expensive. The team can rebuild by partition or tenant, evaluate a representative sample, use a maintenance window, or keep a source-backed fallback. Resource pressure changes the migration technique; it does not make comparison, recovery, or ownership optional.

Old versions also need a deliberate end. Retire their indexes, feature tables, snapshots, caches, evaluation data, permissions, alerts, and cost after rollback is no longer required and retention policy allows removal. Otherwise every successful migration leaves another poorly understood copy behind.

Make the Specialized Store Earn Operations

A specialized vector database is not mandatory because a design contains embeddings. A small corpus or low request rate may fit an existing search engine, a relational extension, a batch-built index, or memory inside one service. A feature store is not mandatory for one model with a few stable batch features. Ordinary search, curated documentation, deterministic rules, or a better workflow may solve a proposed RAG problem with less trust and operating surface.

Compare complete systems rather than feature lists. A separate store adds capacity planning, access control, backups or reproducible rebuilds, schema and index migrations, monitoring, incident response, cost review, upgrades, and an on-call owner. Its case strengthens when the workload requires capabilities the current platform cannot meet responsibly and the specialized system makes correctness and operations more legible. “It supports vectors” or “it has a feature registry” is not enough.

The rejection test is practical. If nobody owns permission and delete propagation, transform versions, evaluation cases, drift signals, rebuild duration, rollback, and cost, the system may remain a useful experiment. It has not earned a durable place in production.

Write the AI Data Contract

Before approving a vector index, RAG corpus, feature store, ranking feature set, or model-memory system, complete one record:

Served decision or user job:
Required result, refusal, and degraded behavior:

Authoritative sources, owners, and sensitivity:
Derived units and source-version links:
Parser, chunking, embedding, feature, filtering, and ranking versions:

Entity, event-time, prediction-time, and late-data rules:
Freshness targets for add, edit, delete, and permission change:
Access checks before retrieval, serving, logging, feedback, and export:

Representative evaluation cases and required outcomes:
Quality, drift, latency, and cost release gates:
Feedback source, sampling, consent, retention, and review rules:

Backfill and catch-up path:
Version comparison, traffic switch, and rollback path:
Derived-artifact deletion and retirement path:
On-call, quality, security, and cost owners:

Why the existing database, search system, or pipeline is insufficient:
Evidence that would cause us to remove this store:

Apply it first to the documentation assistant. Trace the restricted runbook from its source version through every chunk, vector, cache, prompt, citation, log, feedback record, and evaluation case. State how soon each surface must stop exposing the content and how an operator proves the change completed.

Then change the embedding model. Describe the rebuild, catch-up, comparison set, traffic switch, rollback window, and retirement of the old index. Finally, apply the time fields to failed_payment_attempts_7d: recreate its value for a historical prediction, inject a late event, and identify which training and serving outputs should change.

If the record stops at “the vector database handles it” or “the feature store handles it,” the design has named a product where it needed an operating argument.

Specialized stores are valuable when they make an otherwise difficult access path or time-dependent feature tractable. Their novelty does not exempt them from data-system discipline. Once admitted, they become one more copy of important facts—and the next architectural decision is whether the growing collection of stores still has clear authority, update paths, and owners.