Search and Retrieval Systems

The runbook that search almost found

At 02:13, an engineer investigating payment timeouts searches an internal knowledge base for PAY-2047. The first result is a polished troubleshooting guide for generic HTTP 504 errors. The exact incident runbook is fifth. A private postmortem appears in autocomplete, although the engineer cannot open it. The runbook itself was corrected an hour ago, but the result snippet still shows the retired command.

None of these failures means that search returned no matches. The system found plenty. It misunderstood which match mattered, exposed evidence from an ineligible document, and served a stale copy of the right one.

Search is a derived decision system. It copies source records into a form that can be retrieved quickly, interprets a query, gathers possible documents, removes those the user must not see, orders what remains, and returns a small claim about what is useful now. The hard part is preserving that claim as content, permissions, query language, ranking policy, and machines all change.

We can follow PAY-2047 through those decisions. The trace will also show why an inverted index, a vector index, a database lookup, and an autocomplete service solve different problems even when they sit behind one text box.

Decide what the index is allowed to know

The source of truth stores runbooks, incident records, team membership, and publication state. Search should not query that normalized model row by row. It needs a document shaped for retrieval:

document_id:       runbook-882
organization_id:  acme
title:             Payment gateway timeout PAY-2047
body:              ...
service:           checkout-api
incident_codes:    [PAY-2047]
allowed_groups:    [payments, on-call]
published_state:   active
updated_at:        2026-07-15T01:42:00Z
source_version:    918

This denormalized document is useful because the serving path can search the body, match the code exactly, filter by organization and group, boost an active runbook, display its title, and compare index version 918 with the source. It is also dangerous: it is another copy of private text with its own retention, deletion, and access behavior.

Document design therefore precedes engine choice. For each corpus, decide:

• the stable document unit and source record;
• which fields are searchable text and which require exact matching;
• which fields may filter, sort, facet, or appear in a snippet;
• the tenant and permission facts needed at query time;
• the version, lifecycle state, and deletion behavior;
• the lag the product can tolerate for creates, edits, revocations, and deletes.

The index remains a derived read model. It must be possible to replay changes, rebuild the corpus, verify it against the source, and discard it without losing authoritative state.

Give each field an honest language

An inverted index maps a term to a posting list of documents containing it. Postings may also record the field, term frequency, and term positions needed for phrase matching. A lookup for timeout can then visit a small candidate set instead of scanning every runbook.

Before indexing, an analyzer turns text into terms. Lowercasing, word boundaries, stemming, accent normalization, synonym expansion, and n-grams are not harmless cleanup. They define the language the search product understands.

Consider three fields from the runbook:

title text:     "Payment gateway timeout PAY-2047"
body text:      "Requests expire while awaiting provider acknowledgement"
incident code:  "PAY-2047"

The title and body may use language-aware tokenization and positions. The code needs a keyword representation that preserves punctuation, plus perhaps a carefully designed prefix representation. If every field shares a prose analyzer, PAY-2047 may be split into pay and 2047, producing broad matches and losing the exact identifier the engineer supplied.

The same problem recurs with C++, SRE-1234, email addresses, SKUs, usernames, and code symbols. Field analysis should follow intended queries:

• exact identifiers keep an exact field;
• titles can carry more weight than body text;
• prose can use language-specific normalization;
• controlled tags should not be stemmed into new meanings;
• synonyms should be added because judged queries show a miss, not because a thesaurus offers one.

A synonym such as television for tv may join two user vocabularies. A broad expansion of apple can join unrelated intents. Every analyzer change is a ranking change and usually requires a new index version, not an unobserved configuration edit in place.

Gather candidates before choosing a winner

The query PAY-2047 carries strong evidence of exact intent. The serving path should detect that shape and retrieve from the exact code field as well as the ordinary lexical fields. A query such as “gateway requests expire before the provider responds” expresses the same problem without sharing the runbook’s words. That is where semantic retrieval can help.

Lexical retrieval is strong at exact names, rare terms, identifiers, titles, and phrases. Common ranking functions reward evidence such as a rare query term, several occurrences in a document, a title match, or close term positions.

Vector retrieval embeds queries and documents and searches for nearby vectors. Approximate nearest-neighbor indexes trade some recall for speed and memory. They can recover paraphrases, but they can also overlook an exact constraint, return an opaque similarity, or become inconsistent when content and embedding models are versioned separately.

A hybrid path lets each mechanism contribute what it knows:

query: PAY-2047

exact-code candidates   -> runbook-882
lexical candidates      -> runbook-882, guide-104, incident-731
vector candidates       -> guide-104, runbook-882, note-331
                         |
                         v
merge by document ID -> enforce eligibility -> rerank

Candidate generation is deliberately generous. It tries not to lose a useful document too early. Ranking cannot rescue a document that no retrieval path returned, so a relevance failure must first be located: did analysis lose the term, did retrieval omit the document, did a hard filter exclude it, or did the ranker bury it?

Vector similarity never replaces exact lookup or structured filtering. An embedding may place a confidential postmortem close to a public runbook. That says something about meaning and nothing about authority.

Eligibility is part of correctness

For the engineer at 02:13, an eligible result must belong to the organization, be in a visible publication state, and satisfy the current group policy. Those constraints may be applied during candidate retrieval, by an engine-supported filter, or by a carefully sized post-filtering stage. They must hold before any title, snippet, highlight, facet count, suggestion, or generated answer is exposed.

This is stricter than checking permission when the user opens the result. The private postmortem title in autocomplete has already disclosed that the document exists. A snippet can disclose the document itself.

Eligibility also includes product constraints. A marketplace result may need the right region and sale state; a support article may need the active locale; an incident runbook may need published_state = active. These are not ranking hints. An excellent score cannot make a forbidden or unavailable item valid.

Permission filtering creates engineering trade-offs. Encoding every user’s document set directly may be too large. Group filters can become stale after membership changes. Filtering a tiny final page after global retrieval can produce empty pages and leak counts. The design should name the policy owner, the indexed permission representation, the revocation lag, and conservative behavior while that representation is uncertain.

Ranking turns evidence into product policy

After eligibility, the ranker decides what appears first. For PAY-2047, an exact incident-code match should usually dominate a generic semantic match. Within comparable candidates, the system might consider title match, lexical score, semantic score, document quality, publication state, freshness, and service ownership.

That ordering is a product policy, even when a learned model computes it. Popularity can amplify yesterday’s answer. Freshness can bury a definitive but old standard. Personalization can trap a user in familiar documents. A business boost can outrank the item the query names exactly.

Keep enough evidence to explain a bad result:

runbook-882
  exact_code_match: true
  title_match:      high
  semantic_score:  medium
  state:            active
  source_version:   918
  final_rank:       1

The explanation need not expose proprietary model internals to users. It must let operators distinguish an analyzer miss, absent candidate, ACL exclusion, stale feature, and ranking error. “The model chose it” is not a diagnosis.

Pagination must preserve the same honesty. Scores and documents can change between requests, so deep offset pagination is both expensive and unstable. A cursor commonly carries the last sort values plus a deterministic tie-breaker such as document ID. If a user needs an exhaustive export, give that operation a scan or export contract instead of pretending that a changing relevance list is a stable dataset.

Freshness has more than one clock

The corrected runbook is at source version 918 while the snippet still reflects 917. Search lag is visible because the document carries its source version, but the remedy begins in the indexing path:

source transaction
  -> durable outbox or change stream
  -> idempotent document builder
  -> version-aware index update
  -> searchable refresh

Create, edit, permission, and deletion events do not necessarily deserve the same priority. A new article might appear within a minute without harm. A revoked permission or deleted private document may require a much tighter bound, conservative denial while uncertain, and verification across the lexical index, vector index, autocomplete source, and snippet cache.

“Eventually consistent” does not tell the user or operator enough. A useful contract says what becomes searchable within which bound, how lag is measured, and what happens after that bound is missed.

Index lifecycle matters because analysis, document shape, and embedding models change. A safe replacement flow builds a versioned index from the source, replays changes that arrived during the build, compares representative queries and source counts, switches an alias or routing pointer, and retains a rollback path. Malformed documents need a quarantine or dead-letter path; otherwise one poisoned record can silently disappear on every rebuild.

Reconciliation is part of this system, not an emergency script. Compare source and index versions, measure the age of the oldest pending event, retry idempotent updates, and verify high-risk deletes. The previous chapter’s scarce inventory write could not tolerate two authorities. Search can usually lag that authority, but it must make the lag bounded and repairable.

Distribution changes the cost of one query

A large index is divided into shards and replicated for capacity and failure tolerance. A coordinator may send the query to many shards, ask each for its local top candidates, and merge those candidates into a global ranking.

This makes the slowest participating shard part of the user’s latency. More fan-out increases the chance of a straggler or unavailable replica. Sharding by organization or corpus can reduce fan-out when queries naturally stay within that boundary, but a poor key can create hot shards or uneven documents.

The system needs an explicit partial-result policy. For consumer discovery, a clearly measured partial page might be preferable to a timeout. For compliance search or an operator looking for an incident runbook, silently omitting a shard can be a dangerous false negative. The response, logs, and metrics should make that choice visible.

Replicas help read capacity and availability, but index versions must move coherently enough that a coordinator does not merge incompatible analyzer or embedding spaces. Track latency and failures per shard, not only at the front door, or the global p95 will reveal pain without locating it.

Autocomplete is its own disclosure surface

Autocomplete answers a different question: given a short prefix, which query or entity should be suggested before the user finishes typing? It often needs a prefix index, trie, or edge n-grams; strong popularity and abuse controls; and a latency budget tighter than full search.

Running the full search endpoint after every keystroke may work for a small corpus, but it does not create a deliberate suggestion product. Short prefixes are ambiguous, trending signals change quickly, and historical queries can be sensitive. Suggestions must be built from eligible sources and checked before exposure. The private postmortem in the opening is a permission failure even though the user never submitted the query.

Measure the miss at the stage that caused it

The query trace gives the team a compact regression case:

query:               PAY-2047
user groups:         payments, on-call
must rank first:     runbook-882 at source version 918
must not expose:     postmortem-731
maximum query p95:   product-defined target
maximum update lag:  product-defined target

Human relevance judgments say what should rank for representative queries. Offline regression sets catch analyzer, synonym, feature, and embedding changes before release. Online experiments reveal behavior at scale, but clicks need interpretation: a click may mean success, curiosity, or a misleading title, while a good snippet may answer the question without one.

Operational and relevance evidence belong together. Useful signals include zero-result and reformulation rates, task completion, judged relevance, p95 and p99 latency, shard failures, indexing and delete lag, quarantined documents, permission-filter failures, and the index versions serving traffic.

When a result is wrong, walk the same path the query took:

1. Was the intended document present and current in the index?
2. Did the analyzer preserve the query’s meaning?
3. Did lexical, exact, vector, or prefix retrieval produce it?
4. Did eligibility correctly include it and exclude restricted material?
5. Which ranking evidence placed it here?
6. Did shard failure, pagination, or a version mismatch alter the result?
7. Would an offline judgment or production metric have caught the failure?

That sequence is also a useful system-design explanation. Begin with the corpus and the user’s query shapes. Define the document and source of truth. Choose analysis and candidate mechanisms. Attach permission and lifecycle constraints. Then discuss ranking, freshness, sharding, and evaluation in the order that they affect a visible result.

The engineer at 02:13 does not need a system that contains every search feature. They need the exact, current, permitted runbook to appear before a plausible distraction. A sound retrieval design can explain why it will—and can identify the broken decision when it does not.