Feeds, Search, and Recommendations

Why did search miss the document?

Nadia creates a private project document called “Orchid launch plan,” then searches for it by title. The results contain an old launch template and several popular documents, but not the one she just made. A teammate with different permissions runs the same query and should see a different page. While the team investigates, the learned ranker times out.

Which part of the system is broken?

“Search” is too broad an answer. The write may not have reached the index. The query may not have found the right shard or vocabulary. The new document may have been found and then lost below a shard’s cutoff. A permission rule may have removed it correctly—or removed it using stale data. The ranker may have preferred popularity to title match. The serving tier may have returned a cached page. Each explanation points to different state, evidence, and repair.

This is the useful shape beneath feeds, search, autocomplete, recommendations, and trending pages. They all assemble a provisional set of things that might be shown, spend bounded work ordering it, reject things that must not be shown, and return a page under a time budget. Their product intent and storage differ, but the central discipline transfers.

Give the query a contract

Before choosing an index or model, make “good search” concrete. In Nadia’s workspace, assume title and body search over documents. Newly saved documents should normally become discoverable within seconds. Results must respect tenant membership, document permissions, blocks, deletion, and legal or policy state. Exact title matches should beat merely popular documents. A slow shard may produce an explicitly degraded page; an authorization failure may not.

That contract exposes the questions worth asking in an interview:

• What is the user doing: issuing an exact query, browsing, typing ahead, or asking for discovery?
• What does a satisfying result mean for this product—lexical match, freshness, proximity, diversity, availability, or a longer-term outcome?
• How soon must creates, edits, deletes, permission changes, and moderation decisions affect what can be returned?
• Which rules affect quality, and which are correctness boundaries that may never be weakened during degradation?
• What corpus and skew dominate the work: document count, update rate, a few hot tenants, celebrity producers, popular prefixes, or ranking depth?
• Which feedback is observable and trustworthy? A click without an impression record says little about the choices the user was actually offered.

A concise opening answer can now carry the design:

I will separate canonical document state from the retrieval index. The query
path will generate a broad but bounded candidate set, rank it for this search
intent, enforce tenant and document permissions, and serve a stable page. I
will make index lag, filtered candidates, ranker fallback, and partial results
observable separately.

The components no longer arrive as a memorized diagram. Each one earns its place by preserving a part of the contract.

Follow “orchid launch” through the system

The document service owns the canonical body, title, tenant, author, version, visibility, deletion state, and permission policy. Saving a document commits there first. In the same transaction, or through an outbox coupled to it, the service records durable indexing work. The acknowledgement means the document is saved; it need not falsely claim that every search replica already contains it.

Indexing workers extract searchable fields, normalize text, and build an index document containing the canonical document id and source version. They attach the tenant and whatever compact permission attributes the query engine can apply safely. An inverted index maps terms and fields to document ids. If the product genuinely needs semantic retrieval, an embedding index may contribute another candidate source, but it does not replace the lexical index or the permission model.

At query time, the coordinator authenticates Nadia, resolves her tenant and relevant principals, normalizes orchid launch, and chooses candidate sources. It sends the query only to eligible partitions. Each shard retrieves more than the final page size, applies the permission predicates it can enforce before its local top results are chosen, and returns scored candidates with document ids, source versions, and score ingredients. The coordinator merges those lists, removes duplicates, and spends expensive ranking only on a bounded set.

The order of those steps matters. Suppose each shard returns its best ten documents and authorization is checked only after the global merge. If most of those documents are inaccessible to Nadia, an authorized result at position eleven never reaches the coordinator. “Filter at the end” is safe from disclosure only if no protected fields leak, but it can quietly destroy recall. Tenant isolation and stable access predicates should therefore be pushed into retrieval where possible. Complex or rapidly changing rules may still require over-retrieval and an authoritative final check.

The final serve path rechecks the states whose staleness would violate the contract: deleted or quarantined content, revoked access, a tenant boundary, or another urgent policy change. A stale index entry is still only a candidate. If final filtering empties the page, the system can retrieve deeper within a strict work budget or return an honest empty result. It cannot refill the page with documents from the wrong audience.

A useful decision record for one result need not expose model internals to the user, but it should let an authorized operator answer why the document appeared:

query_id: q-81f
query: normalized and access-controlled in diagnostic storage
candidate: doc-204 from title_lexical, source_version 37
rank: title match + freshness; ranker_version r12
filters: tenant allowed, ACL version current, not deleted
serve: position 2, index_generation g91, fallback false

Sensitive queries, user features, and document text deserve strict retention and access controls. Debuggability is not permission to build an unrestricted surveillance log.

Freshness has more than one clock

Nadia’s missing document may simply be between canonical commit and index visibility. Measure that interval directly. Queue age alone is insufficient: workers may be caught up while one shard refuses refreshes, or the index may contain version 36 while the document service has version 37.

For ordinary updates, workers can write small index segments and make them searchable on a near-real-time refresh cadence. A product that promises read-your-own-write search can also maintain a short-lived overlay for the author’s recent documents, or return a “saved, still indexing” state with a direct link. The overlay is a narrow product choice, not a reason to pretend the entire corpus updates synchronously.

Deletes and access revocations often need a faster path than relevance edits. A tombstone or revocation stream can invalidate serving caches and feed a compact deny layer while normal reindexing catches up. The final authority check remains the safety net. This asymmetry is intentional: a stale title is annoying; a revoked private document shown to the wrong user is a breach.

Analyzer changes, schema migrations, embedding changes, and corrupted shards require a full index lifecycle. Build a new generation beside the live one, backfill it from canonical state, compare coverage and representative queries, then switch traffic gradually. Retain a rollback window. Rebuilding the only live index in place turns routine evolution into an outage and makes it hard to distinguish missing data from changed ranking.

Ranking is a policy with a failure mode

For orchid launch, lexical title match is strong evidence. Freshness and document quality may refine it. Workspace activity, author affinity, or a semantic candidate can help with less exact queries, but every signal should serve the declared product goal. A generic engagement score can easily place a widely read template above the document Nadia named.

Candidate generation should be broad and comparatively cheap; ranking should be deep only over a controlled set. A practical query may blend title and body matches, exact entities, and semantic candidates, cap any one source so it cannot occupy the whole pool, enrich the survivors with fresh features, and then rank. The design should record source attribution and feature freshness well enough to explain regressions by tenant, query class, or candidate source.

When the learned ranker times out, the search page still has defensible lexical scores. Fall back to them, shorten the candidate set, and record the fallback. When a nonessential semantic source is slow, omit that source rather than waiting past the page deadline. When one shard fails, the product may return partial results if that behavior is clear internally and appropriate to the surface. None of these degradations may disable permissions, safety, tenant isolation, or deletion checks.

Pagination makes the changing order visible. Re-running an evolving ranking function for every page can duplicate or skip documents. A cursor can bind the query, filters, index generation or short-lived result snapshot, and last sort position. Search-after avoids the cost and instability of deep offsets. The product must still decide how much freshness to trade for a coherent browsing session; “stable pagination” is not one universal mechanism.

Close the measurement loop without teaching the wrong lesson

The system should distinguish candidates retrieved, candidates ranked, candidates removed by each filter, results served, impressions actually rendered, and subsequent actions. That sequence reveals whether an empty page came from poor recall, over-filtering, a slow shard, or a serving defect. It also makes offline evaluation possible: a ranking change can be replayed against logged candidate sets without claiming that historical clicks reveal every result the user would have preferred.

Clicks are selective evidence. Position affects them. Familiar and sensational items attract them. A system trained only on its own prior exposure learns to repeat its earlier choices. Use negative actions, long-term outcomes, explicit quality judgments, and carefully bounded exploration where the product permits them. Compare experiments by segment and guardrail—freshness, empty pages, reports, hides, latency, concentration, and privacy failures—not only by one aggregate engagement number.

Operations follow the same stages. Watch canonical-to-index visibility, version lag, shard latency and errors, candidate depth, filtered-result rate, ranker and feature timeouts, cache age, pagination anomalies, and fallback use. Provide controlled repair: replay indexing work, rebuild a shard, invalidate a query cache, compare index generations, or trace one authorized result. A single “search latency” dashboard can be green while new documents remain invisible for hours.

Cost also attaches to stages. Cache hot public queries and safe prefix results; bound scatter, candidate depth, feature fetches, and reranking; tier cold index data; cap deep result windows; and sample expensive diagnostic detail after retaining enough evidence for failures. The cheapest system that returns the wrong audience’s documents is not a successful optimization.

Bend the retrieval path without breaking it

The document query supplies a set of questions for neighboring prompt families: where candidates come from, how fresh they are, which filters are non-negotiable, what happens when ranking fails, and what evidence closes the loop. The answers change with the product.

Feed: move fan-out out of the ranker

A feed’s user has expressed less intent than Nadia did, so follows, recent posts, topic pools, and recommendation sources do more candidate work. For ordinary producers, asynchronously placing post ids into follower timelines can keep reads cheap. Pushing a celebrity post into millions of timelines can overload the write path, so high-follower producers are often merged at read time. The hybrid threshold should follow the follower distribution and the freshness budget, not folklore about a particular service.

A timeline entry remains provisional. Blocks, private-account changes, deletes, moderation, seen-item rules, and availability may change after fan-out. Recheck urgent state while serving, retrieve deeper when safe filters remove many candidates, and keep repair jobs for lagging or missing timeline entries. A short-lived browsing snapshot or timeline cursor can prevent a rapidly changing ranker from making every next page repeat the previous one.

Autocomplete: spend almost no time and leak nothing

Autocomplete serves before the user has finished expressing intent. Normalize case, accents, language, locale, token boundaries, and entity type, then query a compact prefix structure or finite-state representation. Keep popular safe prefixes close to the serving edge and add a small fresh overlay when inventory, news, or newly created entities genuinely need to appear quickly.

The tiny latency budget does not relax correctness. Private document titles, tenant entities, blocked people, unsafe phrases, and query spam can leak through suggestions before a full search is submitted. Filter what enters the prefix index, scope personalized sources, and enforce the serving boundary again. A safe empty suggestion list is a valid degradation.

Recommendation: make exposure part of the data

A recommendation surface may blend popular items, similar items, followed authors, regional inventory, fresh content, and editorial pools. Cap and label sources, deduplicate them, rank a bounded union, then apply availability, permission, safety, seen-item, repetition, and diversity rules. If online features or the ranker fail, fall back to safe cached results, simple similarity, or popular-by-segment pools without dropping those rules.

Cold start and feedback loops are the structural difficulty. New users need contextual, popular, or explicitly chosen seeds. New items need a chance to be seen before interaction data exists. Record impressions as well as actions, allow bounded exploration, and watch whether the system concentrates exposure so strongly that its own choices become the only evidence it can learn from.

Trending: rank departure from a baseline

Trending is not a lifetime popularity list. Aggregate recent events within declared windows and dimensions such as region, language, tenant, or topic. Compare activity with a suitable historical baseline, decay old evidence, and cache the shared result page. A sudden change can then outrank a permanently popular item without pretending that every small-region fluctuation is a trend.

Manipulation attacks candidate generation itself. Deduplicate actors, limit the influence of repeated events, compare referrers and cohorts, filter spam and unsafe content, and quarantine suspicious spikes for review. Operations should be able to explain whether an item rose through many independent actors, one large referral source, a news event, or coordinated behavior.

Rehearse the missing result

Give a twenty-minute design beginning with Nadia’s save and query. Do not draw a model service until the candidate path requires one. Follow the document through canonical commit, index work, shard retrieval, permission filtering, ranking, final checks, pagination, and the impression log. At each boundary, say what evidence distinguishes “not yet indexed,” “not retrieved,” “ranked below the cutoff,” and “correctly filtered.”

Then change one condition:

• a permission revocation must take effect within seconds;
• one tenant contains half the corpus and most of the query traffic;
• the semantic candidate source exceeds the page deadline;
• an analyzer migration requires a full reindex without downtime;
• the same retrieval service must power a feed rather than explicit search;
• coordinated actors are manufacturing a regional trend.

Carry that change through candidate generation, ranking, filtering, serving, measurement, repair, and cost. If the answer only adds a component, the consequence has not traveled far enough.

The query is resolved when the design can say where Nadia’s document stopped and why. Perhaps version 37 is still waiting for index refresh. Perhaps the candidate was retrieved but a stale ACL summary rejected it. Perhaps lexical fallback ranked it correctly after the learned service timed out. Those are facts an operator can distinguish and a product can explain. Once the stages are visible, feeds, search, autocomplete, recommendations, and trending stop being five diagrams and become variations of one disciplined act of selection.

Related links

• Caching and Content Delivery
• Queues, Logs, and Stream Processing
• Search and Retrieval Systems
• Security and Privacy
• Observability, Performance, and Cost
• Scaling and Partitioning
• Content and Storage Systems