Case Study: Deadline-Aware Search and Autocomplete

A Wayfinder search request has 61 milliseconds left. Lexical retrieval has returned a credible candidate set. Three optional actions are ready:

• vector retrieval is predicted to need 38 ms at p95 and add 0.012 NDCG@10 on similar judged queries;
• neural reranking is predicted to need 46 ms and add 0.008;
• spell expansion is predicted to need 22 ms and add 0.003.

Which work should the scheduler start?

“Run all three and return the best result” is not available. The mobile deadline is 180 ms, the response still needs assembly and network reserve, and a stage that finishes after cancellation can consume capacity without improving the response. “Always run the highest-quality model” is incomplete for the same reason. Quality that misses the user’s decision window is not free quality.

For this request class, Wayfinder selects vector retrieval. It fits the remaining budget and has the highest modeled quality gain per millisecond. That is not a universal claim that vectors beat lexical retrieval, or that NDCG gain divides cleanly by latency. It is a local scheduling decision using a calibrated cost and expected-gain model. A different query, remaining deadline, or capacity state can select another action or stop immediately.

Wayfinder is a fictional service handling 55,000 queries/s: 70% autocomplete and 30% full search. It combines lexical retrieval, approximate vector retrieval, personalization, and a learned ranker. Every number in this case is modeled or simulated teaching evidence. It is not a benchmark of Elasticsearch, Lucene, a vector database, or a production ranking model.

The design lesson is: search performance is an allocation problem—spend scarce latency only where it is likely to improve result quality.

Define the outcome before budgeting stages

Search does not have one scalar objective. Wayfinder’s review ledger keeps four outcome families separate.

Full-search quality uses NDCG@10 on a frozen, versioned judged set for the fixture. That metric values ordering near the top, but it cannot establish business value, policy correctness, diversity, or freshness by itself. Autocomplete has a different unit: one prefix revision and its suggestion set. It measures accepted suggestion, reformulation, abandonment, unsafe suggestion rate, and keystroke-to-render latency. Mixing both products into one p99 or click rate would conceal their different deadlines and intent.

Wayfinder declares a 180 ms mobile edge-to-payload deadline for the modeled full-search class. The online gate uses p99 rather than only the median because multi-shard fan-out and shared dependencies amplify stragglers. The service also tracks p99.9 and timeout population, but the fixture’s release decision is expressed at p99 and timeout fraction.

Freshness is part of correctness, not a ranking feature that may always be traded away. The index target is five minutes for ordinary document changes. A legal block, safety restriction, or inventory authority can require a much newer version. Retrieval candidates carry policy and index generations so the merge layer can reject results older than the request’s minimum acceptable version. If no safe result remains, the response says so; it does not fill ten positions with stale documents to preserve a success counter.

The useful online denominator is eligible requests that receive a renderable, policy-correct response inside the product contract. A timeout, client abandonment before render, undeclared partial response, or late completion after cancellation is not goodput.

Close the 180-millisecond budget

The selected deadline budget is a reservation, not a promise that every stage consumes its whole allocation.

The allocations sum to 180 ms. That arithmetic is necessary and insufficient. Retrieval and ranking distributions are conditional on query class, cache state, shard fan-out, index generation, model version, and fleet utilization. The scheduler passes one absolute monotonic deadline. Each child derives its local stop time from the remaining budget; services do not reset a fresh 62 ms or 46 ms timeout at every retry.

The 18 ms reserve is real capacity, not a bucket that optional stages consume by default. A stage may start only if its predicted completion plus assembly and safety margin fits. If a request arrives at a service after its local useful-work cutoff, the service returns a bounded “not attempted” outcome rather than entering a queue.

The Tail at Scale explains why large fan-out systems must treat variability as an end-to-end concern: even uncommon component delays become common request delays when a response waits on many components. The paper discusses tail-tolerance mechanisms, but it does not choose Wayfinder’s deadline or justify unlimited duplicate work. See The Tail at Scale.

The scheduler spends expected gain, not stage prestige

At classification, Wayfinder assigns a bounded cost class and looks up calibrated distributions by query features and current fleet state. For each optional action i, it estimates:

value_i = expected marginal quality gain_i / expected cost_i

The implementation does not reduce every decision to that ratio. It first enforces correctness and policy constraints, then checks the absolute deadline, stage concurrency, and workload-class admission. Among feasible actions, expected marginal quality per millisecond is one ordering signal. Uncertainty can make a lower-mean but more predictable stage preferable near the deadline.

For the opening request:

The scheduler starts vector retrieval and propagates the same absolute deadline. It may still cancel if lexical confidence already crosses an early-exit threshold, a vector admission queue consumes the useful window, or the request disconnects. The expected gain estimates are trained and validated by query segment. New or rare queries fall back to conservative rules rather than trusting an extrapolated model.

The scheduler records every non-executed action with a reason: NOT_ELIGIBLE, NO_BUDGET, ADMISSION_REJECTED, EARLY_EXIT, or CANCELLED_BY_PARENT. Without those labels, operators cannot distinguish a fast intentional cascade from a failing vector service that silently contributes nothing.

The complete decision surface fits in one visual

The visual uses a coordinate-free frontier because the decision is multi-dimensional. Serial has the best offline NDCG in this fixture and misses the deadline. Speculative has the lowest p99 and the highest CPU cost. Cascade gives up 0.007 NDCG relative to serial, meets the tail objective with reserve, and uses the least CPU. No single left-to-right line can call one design globally best.

Four execution shapes move different costs

The architecture review runs all alternatives against the same query set, index snapshot, model versions, fleet class, warm-up rule, offered-load schedule, and correctness checks.

Serial execution

Serial runs lexical retrieval, then vector retrieval, then personalization, then the full ranker. It is easy to reason about because each stage sees the previous output. It also adds dependency delays on one critical path. The simulated p99 is 248 ms, NDCG@10 is 0.846, and CPU cost is 36 ms/query.

Serial is appropriate when each stage truly depends on the previous result, deadlines are loose, or later work is rare enough that a branch is cheaper than parallel fan-out. It is rejected for this mobile class because the p99 exceeds 180 ms. Raising the client timeout would move abandonment and capacity pressure rather than explain why every query deserves every stage.

Parallel retrieval

Parallel starts lexical and vector retrieval together, merges candidates, then personalizes and ranks. It removes serial wait but performs vector work even when lexical evidence would have permitted an early exit. The fixture produces p99 176 ms, NDCG 0.844, and 41 CPU-ms/query.

Parallel barely meets the deadline, leaving little room for distribution shift. A vector slowdown, cold embedding cache, or fan-out increase can cross the boundary. It remains useful for query classes where both candidate sources have high marginal recall and predictable cost.

Cascade

Cascade runs cheap retrieval first, evaluates confidence and query class, and invokes vector retrieval or expensive ranking only for requests expected to benefit. It prunes 1,200 lexical and 800 vector candidates to 500 merged identities, then caps neural reranking at 200. Simple queries can stop earlier.

The fixture produces p99 151 ms, NDCG 0.839, and 28 CPU-ms/query. Compared with serial, it improves p99 by 97 ms and saves 8 CPU-ms/query while losing 0.007 NDCG. At 55,000 queries/s, the arithmetic represents 440 CPU-seconds of modeled work avoided per wall-clock second. That is a capacity input, not a server-count claim; CPU-ms is not interchangeable across instruction sets, models, or utilization.

Cascade is selected because its quality loss is inside the declared 0.01 gate, it leaves tail reserve, and it reduces work. Its risk is classifier error: a query predicted to be easy may need vector recall or a stronger ranker. The rollout therefore segments quality by early-exit reason and audits false exits, not only aggregate NDCG.

Research on operational cascade ranking demonstrates the broader mechanism of progressively filtering candidates through ranking stages. Wayfinder’s data, models, and thresholds are fictional; the source supports cascade structure rather than these results. See Cascade Ranking for Operational E-commerce Search.

Speculative execution

Speculative starts a likely-needed expensive branch before the cheap branch has proved it necessary. The first valid result can win, and the loser is cancelled. The fixture reaches p99 139 ms and NDCG 0.842 but consumes 47 CPU-ms/query, the highest of the four.

Speculation is bounded to request classes whose tail value exceeds the extra capacity and whose cancellation actually releases work. It is not enabled globally at saturation: duplicating work when queues are longest can deepen the overload it was meant to mask. Wayfinder caps speculative population, includes both attempts in admission, and measures loser cancellation delay. A branch that “cancels” at the caller while continuing a 100 ms accelerator computation is still charged.

The chosen system can use all four shapes in different classes. The decision table is not a tournament that permanently eliminates mechanisms.

Stage evidence must explain the end-to-end tail

A 151 ms p99 is not explained by collecting seven stage p99 values. Percentiles from different requests do not add, parallel branches overlap, and queue time can correlate across services. Wayfinder keeps one sampled trace for each request with the absolute deadline, policy generation, query class, index generations, candidate counts, and stage outcomes.

For every stage, the trace separates:

• admission wait from executor queue wait;
• service time from downstream wait;
• fan-out width, first useful response, and last response used;
• candidates considered, pruned, merged, rejected for policy, and reranked;
• cache and model state without raw sensitive query labels;
• cancellation sent, cancellation observed, and resource release;
• predicted duration and gain from actual duration and judged contribution;
• completion that affected the response from late work that did not.

The critical-path view marks which span delayed assembly. A 70 ms vector span is not on the critical path when an early lexical result caused cancellation after 20 ms; it is still a capacity leak if the vector worker continued. Conversely, a 4 ms merge can be the decisive delay if it begins with only 4 ms of useful budget left. Stage dashboards report both causal latency contribution and consumed work.

Shard aggregation preserves distribution shape. The coordinator records the slowest used shard, the number cancelled, and the candidate contribution of each source. Metrics use bounded shard and query-cost classes rather than raw term or user identifiers. An access-controlled heavy-query report can retain sampled fingerprints for diagnosis, but the main metrics path does not turn arbitrary query text into labels.

The latency predictor is an operating model with calibration evidence. Wayfinder compares predicted p50/p95 and completion probability to observations by query class, fleet state, index generation, and time window. If a 38 ms vector prediction completes within 38 ms only 70% of the time rather than the intended 95%, the scheduler disables that prediction cell or adds conservative uncertainty. A stale predictor can be more dangerous than no predictor because it systematically launches work that cannot finish.

The quality attribution path is equally explicit. Offline replay records whether a candidate came from lexical, vector, both, or a fallback source; whether it survived freshness and policy; and how each optional stage changed the final judged metric. Marginal contribution is evaluated by counterfactual ablation on the same candidate set where valid. It is not inferred from the fact that a clicked item passed through a neural model.

Counterfactual tests keep the scheduler honest

Before rollout, Wayfinder runs a matrix that deliberately removes or slows components:

The generator is open loop: it follows a fixed arrival schedule even when the service slows. A closed-loop client that waits for each response would reduce offered load during failure and make the design appear stable. Correctness reconciliation runs beside load so a fast empty or policy-invalid result cannot pass.

The scheduler also runs shadow counterfactuals on a bounded sample. When cascade exits early, shadow branches may compute what vector retrieval or the full ranker would have changed. This sample is admitted as experimental work and disabled under pressure. It supplies false-exit evidence without making every production request pay the full cost forever.

Transfer limits are recorded. A judged corpus can age as inventory, language, and user intent change. Trace replay preserves old arrivals but may not reproduce cache and queue interactions. Shadow execution shares production state but does not observe how changed results affect users. Online comparison observes behavior but inherits presentation and feedback bias. The release decision needs agreement across these imperfect lenses, not one “ground truth” dashboard.

Query cost classes exist before fan-out

An adversarial or accidentally explosive query must be bounded before it reaches every shard. Wayfinder’s classifier uses syntax shape, token count, Boolean clauses, filters, wildcard or fuzzy expansion, requested fields, personalization availability, vector eligibility, tenant policy, and historical service demand. Raw query text is not placed in high-cardinality metrics.

The policy has three teaching classes:

The deterministic packet also declares absolute implementation-independent ceilings of 1,200 lexical candidates, 800 vector candidates, 500 merged candidates, 200 neural-rerank candidates, 24 accepted Boolean clauses for the modeled API, and 64 vector probes. The visual’s per-class values are tighter policy examples. The lower of endpoint, class, tenant, and current-capacity limits wins.

Approximate vector retrieval exposes an explicit quality/cost control. Elasticsearch’s current kNN reference, for example, documents that increasing num_candidates tends to improve accuracy while taking more time, and that candidate collection and merging occur across shards. Wayfinder is not specified as Elasticsearch, but the transfer is direct: candidate breadth is a budgeted control, not a magical “semantic search” switch. See Elasticsearch kNN search.

Lexical and vector retrieval are not caricatured as exact versus semantic. Lexical methods can model phrase, proximity, field, and learned signals; vector methods can retrieve spurious neighbors and miss policy-critical exact terms. The merge layer preserves source, rank, score calibration version, and reason so offline analysis can identify which branch contributed a useful candidate.

Abuse controls are part of capacity correctness. Anonymous clients receive small token buckets and stricter expansion limits. Authenticated tenants receive declared quotas and cost-weighted admission. Query templates or saved searches can have learned budgets, but a template change resets its evidence. A rejected query returns a stable reason and safe refinement guidance; the service does not queue unbounded work behind a friendly 200 response.

Partial results are a response type

Wayfinder can return a result when one optional branch misses its useful-work cutoff, but only under an explicit contract. The payload includes:

{
  "result_state": "PARTIAL",
  "completed_stages": ["lexical", "policy", "fallback_rank"],
  "omitted_stages": ["vector", "personalization"],
  "reason": "DEADLINE",
  "index_generation": 841,
  "policy_generation": 119,
  "retry_hint": "REFINE_OR_RETRY"
}

PARTIAL does not mean incorrect. Every returned item still satisfies eligibility and freshness. It means some quality-improving stages did not contribute. Policy enforcement, authorization, legal blocks, and minimum freshness are mandatory stages; if they cannot complete, the system fails explicitly rather than returning an unsafe “partial” list.

The selected policy returns declared partial results for 7.5% of simulated requests and marks them correctly in 99.95% of cases. The remaining 0.05% is a release blocker in a real review because an undeclared partial result contaminates quality analysis and user expectations. The fixture gate is 99.9% to demonstrate arithmetic, not to bless the residual defect.

Early exit is different. It stops because existing evidence is sufficient, not because the deadline defeated a planned stage. Thirty-one percent of requests exit early in the fixture. The threshold requires calibrated confidence, policy-complete candidates, enough result diversity, and a segment-specific offline bound. Operators track quality for early exits separately so a classifier shift cannot make latency improve by silently doing less useful work.

Fallback is also different. Twelve percent use a smaller, faster ranking model or a deterministic score composition when the preferred ranker is unavailable, not admitted, or lacks budget. The response records the ranker generation. The fallback must be trained and tested as an independent model; “old model” is not a safety argument if its features or index schema are incompatible.

Elasticsearch’s search API documents concrete timeout and early-termination controls, including per-shard timeout behavior and caution around terminate_after. Those settings do not create Wayfinder’s end-to-end partial-result contract. They illustrate why a coordinator must understand what stopped, what returned, and whether results are partial. See Elasticsearch search API.

Freshness and deadline interact at merge

The fastest candidate is not necessarily eligible. Lexical and vector indexes may advance at different rates. Personalization features may refer to a previous catalog generation. The merge stage receives version metadata and applies a minimum generation from the request’s policy context.

If vector retrieval returns generation 839 while the request requires 841, Wayfinder can discard vector candidates and return a declared lexical-only result if the lexical index and policy state are current. It cannot compare incomparable scores and call the merged list fresh. During an index rollout, old and new generations are not mixed unless the compatibility contract explicitly defines it.

Freshness telemetry includes age distributions by index family, generation divergence, documents rejected at merge, and queries degraded because one source was stale. A five-minute aggregate index-age SLO can hide a hot shard that stopped refreshing. The gate samples query paths and asserts that returned documents meet the request’s generation boundary.

Autocomplete has an even tighter state loop. Each new prefix cancels the previous request through the full call tree. Results carry prefix revision; the client renders only the latest revision. Without that identity, a fast response for “ca” can arrive after a slower response for “camera” and replace it with stale interaction state even though both server requests met their latency SLO.

The frontier must include abandonment and capacity

Offline quality alone selects serial execution: NDCG 0.846 is the highest fixture value. End-to-end product evidence changes the result.

The selected cascade reduces simulated abandonment from 9.2% to 7.1%, an improvement of 2.1 percentage points, while timeout fraction moves from 1.9% to 0.4%. These are fixed simulated outcomes. They do not prove that lower latency caused every abandonment movement; result composition, query mix, client rendering, and experiment assignment must be checked.

The frontier is segmented. A 0.007 aggregate NDCG loss may conceal a 0.04 loss for rare multilingual queries or accessibility-related intent. Wayfinder requires non-inferiority or an explicit product decision for protected and high-value segments, policy-sensitive queries, head and tail frequency buckets, new versus returning users, and low-connectivity clients.

Cost is reported per correct useful response as well as per request. A design that saves CPU by timing out more users can look efficient per attempt. A speculative design that improves tails but exhausts fleet reserve can degrade everyone during a burst. The release model replays offered load at steady, peak, one-shard-slow, cold-model, and dependency-recovery states.

Roll out quality online without trusting clicks blindly

The candidate first passes a frozen judged set, replayed production traces with privacy controls, policy/freshness checks, and open-loop load. Shadow execution then computes candidate rankings without serving them and records latency, candidates, score features, cancellations, and disagreement within a bounded sample.

Serving rollout advances through 1%, 5%, 10%, 25%, 50%, and 100% with hold periods. Each stage gates:

• p99 at or below 165 ms and timeout fraction at or below 0.7%;
• cancellation work and queue age returning to steady bounds;
• quality loss no worse than 0.01 NDCG@10 on the frozen set and no blocked segment regression;
• policy violation and stale-result counts at zero for mandatory rules;
• declared-partial correctness at or above 99.9%;
• CPU, accelerator, index I/O, and fan-out attempts inside capacity reserve;
• abandonment, reformulation, zero-result, and downstream conversion guardrails.

For a bounded eligible population, Wayfinder uses team-draft-style interleaving between baseline and candidate rankings. Items from both rankers are mixed, interactions assign comparative credit, and the experiment asks which ranker is preferred with fewer users than a conventional A/B test might require. The fixture observes candidate win fraction 0.523 with a modeled interval of 0.516–0.530.

Interleaving is not truth. Position bias, presentation changes, duplicate or near-duplicate items, novelty, query abandonment, and eligibility differences can distort credit. It cannot compare responses with incompatible layouts or policy states without additional design. The research literature describes interleaving as an online comparison method that mixes rankings and interprets feedback; Wayfinder adopts the method only inside those limits. See Generalized Team Draft Interleaving.

Online p99 in the fixture is 154 ms, timeout fraction 0.44%, and abandonment 7.0%, all inside gates. Advancement still waits for minimum sample and time coverage, including peak periods and index refresh. A positive aggregate preference does not override a tail or safety regression.

Rollback switches serving authority to the baseline ranker and scheduler policy. It does not delete candidate evidence. In-flight requests keep the policy generation they started with. Index or feature changes that are not backward compatible have their own migration and cannot be “rolled back” merely by changing a model pointer.

The compact decision record

Decision. Use an absolute 180 ms deadline; classify cost before fan-out; run a deadline-aware cascade by default; execute lexical and vector retrieval in parallel only for classes with demonstrated marginal recall; cap candidates and rerank work; preserve mandatory policy and freshness; and return typed early-exit, partial, fallback, rejection, or timeout outcomes.

Evidence. wayfinder-deadline-v1, 55,000 modeled queries/s, frozen judged set and index snapshot, fixed alternative observations, open-loop tail trials, cancellation and generation traces, shadow disagreements, and a guarded interleaved rollout. The deterministic packet establishes internal arithmetic, not search-engine performance.

Rejected as defaults. Serial full execution misses the mobile tail. Parallel execution has insufficient reserve and spends vector work on high-confidence queries. Global speculation consumes the most CPU and becomes unsafe near saturation. One undifferentiated timeout cannot express mandatory versus optional work or explain a partial response.

Rollback. Restore baseline scheduler and ranker authority while retaining request generation and response-state semantics. Stop new speculative or candidate-policy work first; allow in-flight requests to finish under their starting generation or cancel at their original deadline.

Revisit. Any segment loses more than its approved quality bound; p99 exceeds 165 ms during staged rollout; timeout exceeds 0.7%; partial declaration falls below 99.9%; cancellation release grows; index or policy generations diverge; CPU reserve falls below the failure target; query mix or model service demand moves by 15%; or the product changes its mobile deadline, freshness, or safety contract.

The record deliberately does not choose a search product, similarity metric, embedding model, or learning-to-rank algorithm. Those choices affect service demand and quality, but none removes the need for an absolute deadline, bounded fan-out, response semantics, and segmented evidence. A faster model can enlarge reserve; the scheduler should not immediately spend every saved millisecond on another stage. Some gain belongs to resilience under a cold cache, slow shard, deployment, or traffic shift.

The decision also separates optimization from product policy. If the business wants richer discovery, it must state which query classes may spend more time or cost and which quality evidence justifies that spend. If mobile abandonment makes 180 ms non-negotiable, the design may need better first-stage recall, narrower data placement, precomputation, or more reserved capacity. Relabeling a timeout as partial or lowering candidate counts without measuring segment quality does not satisfy either objective.

Finally, every request carries its policy generation into logs and response metadata. When outcomes move, reviewers can reconstruct which budget, caps, model, index, and fallback rules applied. Without that identity, an online metric combines several control systems and makes a regression impossible to attribute. Deadline awareness is therefore as much an evidence and versioning discipline as a scheduler feature.

Overload changes the allocation policy

At high utilization, stage latency predictions and queue waits change. If the scheduler responds by launching more speculative work, it creates a positive feedback loop. Wayfinder’s overload mode narrows work before queues become unbounded:

1. stop optional speculation and broad query expansion;
2. reduce candidate and rerank caps by class;
3. prefer calibrated early exit and fallback;
4. preserve mandatory policy and freshness checks;
5. reject expensive or low-priority queries at admission with bounded retry guidance;
6. cancel expired work through every dependency and verify resources release;
7. restore stages one at a time after queue age and useful completion recover.

The controller uses goodput: policy-correct responses delivered before deadline. Raw shard searches, candidate scores, and late ranker completions are work counters. If offered work rises while goodput falls, raising concurrency without a new service-demand model is not recovery.

Query classes receive reserved capacity. Autocomplete cannot consume the entire full-search fleet during a typing burst, and a small population of adversarial searches cannot evict navigational traffic. Tenant and anonymous quotas are cost-weighted so one 64-probe query is not priced as one simple prefix lookup.

The evidence packet fixes claims and gates

Run the packet:

cd examples/performance-engineering-system-design-handbook/part-08/deadline-aware-search-autocomplete
node analyze.mjs
node verify.mjs

It closes the 180 ms budget, derives 38,500 autocomplete and 16,500 full-search queries/s, ranks the 61 ms scheduler choices, identifies the alternatives that meet the deadline, and reproduces the 0.007 quality movement, 97 ms p99 improvement, 8 CPU-ms/query saving, 440 CPU-seconds/s modeled fleet movement, 2.1-point abandonment movement, and all rollout gates.

The packet does not execute retrieval, measure NDCG, fit a latency predictor, estimate confidence intervals from samples, or prove causality. Those values are declared fixture inputs. Production evidence requires a judged corpus, query and index snapshot, model artifacts, trace replay, fixed offered-load generation, stage and end-to-end distributions, correctness reconciliation, cancellation verification, and online experiment analysis.

Failure modes after selection

Design exercise: quality loss is concentrated

After rollout, aggregate NDCG loss remains 0.007 and p99 remains 154 ms. Multilingual discovery queries, 4% of traffic, lose 0.035 NDCG and show a 3.8-point reformulation increase. Their vector branch has p95 52 ms rather than 38 ms because embeddings and filters fan out to more shards. Navigational queries improve slightly.

Redesign the policy without discarding the deadline model. Specify class detection, stage allocation, capacity, online measurement, and what you will do if the multilingual class cannot meet both quality and 180 ms.

Answer guide

Aggregate acceptance must stop. The class violates the segmented quality obligation even though the fleet-wide value passes. A defensible design identifies multilingual discovery before or immediately after cheap retrieval using locale, script, language confidence, query history, and retrieval disagreement without logging raw sensitive text in general telemetry.

The class can receive a different policy: reserve vector capacity; start lexical and vector retrieval in parallel; reduce unrelated personalization; use a multilingual first-stage model; narrow shard routing; precompute or cache safe query representations; or allocate fewer rerank candidates so the 52 ms branch still leaves assembly reserve. Every change needs a class-specific quality/latency/cost frontier. Giving the class more time by silently spending the 18 ms reserve makes the tail objective fragile.

Capacity planning charges the 4% population at its higher fan-out and service demand. Reserved capacity must exist across embedding, vector shards, merge, and ranking. If parallel retrieval adds work, admission and overload behavior are re-tested; the system must not improve one segment by crossing the fleet knee.

Online evaluation stratifies the multilingual class and uses judgments from qualified raters plus guarded user signals. Interleaving can compare compatible rankings, but reformulation, abandonment, policy, and presentation remain guardrails. The rollout starts within the affected class and retains baseline authority elsewhere.

If no design meets both 180 ms and the minimum quality, the product must choose explicitly: a longer declared deadline for that interaction, a two-phase response that clearly updates results, a narrower supported query contract, or more capacity/model work. Returning predictably worse results and hiding them in the aggregate is not a performance solution.

Field review card

When designing deadline-aware search:

• Define quality, latency, freshness, correctness, abandonment, and cost separately.
• Give autocomplete and full search their own units and deadlines.
• Pass one absolute monotonic deadline through the call tree.
• Reserve assembly and tail time; do not spend the full budget on retrieval.
• Estimate marginal quality and cost by query class and current state.
• Record why optional work did not execute.
• Compare serial, parallel, cascade, and speculative shapes under one workload.
• Charge cancelled and losing speculative work to capacity.
• Bound candidates, clauses, probes, fan-out, concurrency, and tenant demand before shards.
• Treat lexical and vector retrieval as complementary mechanisms, not slogans.
• Make early exit, declared partial, fallback, rejection, and timeout distinct outcomes.
• Keep policy and minimum freshness mandatory in every response branch.
• Track index and model generations through merge and ranking.
• Evaluate a quality-latency-cost frontier by segment, not only in aggregate.
• Use offline judgments, trace replay, open-loop load, shadow evidence, and guarded online comparison.
• Stop rollout on tail, timeout, safety, segment-quality, or capacity regression even if clicks rise.
• Degrade optional quality work before mandatory correctness work under overload.
• State what modeled NDCG and latency values cannot transfer to another corpus or engine.

Wayfinder’s selected cascade is not “the fast architecture.” It is a policy for allocating a finite deadline under one workload, index, model set, and evidence boundary. The durable method is to make each stage earn its time, preserve explicit response semantics when it does not run, and verify quality and tails together as the system changes.

The next allocation question is larger than one request: when tenants and workload classes compete for CPU, memory, scan, spill, and queue time, which promise owns the scarce capacity? Chapter 71 moves the same discipline from stage budgets to multi-tenant scheduling and isolation.