Low-Latency Request/Response Services

A mechanism can be valid in isolation and still compose badly with every mechanism around it. The archetype-scale question is harder: which mechanisms belong together for a particular class of work, and where do their contracts meet?

A low-latency request/response service is not “an API behind a load balancer.” It is a finite synchronous path from a caller’s decision to a useful, correctly interpreted response. Its behavior emerges from edge queues, connection state, compute scheduling, dependency fan-out, state freshness, retries, deployment state, and the slowest required branch. An individually sensible timeout, cache, retry, pool, and autoscaler can compose into an unbounded path.

The governing rule is:

Keep the synchronous critical path shallow, bounded, cancellable, and observable. Move nonessential work out of it only with explicit completion semantics.

At this scale, one cache or scheduler is no longer the unit of judgment. A complete service must remain explainable under skew, overload, dependency failure, cold deployment, and growth.

Start with endpoint cards, not an average API

“Twelve thousand requests per second” describes traffic volume, not workload. A search request, a product-detail read, and a stock reservation can have different payloads, state paths, service demands, objectives, and failure semantics while sharing a hostname.

Mercury API records one card per endpoint and client mode:

The population and mode qualify each objective. Mobile clients across a high-round-trip geography do not share the same transport budget as a colocated service client. A 150 ms successful-response p99 says nothing about rejected, cancelled, or timed-out attempts unless those outcome classes are also measured.

Client behavior is part of demand. Type-ahead requests are superseded; cancellation can save work. A reservation attempt remains relevant after a screen transition if its outcome may have committed. Treating both as disposable reads either wastes search work or loses payment-like outcomes.

For capacity, record demand by endpoint and state, not just count:

These are modeled means for the teaching fixture, not measurements. A real card carries distributions by payload class, tenant, geography, cache state, deployment generation, and outcome.

Three viable shapes—and their rejection criteria

An archetype is a family, not one diagram. Three common shapes can satisfy a low-latency service contract.

Region-local modular path

The edge routes to a regional stateless handler. The handler issues a small number of bounded parallel calls to authoritative or declared-fresh dependencies and aggregates them. This works when ownership boundaries matter and the required fan-out stays shallow.

Reject it when the endpoint must synchronously compose dozens of remote owners, when correlated branch tails consume the objective, or when partial results have no honest product meaning. Adding parallelism cannot repair an unbounded number of required branches.

Read-optimized projection path

The handler reads one versioned projection shaped for the endpoint. Authority updates the projection asynchronously; the response exposes a version or as-of boundary and can fall back to authority for selected read-after-write cases. Coalescing absorbs concurrent cold misses.

This is appropriate when read volume dominates, a freshness contract is meaningful, and rebuild/reconciliation are funded. Reject it for an admission decision that requires current authoritative stock, authorization, or money movement. “Fast local read” does not grant correctness authority.

Cell-local isolated path

Routing, compute, cache, and state ownership align within a bounded cell or tenant shard. The design reduces cross-cell fan-out and blast radius, at the cost of placement constraints, imbalance, cell movement, and duplicated operating capacity.

Use it when isolation and predictable locality justify those costs. Reject it when requests routinely cross most cells or when the routing key is unknown until after expensive global work.

The choice is conditional:

Terminate the edge without moving the mystery

Edge termination owns TLS, protocol negotiation, request-size enforcement, authentication handoff, coarse admission, routing, and connection lifecycle. It should remove repeated work from the application path, not hide queueing in an opaque proxy.

Connection reuse avoids repeated handshakes and congestion warm-up, but a connection is state with lifetime, capacity, and failure behavior. Too few connections serialize work. Too many increase memory, handshake bursts, file descriptors, load-balancer state, and uneven placement. Long-lived connections can pin clients to old instances or old routing decisions.

HTTP/2 multiplexes exchanges as streams on a connection, but it retains connection-level contention and flow control. It does not make downstream pools unlimited or eliminate transport loss effects. HTTP/3 changes transport behavior but does not erase application queues, dependency fan-out, or server work. Choose a protocol from observed request/response size, concurrency, loss, intermediaries, connection churn, client support, and operational visibility—not from a promise that “multiplexing lowers latency.”

Routing needs a stability contract. Per-request balancing can spread stateless reads but harm cache locality. Affinity can preserve locality but create skew and complicate draining. Key-aware routing can align with state ownership, provided stale routes are fenced or redirected rather than allowed to reach two authorities.

At every hop, expose connection setup, reuse, active streams, flow-control stalls, resets, queue age, request size rejection, and route decision. A trace that begins after the edge has already queued for 70 ms is not an end-to-end trace.

“Stateless compute” still carries request state

A stateless handler means durable business authority is outside the process. The process still carries request context:

• stable logical-operation and attempt identities;
• caller deadline and cancellation signal;
• authentication and tenant context;
• trace and exemplar identity;
• authoritative or projection version requirements;
• degradation mode; and
• resource accounting class.

Propagate only what downstream work needs. A context object that grows without ownership becomes payload and coupling. An omitted deadline makes a descendant outlive the caller. An omitted tenant class bypasses admission. An omitted version lets a projection answer a read-after-write request with older state.

Place state according to the decision. A detail page may use a regional projection for descriptive catalog text, require a versioned price, display bounded-stale availability, and send the actual reservation to stock authority. One response can legitimately compose several state classes, but each field needs a declared source and stale behavior.

Avoid synchronous remote joins disguised as service purity. If one endpoint retrieves 25 rows and calls four owners per row, its logical elegance has created up to 100 remote opportunities for queueing, failure, and tail amplification. Batch by owner, materialize a projection, change the response shape, or admit that the objective is not feasible.

Spend 150 milliseconds once

Mercury’s detail endpoint has a modeled 150 ms p99 objective for successful normal-mode requests. The architecture allocates the budget before measuring a candidate implementation:

The required price and stock branches are modeled at 38 ms and 44 ms; recommendations are optional and cut off at 30 ms. Because the branches run in parallel, the observed dependency contribution is 44 ms, not 38 + 44 + 30. The modeled path is:

18 + 7 + 10 + max(38, 44, 30) + 12 + 13 = 104 ms
150 - 104 = 46 ms modeled margin

The allocation still reserves up to 55 ms for the branch envelope and 35 ms for uncertainty. Do not convert this arithmetic into a statistical claim. The sum of stage p99 values is not the end-to-end p99: samples differ, branches correlate, and the maximum of parallel work has its own distribution. Observe the whole request population and use correlated stage spans to explain it.

Deadline ownership runs from caller to every descendant. Each hop subtracts elapsed time and a return reserve before starting new work. A timeout bounds one wait; a deadline bounds the logical operation. When the request is cancelled, the handler must stop local work, release permits, cancel outgoing calls where safe, and prevent a late optional result from mutating the response.

Cancellation is cooperative in many application runtimes. A library signal does not necessarily interrupt a CPU loop, database query, queued task, or side effect. Validate cancellation at each boundary and count work completed after caller abandonment.

Bound concurrency where work becomes scarce

An endpoint limit, worker limit, database pool, and dependency limit protect different resources. One global semaphore cannot express all four.

Use Little’s Law as a consistency check. At 12,000 requests/s and 150 ms average residence—not p99—the system would hold about 12,000/s × 0.150 s = 1,800 requests in flight. That calculation does not set a safe limit because the mean, mix, and resource demands matter. It does expose impossible dashboards: a reported steady 12,000/s, 150 ms mean, and only 200 total in-flight requests do not describe the same boundary.

Place admission before the scarce queue. A bounded worker queue does not protect a database if every admitted handler immediately waits for a connection. Give the detail and reservation paths separate concurrency budgets when their cost and importance differ. Queue age is often more decisive than queue length because a request can already lack enough remaining deadline to finish.

Low CPU, rising latency

Mercury shows 38% application CPU while latency climbs with concurrency in a newly introduced uncached detail cohort. Unlike the all-detail average in the endpoint card, this incident cohort issues two database operations per request. The database pool has 48 connections. A database operation holds a connection for a modeled mean of 18 ms:

pool service capacity = 48 × (1,000 ms/s / 18 ms/op)
                      ≈ 2,667 operations/s

arrival demand = 1,400 requests/s × 2 operations/request
               = 2,800 operations/s

modeled utilization = 2,800 / 2,667 = 1.05

At constant assumptions, the connection queue cannot drain. Application CPU can remain low because handlers are waiting. Evidence should show pool wait time and age, checked-out connections, database service time, queries per logical request, transaction duration, and deadline remaining at checkout. Raising the pool may simply move contention into the database; it is a hypothesis to test, not the repair.

Possible repairs are to reduce operations per request, shorten transaction/connection hold time, cache a safe read, combine queries, reject earlier, or add proven database capacity. The fixture at examples/performance-engineering-system-design-handbook/part-05/request-response/ reproduces the budget, endpoint demand, and pool bound.

Thread-pool exhaustion has a similar signature when blocking work occupies every worker. Separate CPU-ready time from blocking time, keep blocking pools bounded, and ensure health/control work cannot be starved behind user work.

Cache as a response contract

A cache belongs on the path only when its key, authority, freshness, invalidation, miss behavior, and failure mode are explicit.

For each response field, decide:

1. Which version or as-of point does the entry represent?
2. Which request dimensions affect correctness—tenant, authorization, locale, experiment, representation, and schema?
3. May stale data be served in normal, dependency-impaired, or overload mode?
4. Can a caller require a minimum version after a write?
5. What happens when the cache is empty, slow, corrupt, or partitioned?

Request coalescing lets concurrent misses share one fill, but the leader needs a deadline and failure policy. Followers must not wait beyond their own deadlines. A failed leader should release or transfer ownership without starting a synchronized second wave. Add randomized expiry, refresh-ahead under admission, and a cap on concurrent fills per key and dependency.

The familiar rule “a cache reduces latency” fails when the cold path is much slower and cache misses synchronize. A 99% hit ratio can still dominate p99 for a hot key after a deployment flush. Validate warm, cold, expired, invalidated, and dependency-failed states separately.

Degradation is a ladder, not improvisation

Mercury declares modes before an incident:

An optional dependency must be optional in code, product meaning, and operations. If its absence causes a nil dereference, changes authorization, or makes the response misleading, the architecture cannot call it optional.

Isolation assigns separate concurrency, connection, retry, and circuit state where one dependency could consume another’s budget. A circuit breaker without admission can produce synchronized probes. A fallback that calls a second slow dependency can increase work during failure. Test the whole degradation transition, including recovery and stale-state convergence.

Reads, writes, retries, and completion

Retry-safe reads are side-effect free at the declared boundary and tolerate repeated observation. That does not make every GET safe: a handler that mutates session state, charges quota, or starts a workflow has hidden effects.

Writes need stable logical-operation identity, canonical request comparison, durable outcome state, and replay behavior. A client retry after an ambiguous reservation response must learn whether the original operation committed; a fresh attempt can oversell or duplicate an effect. Keep attempt identity distinct so traces show one logical operation with several network attempts.

Retry budgets limit additional work by endpoint, dependency, outcome, and remaining deadline. Retry only failures likely to be transient and only when enough time remains for a useful response. Jitter spreads eligible retries, but it does not repair overload. Hedging a read consumes extra capacity and can worsen tails when the system is saturated; gate it on measured benefit, idempotence, and an explicit duplicate-work ceiling.

Moving audit emission, analytics, email, or index updates out of the request path can be correct. The acknowledgment must then say what completed, the asynchronous record must be durable enough for its promise, and operators need lag, retry, poison, and reconciliation evidence. “Fire and forget” is an unnamed loss policy.

Scale the resource vector, not the request counter

Using the endpoint card means Mercury’s 12,000 requests/s consume:

detail:  5,000/s × 1.7 CPU-ms = 8,500 CPU-ms/s
search:  6,000/s × 0.9 CPU-ms = 5,400 CPU-ms/s
reserve: 1,000/s × 3.1 CPU-ms = 3,100 CPU-ms/s
total:                              17,000 CPU-ms/s

The fleet’s protected operating envelope is 15,600 CPU-ms/s, so the modeled mix exceeds it by 1,400 CPU-ms/s even though “12,000 requests/s” may have been safe under a cheaper mix. CPU is only one dimension. Repeat the calculation for database operations, bytes, cache fills, locks, connections, memory residency, and dependency quotas.

Autoscaling should consume endpoint-weighted demand and leading queue/concurrency evidence, then respect provisioning delay, warm-up, and state movement. Scaling on aggregate CPU alone will not repair a fixed database pool or a hot authority key. Scaling on request count alone can scale down during a shift toward expensive reservations.

Keep failure reserve explicit. Normal capacity, one-instance loss, one-zone loss, deployment overlap, and dependency impairment are different modes. “Autoscaling will add capacity” is not a substitute when queue tolerance is 150 ms and instances need minutes to become warm.

Deployments change the path

A new instance is routable before it is necessarily ready for the latency population. It may lack compiled code, connection pools, caches, routing tables, certificates, projection generations, and runtime profiles.

Warm-up should use bounded representative work, not an uncontrolled thundering herd. Readiness gates verify required state, connection health, handler capacity, and a small latency/correctness sample. A canary compares endpoint and state cohorts, not only aggregate error rate.

Draining reverses admission before termination:

1. stop new connections or streams according to protocol;
2. remove the instance from new routing;
3. allow bounded in-flight work to finish within its original deadline;
4. cancel safe abandoned reads;
5. reconcile ambiguous writes by operation identity;
6. close pools without synchronized reconnect elsewhere; and
7. terminate only after a declared drain ceiling.

Long-lived streams, background fills, and stuck transactions need separate ownership. A deployment that kills them at the generic process timeout can turn routine rollout into retry and reconnect storms.

Observe the endpoint and the logical operation

A useful endpoint record joins:

endpoint and client/geography class
logical operation and attempt identity
deployment generation and degradation mode
payload and response-size class
cache/projection version state
admission, queue, service, dependency, join, and egress times
deadline at entry and remaining at each child call
resource demand and pool waits
outcome: success, degraded, rejected, cancelled, timeout, ambiguous
trace exemplar linked to the same population

Histograms are split by endpoint and mode before aggregation. Exemplars link a tail bucket to a trace with queue spans, connection waits, branch overlap, cancellation propagation, and response version. Profiles answer whether service time is CPU, allocation, lock, I/O, or runtime delay. Dependency dashboards must not substitute for caller-observed latency.

Avoid unbounded metric cardinality: stable endpoint templates and declared cohorts belong in metrics; request IDs, raw URLs, and user identities belong in sampled traces or logs with appropriate controls.

Validation campaign

A low-latency design is incomplete until it predicts and tests states beyond a warm steady run.

Run correctness assertions with the load: response versions, stock behavior, idempotent outcomes, and authorization boundaries. A fast incorrect response is a failed test.

Do not use this archetype when

• The useful result is a long-running job whose honest contract is accepted, durable, progressing, and retrievable rather than synchronously complete.
• The workload is a continuous stream where event-time, backlog, checkpoint, and replay semantics dominate a caller deadline.
• The operation’s central problem is a multi-record invariant and durable commit; use the transactional archetype and treat latency within that boundary.
• The response requires an unbounded query or analytical scan; make it asynchronous, constrain it, or use an analytical archetype.
• Intermittent clients need offline ownership and later reconciliation rather than a continuously available synchronous authority.

Calling any of those systems “an API” does not make request/response their governing architecture.

Reference low-latency service checklist

• Is each endpoint population defined by mix, payload, geography, client behavior, state, and outcome?
• Is there one end-to-end objective and one budget owner, with reserve protected?
• Are required and optional branches explicit, shallow, and bounded?
• Do deadline and cancellation propagate to local work, pools, and descendants?
• Is admission placed before every scarce queue, with queue age visible?
• Are cache keys, versions, stale behavior, coalescing, and leader failure specified?
• Are write retries tied to a durable logical-operation identity?
• Does degradation preserve correctness invariants and declare recovery gates?
• Is capacity modeled by endpoint service demand across all dominant resources?
• Are cold start, warm-up, connection churn, and drain tested under load?
• Can an endpoint tail exemplar expose edge time, pool wait, branch overlap, mode, and version?
• Does the asynchronous work moved off-path have completion, lag, poison, and reconciliation semantics?

Design and diagnostic work

Field design — defend 150 ms. Reproduce the budget and 104 ms modeled path. Add 18 ms to the stock branch and 9 ms to edge transport. Decide whether to spend reserve, change the response contract, relocate state, or reject the objective. Produce three alternatives and state the evidence that would distinguish them.

Field diagnosis — low CPU, high concurrency. Start with the 48-connection, 18 ms/operation, 2,800 operations/s packet. Rank connection starvation, database slowdown, thread starvation, and downstream flow control. Request one discriminating observation for each. Explain why increasing the pool may worsen the true constraint.

Principal red team — synchronized cold path. Assume a regional cache generation expires during a rollout while price is degraded and clients retry. Specify fill ownership, admission, stale rules, retry budget, deployment halt, and recovery hysteresis. Reject any answer that lets optional enrichment weaken price or stock semantics.

Durable rules for a bounded request path

1. Model endpoints and client behavior separately; aggregate request count is not workload.
2. Allocate one end-to-end budget and observe its joint distribution rather than adding stage percentiles.
3. Keep required fan-out small; parallel branches still amplify tail and resource demand.
4. Bound queues at the scarce resource and reject work that cannot finish within its remaining deadline.
5. Propagate cancellation, then prove that application work and descendants actually stop.
6. Treat caches and projections as versioned response contracts, not transparent speed layers.
7. Degrade optional value through declared modes; never degrade an invariant by accident.
8. Make write retries replay a logical outcome, not create a fresh effect.
9. Scale endpoint service-demand vectors and preserve failure/warm-up reserve.
10. Validate cold, skewed, overloaded, failed, recovering, deploying, and draining states.

The synchronous path is now bounded. The next archetype changes the center of gravity: Ledgerline’s reservation path is not primarily a fan-out problem. Its architecture must serialize or safely decompose a business invariant, produce a durable acknowledgment, and remain operable under contention, plan change, migration, and restore.

Evidence and transfer limits

• RFC 9113 specifies HTTP/2 streams, multiplexing, flow control, and connection-level interaction. It supports the protocol discussion, not a claim that HTTP/2 will improve a particular endpoint; intermediaries, implementations, payloads, loss, and workloads require measurement.
• The current gRPC guides for deadlines and cancellation explain deadline propagation and cooperative cancellation behavior in that ecosystem. Language support differs, and these guides do not prove Mercury’s application work, queries, or side effects stop.
• Dean and Barroso’s The Tail at Scale establishes how component variability and fan-out influence large online-service tails and surveys mitigation techniques. Its reported systems are not sizing evidence for Mercury.
• The deterministic fixture at examples/performance-engineering-system-design-handbook/part-05/request-response/ verifies the teaching arithmetic. It assumes fixed mean service times, perfect branch parallelism, constant mix, no skew, and no retry, scheduling, runtime, network, or failure overhead. Replace it with correlated endpoint traces and measured service-demand distributions before making a production decision.