Resilience Under Overload and Recovery

A resilient service does not promise to behave normally at every load. It specifies how behavior changes when demand exceeds capacity.

That specification needs more than a threshold and an autoscaler. It needs a finite set of operating states, evidence for entering and leaving each state, automatic and manual actions, invariants that survive degradation, and a recovery rate that cannot consume the capacity just restored. Without those elements, overload policy is whatever a retry loop, unbounded queue, health check, or hurried operator happens to choose.

Use six states as the control surface:

These are not severity labels. They are states in which different actions are legal. A service can be degraded but safe; it can return fewer optional fields while completing nearly every core operation correctly and on time. It can also look lightly utilized yet be unsafe because the admission controller cannot reach its policy store or the authoritative write path has lost fencing.

The operating rule is: optimize overload for completed, valuable work and controlled recovery—not for accepting every request or keeping every feature enabled.

Make transitions depend on outcomes and mechanisms

CPU at 85% is not a universal transition. Saturation may appear first as lock wait, storage latency, connection acquisition, queue age, memory reclaim, a downstream quota, or a serialized authority. A useful transition combines user outcomes with the mechanism that threatens them.

For Mercury’s customer-read API, a state record might say:

Unit of demand: one logical customer read; attempts reported separately
Core invariant: authoritative customer data is correct
Objective: correct response within 750 ms for the declared population
Optional capability: profile enrichment, explicitly labeled when omitted
Normal reserve: survive the tested cell or dependency-loss condition
Elevated: queue age or reserve burns for 3 windows; defer exports/prefetch
Saturated: admitted work cannot finish inside remaining deadlines
Degraded: enrichment omitted; standard admission bounded; priority preserved
Recovering: initiating fault clear; queue slope negative; replay capped
Unsafe: authority, control, or correctness signal untrusted

Use hysteresis. Entering elevated after three bad one-minute windows and leaving after one good window creates a flapping control. The reverse transition should normally require stronger or longer evidence: stable goodput, negative backlog slope, recovered dependency margin, and a hold long enough to include the mechanism’s delay. Thresholds also need a maximum dwell. A service that stays elevated for six hours has normalized an incident, not recovered.

Automatic actions must be bounded. “Shed 10%” needs a population, priority policy, earliest enforcement point, maximum duration, and rollback condition. Manual actions need one authority and an expiry. An operator override that disables admission indefinitely is a latent outage. Record who set it, why, its maximum lifetime, which invariant it may relax, and how the system returns to automatic control.

The unsafe state is deliberately different. Do not automate a blind return from uncertain authority or observability. If the policy store is stale, the dependency health view is missing, or writes may reach two authorities, the system cannot infer safety from low CPU. Preserve the smallest correctness-safe surface and require an owned decision.

Count goodput, not activity

Throughput counts completed work. Goodput counts work that is correct, valuable for the declared class, and complete within its objective. During overload, the difference is the operating signal.

For a window of duration T:

G = N_correct,valuable,within-deadline / T

Report the denominator and exclusions. A fast rejection may be operationally correct but is not completed business work. A response after the caller’s 750 ms deadline consumes resources without contributing deadline goodput. A successful optional refresh is less valuable than an authoritative read during a core-only state. Attempt rate is not logical demand when retries exist.

The simulated Mercury packet begins with 10,500 logical arrivals/s and 14,700 observed attempts/s, an amplification of 1.4. The degraded state admits 8,000/s: 2,400 priority and 5,600 standard, with optional work at zero. At 99.8% correctness and 98.5% deadline success among admitted work:

G = 8,000 × 0.998 × 0.985 = 7,864.24 outcomes/s

Admission and retry suppression remove 6,700 attempted operations/s from the constrained path. The accepted count fell; valuable completion rose relative to an uncontrolled collapse. A dashboard that celebrates 14,700 attempts/s and condemns 2,500 early rejections would reward the feedback loop.

Pair goodput with deadline-miss outcomes, rejection reason, queue age by class, accepted-versus-original demand, retry amplification, and the constrained resource. A single aggregate can hide a tenant or priority class that has lost all service.

Admit work before it becomes expensive

Admission is a promise that the system has a credible path to completion. Enforce it at the earliest point that knows priority, remaining deadline, cost, tenant policy, and current capacity. A proxy that can count requests but cannot distinguish a 2 ms cached read from a 400 ms fan-out cannot make the entire decision alone.

Bound at least three things:

1. Concurrency: limits in-flight claims on threads, connections, memory, accelerators, locks, and downstream calls.
2. Queue occupancy and age: prevents accepted work from waiting past the point at which completion can still be valuable.
3. Arrival share: prevents one tenant, operation, or retry class from consuming the whole admission surface.

Queue limits follow deadlines and service demand, not a round number of messages. If the oldest queued item has 80 ms remaining and modeled service plus downstream time is 130 ms, starting it wastes scarce capacity. Reject or expire it before performing the work, and return an outcome that tells a well-behaved caller whether and when retry is meaningful.

Priority is a resource policy, not a label on a request. Reserve some capacity for critical work; otherwise a high-priority request arriving behind a full low-priority queue still waits. Prevent starvation with explicit minimum shares, maximum bursts, and fairness across tenants. Authentication, authorization, quota, and cheap validation should occur before expensive fan-out, while ensuring the admission gate itself cannot be exhausted by invalid traffic.

Do not move the queue and call it bounded. A rejected frontend request that every client writes to a durable retry topic creates a larger recovery problem. Count upstream buffers, client offline queues, broker partitions, scheduled retries, repair streams, and operator replay files in the same workload inventory.

Degrade capability without degrading truth

A degradation ladder orders optional work by removable cost while preserving named invariants.

The ladder is only safe if the response contract distinguishes optional from required. Skipping authorization, fraud controls, durability, fencing, or mandatory audit is not graceful degradation. Nor is serving stale state without an age bound when correctness depends on freshness. A lower model quality can be a valid rung only if the product has an agreed floor, affected populations are visible, and the lower result does not silently change a safety or compliance decision.

Choose rungs by work removed at the actual constraint. Removing a cheap frontend decoration does not protect a saturated database. Disabling a read cache during database overload makes the constraint worse. Record the expected savings in CPU-seconds, bytes, I/O, connections, queue slots, or downstream calls per logical operation, then verify after the transition.

The order can be dynamic, but dynamic does not mean improvised. A brownout controller can select a rung from goodput, queue age, dependency saturation, and reserve. Give it hysteresis, rate limits, a minimum hold, a maximum degradation duration, and a fail-safe state. Avoid synchronized control: if every cell crosses the same threshold and refreshes the same cache at once, the controller creates a correlated pulse.

Operators need override authority because models are incomplete. They should be able to hold a rung, disable a faulty automatic transition, or isolate one tenant. The override must be narrower than “force healthy,” must not falsify telemetry, and must expire or hand back through a reviewed transition.

Isolation decides the blast radius

Bulkheads reserve finite resources for classes that must fail independently. Cells make a larger slice of the stack—compute, queues, state paths, and control dependencies—independent enough that overload in one slice does not consume all others. Tenant isolation prevents a hot or pathological tenant from turning shared efficiency into fleet-wide failure.

Isolation is real only at the constrained resource. Separate worker pools that share one exhausted database connection pool are not independent. Separate cells that share a global configuration service on every request share a failure path. Trace CPU, memory, queues, threads, connections, cache keys, storage partitions, coordination authorities, network, and operational controls.

Useful policies include:

• per-tenant concurrency and queue caps with a protected fleet floor;
• separate priority pools with controlled borrowing rather than permanent idle reservation;
• shuffle-sharded worker subsets so two tenants are unlikely to share the same complete failure set;
• cells with bounded state and deployment blast radius; and
• independent control capacity that remains usable while the data plane is saturated.

Borrowing improves utilization in normal operation. It also creates recall risk. Define how quickly borrowed capacity returns, which work is preemptible, and what happens to partially completed work. A priority pool that cannot reclaim resources until ten-minute jobs finish is not a short-term reserve.

Failover is another isolation test. Mercury’s modeled normal warm capacity is 12,000 operations/s; its reduced failover capacity is 8,400/s. The degraded state admits 8,000/s, leaving only 400/s—95.24% utilization of that reduced capacity. That reserve can support probes and variance, not backlog replay or optional restoration. Moving the original 10,500/s plus retries to the destination would collapse it. Before failover, recompute demand, degradation, destination warmth, shared dependencies, state authority, and the traffic-transfer rate.

AWS’s account of minimizing correlated failures describes cells, velocity controls, shuffle sharding, and jitter as ways to reduce shared-fate behavior. The mechanisms are transferable; Mercury’s cell size and reserves are not.

Suppress retries and protect dependencies

An overloaded dependency cannot distinguish “important retry” from more work unless the protocol tells it. Protect it at both ends:

• the server rejects before expensive work and returns a stable overload outcome;
• the caller limits attempts, uses a total deadline, and never starts an attempt that cannot finish;
• retries consume a bounded budget separate from new logical demand;
• exponential backoff includes jitter;
• only one appropriate layer owns retries; and
• admission uses dependency-specific concurrency, not only fleet CPU.

Three layers making four total attempts each can turn one logical operation into 64 lowest-layer attempts. The exact multiplier depends on when failures occur, but the structural danger is multiplication across layers. Measure original operations, attempts by layer, retry cause, and successful work recovered by retry. If a retry budget consumes capacity but recovers almost no goodput, suppress it.

Circuit breakers can stop calls to a dependency known to be failing, but a fleet-wide synchronized half-open probe can become a load spike. Bound and jitter probes, protect a small observation class, and close only after the dependency demonstrates useful capacity under representative work. A breaker also needs a correctness fallback: omission, stale-within-bound data, alternate authority, or explicit failure.

Google SRE’s cascading-failure guidance connects overload, load shedding, graceful degradation, and bounded randomized retries. Amazon’s load-shedding account emphasizes completing accepted work before callers time out. Both support the mechanism; neither chooses Mercury’s priorities or declares its enrichment optional.

Recovery is a new workload

When the initiating fault clears, the system is not normal. Deferred messages, retries, repair queues, cold caches, replica catch-up, compaction, state transfer, client reconnects, and missed schedules arrive together. Returning every switch to full can create a second collapse.

Budget recovery on the restored bottleneck:

R_replay <= C_restored − F − R_failure − W_cache − O_recovery

where all terms are in constrained-resource-equivalent operations per second:

• C_restored: verified restored capacity;
• F: current foreground work;
• R_failure: reserve for variance and another failure;
• W_cache: deliberate cache and connection warm-up; and
• O_recovery: replication, repair, compaction, validation, and other recovery work.

For the teaching packet, restored capacity is 15,000 operations/s, foreground is 9,000/s, required reserve is 1,500/s, and cache warm-up consumes 500 operation-equivalents/s. The analytical replay ceiling is 4,000/s. Mercury chooses 3,500/s, retaining a 500/s uncertainty margin.

For a 54-million-operation backlog:

T_drain = 54,000,000 / 3,500 = 15,428.57 seconds = 4.29 hours

That estimate is valid only while foreground demand, per-item cost, restored capacity, and retry/poison rates remain within the modeled envelope. Recalculate from remaining work and observed service demand. Pause replay when deadline goodput, queue age, replication lag, or reserve breaches its hold condition. Prioritize by expiry, correctness dependency, tenant fairness, and business value; FIFO is not automatically safe.

Cache recovery deserves its own limiter. The packet models 2.4 million keys at 20,000 fills/s, an ideal 120-second fill. That is not permission to launch every fill at once. Coalesce concurrent misses per key, jitter expirations, cap fills at the dependency, warm the high-value working set first, serve bounded stale values where correct, and prevent failed fills from becoming synchronized retries. A 120-second ideal rate can take longer under skew and foreground interference.

Advance from recovering to normal only after the system has held representative demand, replay is complete or durably governed, caches and pools are warm, replicas are caught up, deferred correctness work is reconciled, and the tested failure reserve has returned. “Backlog zero” can be a bad signal if workers discarded poison items or stopped accepting new ones.

Validate the envelope before it is needed

A load test that stops at the advertised rate does not validate overload. Increase demand beyond saturation and observe where goodput peaks, whether queues remain bounded, which rejection is cheap, and whether the control plane remains responsive. Then remove a dependency or cell, introduce latency and errors, reduce capacity, and verify the declared state transitions.

Chaos experiments and game days should test a hypothesis with bounded blast radius. Examples:

• one cell loses 30% capacity while tenant skew increases;
• enrichment latency rises beyond the core deadline budget;
• retry responses are delayed or malformed;
• the brownout controller receives stale telemetry;
• failover begins with cold caches and only 8,400/s destination capacity; or
• recovery replay meets a new foreground peak.

Define steady state in user outcomes and invariants, not “all pods running.” Name abort conditions, observation windows, control owners, and a rollback path for the experiment. Start where isolation is strongest, prove telemetry before fault injection, and include recovery. An experiment that injects failure and ends when traffic shifts has not tested the second-collapse risk.

No game day proves resilience for all workloads. It supplies observed evidence for one fault, population, environment, version, and interval. Retain the workload, actions, state transitions, raw signals, surprises, and resulting design change.

Overload readiness checklist

OVERLOAD AND RECOVERY READINESS
Capability, population, unit, deadline, correctness invariant: __________
Normal/elevated/saturated/degraded/recovering/unsafe evidence: __________
Goodput definition; attempts and rejections reported separately: _______
Constrained resources and per-operation service demand by class: ________
Admission point; concurrency/queue-age/tenant/priority limits: __________
Retry owner, total deadline, attempt cap, backoff, jitter, budget: _______
Degradation rungs; removed work; preserved invariant; user signal: ______
Bulkhead/cell/tenant boundaries at the actual constraints: _____________
Brownout inputs, hysteresis, rate, holds, fail-safe, override expiry: ____
Failover capacity after transferred demand, coldness, and shared deps: ___
Recovery inventory: replay/repair/cache/replica/compaction/reconnect: ____
Restored capacity - foreground - reserve - recovery work = replay cap: __
Replay priority, checkpoint, poison policy, pause and completion rules: __
Game-day hypothesis, blast radius, abort, evidence, recovery validation: _
Owners for automatic policy, manual override, and unsafe-state decision: _

The checklist is a review artifact, not a guarantee. Every filled line should point to a measurable policy, executable control, or named owner.

Applied work: define the states and the drain

Run the deterministic packet:

cd examples/performance-engineering-system-design-handbook/part-07/overload-recovery
node analyze.mjs
node verify.mjs

Then produce two artifacts.

First, complete the readiness checklist for Mercury. Define evidence and automatic/manual actions for all six states. Explain why 14,700 attempts/s is not demand, calculate the 1.4 amplification, derive 7,864.24 deadline-good outcomes/s, and show where 6,700 attempted operations/s disappear. Defend the 2,400/5,600/0 class allocation. Challenge it with a tenant that owns half the priority traffic and with loss of the policy store.

Second, write the recovery plan. Recalculate the 4,000/s analytical ceiling, choose the 3,500/s operational replay rate, and derive the 4.29-hour ideal drain. Specify pause conditions, replay ordering, checkpoints, poison handling, cache-fill coalescing, and the evidence required to restore optional enrichment. Explain why replay is forbidden while failover capacity is only 8,400/s and accepted core work is already 8,000/s.

A strong submission distinguishes automatic saturation control from incident command, preserves correctness on every degradation rung, keeps the control plane usable, and treats recovery estimates as conditional. It does not use a queue length, CPU threshold, successful-request latency, or backlog-zero signal alone.

Evidence and transfer limits

• Google SRE on cascading failures provides primary practitioner evidence for overload testing, graceful degradation, load shedding, and retry containment. Its examples do not establish Mercury’s thresholds.
• Amazon’s load-shedding article explains deadline-aware accepted work and goodput under overload. Product and client semantics still determine what may be rejected.
• Amazon’s correlated-failure article motivates cells, shuffle sharding, jitter, and operational velocity controls. Independence must be verified at this system’s bottlenecks.
• All Mercury numbers are simulated teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-07/overload-recovery/. Passing checks validates arithmetic and the declared state inventory, not a production controller or recovery guarantee.

The decision rule is: preserve correct, valuable, deadline-compliant work with bounded admission and declared degradation, then restore deferred work below verified spare capacity. Resilience consumes headroom, isolation, engineering time, and sometimes idle-looking capacity. The next decision is how to price those choices without optimizing away the safety they buy.