Partial Failure, Tail Amplification, and Recovery

After Mercury loses one of three zones, its capacity dashboard looks reassuring:

two surviving zones × 5,000 safe work/s = 10,000 work/s
foreground demand                         =  9,000 work/s
reported spare capacity                  =  1,000 work/s

The service is not safe. The failed zone held caches, connection pools, search fragments, consumer ownership, and replicated state. Refill, replay, re-replication, and reconciliation require another 1,200 request-equivalent work units per second on the same storage and network path:

mandatory foreground     7,200 work/s
optional enrichment      1,800 work/s
recovery                 1,200 work/s
                         ------------
total                    10,200 work/s > 10,000 safe

The system can survive the first failure and still be unable to recover. At 10,200 work/s, queues grow, tail latency rises, probes fail, replicas churn, and recovery stretches. The apparent 1,000 work/s of headroom did not include the work needed to restore redundancy.

Mercury enters a declared brownout and removes the 1,800 work/s optional path:

mandatory foreground     7,200 work/s
recovery                 1,200 work/s
                         ------------
protected total          8,400 work/s
recovery headroom        1,600 work/s

That result reverses a common priority. Recovery is not background housekeeping performed after serving has been protected. Recovery and serving compete inside one capacity plan. Degraded behavior creates the space in which redundancy can be rebuilt.

The governing rule is: engineer recovery load and degraded goodput as carefully as steady-state capacity; a system that survives failure but cannot recover is not resilient.

Partial failure makes “up” an incomplete state

In one Mercury checkout, the gateway is reachable, identity accepts the token, inventory commits a reservation, recommendation times out, the response exceeds its deadline, and the order-status lookup later succeeds. Was checkout available?

The answer depends on the unit of success and observation point:

• the gateway process was alive;
• the required inventory effect completed;
• the optional recommendation did not;
• the client did not receive a timely result;
• the operation outcome was initially ambiguous to the client; and
• the status API eventually revealed a committed result.

Binary component health erases these distinctions. Model operation state instead:

Partial failure is also spatial and temporal. One tenant, partition, feature, dependency, region, software version, or request class may fail while aggregate goodput looks healthy. A system may alternate between healthy and impaired faster than a dashboard window can show. State, population, and time interval belong beside every availability claim.

Ambiguous completion is not unique to writes. A read may return a partial result without coverage metadata. A stream may advance one partition while another is stalled. A shutdown may stop accepting requests while orphaned background work continues. The architecture must say what completed, what remains owned, and which evidence resolves uncertainty.

Failure detection is a policy under uncertainty

A timeout observes the absence of an expected event before a local deadline. It does not prove the remote process failed. The same observation can result from:

• process or host failure;
• network loss, partition, congestion, or route change;
• queueing or saturation at the target;
• a stop-the-world pause or scheduler starvation;
• an overloaded observer unable to process the reply;
• connection-pool, DNS, proxy, certificate, or control-plane delay; or
• a detector threshold shorter than normal tail behavior.

Detection chooses among three competing quantities:

1. speed: how long real failure remains on the path;
2. accuracy: how often healthy but slow members are suspected; and
3. disruption: what a suspicion triggers—traffic shift, leader election, restart, re-replication, or human page.

A faster detector can lower exposure to a dead dependency while increasing false removal during load spikes. False removal reduces capacity, shifts traffic, and may turn slowness into an outage. A slower detector avoids churn but spends more user deadlines on a component that cannot answer.

Use a state machine instead of jumping from one missed probe to destruction:

HEALTHY -> SUSPECT -> DEGRADED -> ISOLATED -> RECOVERING -> HEALTHY
             |                                            ^
             +------ sufficient contrary evidence --------+

SUSPECT is an evidence state. It can reduce new work, increase sampling, or request an indirect probe without destroying state. DEGRADED confirms an impairment relevant to a traffic class and applies cheaper semantics or lower concurrency. ISOLATED removes authority or serving traffic through a fenced transition. RECOVERING receives bounded probes and warm-up load, not the full queue. Return to HEALTHY only after useful-work evidence and a stability window.

SWIM’s design is instructive because it separates failure detection from dissemination and uses suspicion plus indirect probes to reduce false positives while controlling message cost. Its protocol is not a drop-in Mercury membership guarantee. Mercury still needs workload-specific thresholds, authority fencing, and disruption policy.

Record detector quality:

• probe interval and timeout distribution;
• detection delay from independently known failures;
• suspicion duration;
• false-suspicion rate by load, zone, version, and network state;
• time from suspicion to safe traffic removal;
• traffic shifted and retries created;
• recovery probation duration; and
• user goodput during detector actions.

The detector’s own queue, CPU, thread pool, DNS, credentials, and dependencies belong in the failure graph. A health endpoint scheduled behind saturated user work will report the saturation, but restarting the process may only move demand to fewer replicas.

Slow and failed dependencies consume different budgets

A failed dependency that rejects immediately may be cheaper than a slow dependency that holds connections, buffers, threads, memory, and caller deadlines before failing. “Success rate is 99.9%” can coexist with unusable tails if the remaining requests occupy resources for seconds.

For each dependency edge, define:

• whether its outcome is required, optional, substitutable, or deferrable;
• caller deadline and reserved response/merge budget;
• maximum concurrent and pending work;
• cancellation behavior and orphan ownership;
• retry owner, identity, attempt budget, and backoff;
• slow threshold based on remaining user budget, not a global adjective;
• degraded response and correctness bound;
• breaker/isolation state and probe policy; and
• outcome retrieval after ambiguity.

Suppose checkout has 180 ms remaining when it calls inventory. Inventory is required and normally completes in 25 ms, but its p99 during one shard movement rises to 210 ms. A 500 ms generic timeout means almost every slow attempt has already missed the journey before the caller gives up. The request remains in flight, holds inventory concurrency, then may be retried by a caller whose own deadline expired.

The repair propagates the operation deadline, reserves local response time, and gives inventory a bounded attempt budget such as 140 ms for this path. That number is a budget allocation, not a claim that 141 ms means the server is dead. Inventory returns committed/rejected/pending/unknown by operation key. Checkout does not repeat an unknown reservation with a new identity.

Slow-path telemetry must distinguish connection acquisition, network, remote queue, service, lock/storage wait, response transit, and abandoned work. A circuit breaker fed only by response codes can miss slow saturation. A latency-only breaker can eject healthy instances during a shared network delay and concentrate the remaining load.

Wide and deep graphs amplify tails

Serial dependencies add elapsed-time distributions through a joint workload, not by adding percentiles. Wide fan-out completes according to a maximum, quorum, threshold, first-valid, or deadline-partial rule. Waiting for every branch exposes the request to every branch tail.

For a teaching bound, assume each of n identical, independent branches meets its deadline with probability p. The probability that all branches meet it is:

[ P(\text{all on time}) = p^n ]

With p = 0.995:

1 branch:   0.995^1  = 0.9950  = 99.50%
8 branches: 0.995^8  = 0.9607  = 96.07%
24 branches:0.995^24 = 0.8867  = 88.67%

A component number that looks excellent becomes an 11.33% modeled deadline-miss risk when the request waits for all 24 branches. This is not a production forecast. Branches share request complexity, hot keys, hosts, networks, power, deployments, caches, dependencies, and control planes. Positive correlation can make simultaneous slowdowns worse than the independent model; other completion rules can make the request less exposed.

The decisive evidence is the joint distribution under the real fan-out and completion rule:

• per-request branch count and selected shards;
• maximum, second-largest, and merge time;
• branch latency conditioned on request class and key heat;
• correlation by host, rack, zone, dependency, and release;
• abandoned branch work after caller completion;
• partial-result coverage and correctness; and
• retries or hedges added per logical request.

Reduce tail exposure by changing the mechanism, not hiding the percentile:

• prune partitions from indexes or routing metadata;
• co-locate data that must be combined;
• use hierarchical aggregation to bound connections and merge queues;
• define quorum, threshold, first-valid, or partial completion only when semantics permit;
• reserve merge and response time before allocating branch deadlines;
• cancel unnecessary work while accounting for cancellation delay; and
• hedge only where independent alternatives, idempotent reads, spare capacity, and a tail trigger justify duplicate work.

The counterexample is a hedge during correlated overload. It duplicates the most expensive slow requests onto replicas sharing the same constrained storage, raising queueing and worsening both copies. Chapter 25’s hedge frontier and attempt budget still apply.

Breakers and isolation bound damage; they do not create capacity

A circuit breaker stops or limits new attempts when evidence crosses a policy threshold. It can fail quickly, protect a dependency from hopeless work, and give recovery probes a controlled path. It cannot repair the dependency, guarantee an alternative result, or make rejected work disappear.

Useful controls sit at several boundaries:

• concurrent requests: cap in-flight resource exposure;
• pending requests: bound waiting and memory;
• connections: prevent one upstream or traffic class from consuming all sockets;
• retry/hedge budgets: bound duplicate work per logical operation and fleet interval;
• bulkheads: separate pools, queues, caches, or replicas by dependency, class, tenant, or cell;
• deadline admission: reject work whose remaining budget cannot fund useful completion; and
• breaker state: closed, open, and bounded half-open probes with explicit reset evidence.

Envoy’s breaker documentation is a concrete implementation example: limits are local to Envoy processes rather than a globally synchronized capacity oracle. A configured threshold therefore needs fleet overshoot analysis. Ten clients each allowed 100 pending requests can expose a backend to 1,000 pending requests before other layers are counted.

Place protection near the constrained resource and admission near the point where work can still be rejected cheaply. A gateway-only breaker cannot see per-shard saturation. A backend-only rejection may arrive after the gateway and intermediate services have already spent most of the deadline and CPU.

Half-open recovery deserves a budget. If every client sends probes simultaneously after a 30-second timer, the “test” becomes a synchronized restart storm. Randomize probe timing, cap fleet-wide probe work where possible, and require useful responses—not TCP success alone—before ramping.

Design the degraded answer before the incident

Mercury’s three-dependency checkout makes the semantic choices explicit:

This is degraded design, not error-message design. It says which user journey remains correct and which work disappears.

Brownout options include:

• omit optional enrichments, previews, counts, or secondary ranking;
• serve a bounded stale representation with version and freshness marker;
• return partial coverage when the product contract supports it;
• accept a durable asynchronous job instead of holding an interactive request;
• lower fidelity, resolution, model size, or analysis depth under an explicit quality floor; and
• reserve capacity for high-value or safety-critical operations while rejecting cheap-to-retry work early.

Each option needs an invariant. A stale product description may be acceptable while a stale price is not. A search result with 18 of 20 shards and coverage metadata may be useful while an account balance missing two partitions is false. “Graceful degradation” without named correctness and freshness bounds is merely an optimistic label.

Avoid recursive health dependencies. If checkout’s readiness endpoint synchronously calls identity, inventory, recommendation, DNS, the config service, and telemetry, one optional impairment can remove every checkout replica. Probe the component’s ability to perform its declared work from locally applied state, and test critical dependencies through separate synthetic or dependency signals whose failure policy is known.

Recovery has its own workload model

Restoring a distributed system can create more work than the failure removed:

• cache refill: cold instances convert hits into backend reads and allocate memory rapidly;
• queue replay: accumulated events compete with live traffic and repeat idempotency lookups or effects;
• re-replication: large state transfers consume storage I/O, CPU, network, checksums, and compaction;
• leader churn: elections invalidate caches/connections and move hot ownership repeatedly;
• restart storms: simultaneous initialization, image pulls, JIT, schema loads, and connection creation concentrate load;
• reconciliation: unknown operations query or compare authoritative state;
• deferred maintenance: checkpoints, cleanup, compaction, backup, and indexing catch up; and
• operator changes: traffic shifts, rollbacks, and diagnostics consume the same control planes.

Measure recovery work in resource demand, not task count. One cache entry, event, or replica byte is not equivalent across keys and state. Identify the bottleneck path and convert foreground and recovery classes into a comparable constrained-resource unit such as CPU ms/s, storage IOPS, bytes/s, or measured request-equivalent work/s.

Mercury’s capacity model is deliberately simple. Three zones provide 15,000 safe work/s before failure. Losing one leaves 10,000. Nine thousand foreground work/s appears to fit, but 1,200 recovery work/s makes 10,200. Removing 1,800 optional work/s yields 8,400 and restores 1,600 work/s of modeled headroom.

The arithmetic says what to test:

• can two zones each sustain 5,000 work/s for the degraded mix, not the normal mix?
• does traffic distribute evenly after failure, including hot tenants and shards?
• do identity, storage, queue, and network dependencies also have N−1 capacity?
• does the 1,200 work/s recovery cap hold at every downstream boundary?
• can optional work actually be removed without synchronous policy or cache calls?
• is 1,600 work/s enough for variance, control traffic, probe work, and another fault?
• what stops recovery automatically when queue age or mandatory goodput worsens?

N−1 is a topology claim only after common dependencies and service demand are included. “Two replicas remain” is not a capacity proof.

Drain backlog without refilling it

Suppose the impaired interval creates 864,000 request-equivalent work units. At a governed net recovery allocation of 1,200 work/s:

drain time = 864,000 work / 1,200 work/s = 720 s = 12 min

This assumes the 1,200 work/s is net drain after new live work, retries, and new recovery work. If it is gross service and new backlog continues at 900 work/s, net drain is only 300 work/s and recovery takes 48 minutes. State restore, cache warm-up, partition skew, compaction, and downstream limits add time.

Track oldest work age and estimated drain time, not only depth. A queue can shrink while the oldest high-cost or poison work remains. Recovery is complete when redundancy, user goodput, tail behavior, backlog age, state convergence, and control stability meet declared objectives—not when the first replacement process starts.

Correlated failure follows hidden dependencies

Independent-replica diagrams often share:

• region, zone, rack, power, network, storage, DNS, identity, certificates, secrets, or time sources;
• one database cluster, queue, cache, object store, control plane, quota, or egress path;
• one library, binary, configuration, schema, credential, feature flag, or rollout wave;
• one tenant, hot key, traffic source, abuse pattern, or malformed payload;
• one operator, runbook, automation, or emergency permission; and
• one recovery destination that all failed components target.

Draw a failure graph for four modes: steady serving, change, observation, and recovery. A control plane absent from the request path can still prevent failover or push a bad global configuration. An observability pipeline can disappear exactly when operators need to distinguish failure from overload. A backup stored behind the same identity or key authority may be unreachable during recovery.

Correlation also changes probability. Do not multiply replica availabilities or branch deadline probabilities as if shared causes were absent. Use fault-domain inventories, incident co-occurrence, topology-aware telemetry, rollout grouping, and fault injection. Treat the independence calculation as a diagnostic prompt: which assumptions would make the multiplication invalid?

The familiar counterexample is “move traffic away from unhealthy instances.” If all instances are slow because their shared database is saturated, ejection concentrates traffic on fewer instances without changing database demand. The policy improves a local error-rate metric while reducing global goodput.

Health checks must test the right layer

Separate at least three questions:

1. process liveness: Is the process making enough internal progress that restart is likely to help?
2. traffic readiness: Should this instance receive new work for a declared class now?
3. work readiness: Can the instance complete representative useful work within the current objective and correctness boundary?

Kubernetes documents startup, liveness, and readiness probes with different actions. A liveness failure can restart a container; a readiness failure removes it from normal service routing. Misconfigured liveness probes can create cascading failures by restarting overloaded instances and shifting traffic to fewer peers. The product provides mechanics; the application must provide semantics.

A cheap liveness probe should not wait behind the same unbounded user queue, yet it must detect loss of meaningful progress rather than merely return from a dedicated thread while the service is deadlocked elsewhere. Readiness can consider local admission state, warmed indexes, applied configuration epoch, critical pool availability, and ability to reach required state under bounded conditions. It should not recursively demand that every optional downstream be healthy.

Use synthetic work and shadow probes for deeper readiness:

process: event loop advances; watchdog generation changes
local readiness: config epoch applied; required indexes loaded; admission open
dependency readiness: bounded operation reaches authoritative test key
service readiness: canary logical operation meets correctness and latency objective
recovery readiness: probation traffic succeeds without queue-age or error rebound

Keep probe traffic bounded and distinguish probe failure caused by the observer, route, authentication, or target. Record the version and topology under test. A green /health that bypasses authorization, storage, queues, and the real execution pool proves very little about useful work.

Drain and quiesce before removing authority

Graceful shutdown is a protocol, not a signal handler that sleeps.

A server leaving service should:

1. mark itself draining and stop admission of new long-lived or ordinary work;
2. propagate routing/readiness change and account for convergence delay;
3. retain authority only for work it can finish safely, or transfer authority with a new fenced epoch;
4. cancel or hand off queued work according to its durable ownership;
5. complete admitted operations within their deadlines and grace budget;
6. persist terminal or resumable state for ambiguous operations;
7. close streams, leases, and consumer ownership without acknowledging unfinished effects;
8. flush only bounded critical evidence—never block forever on telemetry;
9. report in-flight, queued, orphaned, and transferred counts; and
10. force stop at the declared bound, with recovery able to classify interrupted work.

Quiescence means no new work can enter a boundary and all accepted work is terminal, durably handed off, or explicitly recoverable. Zero active sockets is neither necessary nor sufficient. A background task may hold an authority lease without a socket; a keep-alive connection may be idle.

Kubernetes’ pod termination flow sets terminating endpoints not ready for regular traffic, invokes configured shutdown behavior, sends a termination signal, and enforces a grace period before forced termination. Endpoint propagation and external load balancers are not instantaneous. Applications still need admission closure, connection drain, operation identity, and state handoff.

Test shutdown under:

• streaming and long-running operations;
• full queues and expired deadlines;
• consumer processing between effect commit and transport acknowledgment;
• leader or shard-owner transfer;
• control-plane delay;
• telemetry outage;
• forced kill at each lifecycle point; and
• simultaneous rollout during partial failure.

“No errors during a quiet restart” is not evidence for graceful drain under peak load.

Recovery objectives are performance requirements

RPO and RTO alone often hide the behavior users and operators need. Record a recovery ledger:

affected operation/population: checkout writes, tenants in zone B
success during degradation: authenticated order accepted, rejected, pending, or unavailable
minimum protected goodput: 7,200 mandatory work/s at declared mix
maximum degraded p99: scoped by successful/pending population and interval
detection objective: evidence to SUSPECT, not automatic destruction
isolation objective: fenced removal of failed authority
state objective: maximum lost or unreconciled committed state
restore objective: state available for validation
warm-up objective: representative work meets probation criteria
backlog objective: oldest age and drain time, including live arrivals
redundancy objective: N−1 restored with uncertainty headroom
ramp objective: bounded stages with rollback and stability windows
completion evidence: goodput, tails, invariants, backlog, convergence, probe quality

Break recovery time into terms:

[ T_{recover} = T_{detect} + T_{isolate} + T_{restore} + T_{warm} + T_{drain} + T_{ramp} ]

The terms can overlap, but naming them prevents “RTO met” when a process is alive while caches are cold and six hours of work remain. Define start and end events for each term, population, success criteria, and whether the evidence is observed, modeled, or inferred.

Recovery objectives also constrain design. A 15-minute redundancy objective may disqualify a 20 TiB full-copy recovery path unless incremental replication and bandwidth reservation are proven. A five-minute serving objective may require warm standby, preloaded indexes, reserved credentials, and live probe traffic. A zero-data-loss claim may require synchronous coordination that raises steady latency and changes partition behavior. State the trade.

Recovery and degradation drills

Classify partial outcomes. For connect refusal, timeout before headers, timeout after a commit, explicit overload rejection, stale response, partial fan-out, and shutdown interruption, state what is known, what remains ambiguous, who owns resolution, and which identity retrieves the outcome.

Tune detection by action. Compare a detector that marks SUSPECT, one that removes traffic, one that triggers leadership change, and one that restarts the process. Set evidence and delay proportional to disruption. Inject overload at the observer, a network partition, a 900 ms runtime pause, and a true crash.

Challenge tail arithmetic. Reproduce 99.50%, 96.07%, and 88.67%. Then list at least five correlations in the target system and choose a measurement that captures the joint tail. Change completion from all-24 to 20-of-24 with coverage metadata and state which correctness rule makes that legal.

Design three dependency responses. For identity, inventory, and recommendation, specify required outcome, remaining deadline, concurrency/pending bounds, degraded answer, stale-data rule, retry owner, ambiguous-outcome query, and recovery evidence. Reject any table that says only “use a circuit breaker.”

Prove N−1 with recovery. Reproduce 10,200 work/s unsafe demand, 8,400 corrected demand, and 1,600 headroom. Replace request-equivalent units with measured CPU, storage, and network demand. Add a hot tenant, one shared database bottleneck, and a second fault. Identify the first invalid assumption.

Drain the backlog. Reproduce 720 seconds at 1,200 net work/s. Then reinterpret 1,200 as gross service with 900 work/s new arrivals and calculate the 48-minute drain. Add poison work and state how oldest age, quarantine, and stop rules change.

Kill during drain. Stop a server after admission closes, after routing change, during an in-flight write, after effect commit, before response, during consumer acknowledgment, and after authority epoch transfer. Prove every accepted operation is terminal, handed off, or queryable.

Durable rules for partial failure and recovery

1. Define health and availability by operation, population, objective, and state—not process reachability alone.
2. Treat timeout as evidence of delay or ambiguity, not proof of remote failure.
3. Tune detection speed and confidence according to the disruption caused by suspicion.
4. Bound concurrency, pending work, retries, hedges, and probes at the constrained boundary.
5. Model deep and wide tail behavior from joint distributions and the actual completion rule.
6. Give every dependency a required/optional classification, degraded behavior, correctness bound, and recovery evidence.
7. Make brownout and stale serving explicit product semantics rather than incident improvisation.
8. Include cache refill, replay, repair, re-replication, reconciliation, and control work in N−1 capacity.
9. Draw common dependencies across serving, change, observation, and recovery paths.
10. Separate process liveness, traffic readiness, useful-work readiness, and recovery probation.
11. Drain by stopping admission and proving terminal ownership; pair authority transfer with fencing.
12. Define recovery through goodput, tails, state, backlog age, redundancy, and safe ramp—not first process start.

Mercury’s next boundary is geographic. Once a zone or region is impaired, propagation delay, data residency, replication lag, traffic steering, evacuation capacity, and failback change the available choices. Chapter 35 carries the recovery model across that physical distance.

Evidence and transfer limits

• Dean and Barroso’s primary paper, “The Tail at Scale”, explains how component tail behavior affects large fan-out services and discusses tail-tolerance techniques. Mercury’s 0.995 calculations are a separate independence model, not measurements from that paper.
• Google SRE’s “Addressing Cascading Failures” documents overload feedback, retry amplification, health-check cascades, degraded modes, traffic reduction, and gradual recovery. Its operational examples motivate tests; they do not supply Mercury’s capacity values.
• The primary SWIM paper separates failure detection and dissemination and analyzes detection time, message load, suspicion, and false positives under its protocol. Mercury’s application health and authority transitions require additional semantics.
• Official Envoy Gateway circuit-breaker documentation describes connection, concurrent-request, and pending-request limits and notes that counters are distributed across Envoy processes. Product defaults and mechanics must be tested against Mercury’s workload and fleet overshoot.
• Kubernetes’ official documentation distinguishes startup, liveness, and readiness probes and warns that incorrect liveness behavior can cascade. Its pod lifecycle documentation describes termination, endpoint state, signals, and grace periods. These platform mechanics do not prove application quiescence or operation completion.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/partial-failure-recovery/ reproduces the independent-branch probabilities, 15,000-to-10,000 work/s zone-loss capacity, 10,200 work/s unsafe demand, 8,400 corrected demand, 1,600 headroom, and 12-minute backlog drain. These are deterministic teaching calculations, not measured production capacity.