Case Study: Anatomy of an Overload Incident

At 14:00, Northstar Checkout receives 9,000 logical requests per second. That is an ordinary Tuesday peak. At 14:21, it is completing fewer than half that many purchases even though the edge reports no comparable increase in customers and the platform has requested twenty more application instances.

The first incident channel excerpt contains no root cause, only events:

14:00  dependency p99 rises from 95 ms to 410 ms; error rate still below page threshold
14:02  checkout timeout alerts; edge retries rise
14:04  application queue age crosses 800 ms; service retries rise
14:06  cache hit ratio falls; storage calls rise; autoscaler requests 20 instances
14:09  regional failover shifts 15% more traffic into the cell
14:12  new instances report process-ready but miss caches; goodput continues down
14:13  incident command suppresses nonessential retries and receipt enrichment
14:15  edge admission holds physical attempts below recovery capacity
14:18  queue age falls; cache and dependency latency recover
14:21  gated traffic ramp begins

Every mechanism has a plausible local explanation. The client retries a timeout to improve success. The service retries a dependency call to mask a transient failure. The autoscaler adds replicas when utilization rises. The cache evicts entries under memory pressure. The regional router moves traffic away from an unhealthy cell. Health checks remove slow instances. Together, they convert one dependency slowdown into more attempts, older queues, colder capacity, more backend work, and less useful completion.

This case does not search for a component to blame. It asks a stricter question at each minute: what resource or control currently limits deadline-valid goodput, and which action reduces work on that constraint soon enough to matter?

Northstar, its timeline, and all measurements are fictional. The evidence packet simulates declared arithmetic; it is not a production incident report.

Keep four populations separate during the incident

The command team first repairs the dashboard vocabulary.

• Logical offered load is user operations presented to the checkout boundary, deduplicated by logical request identity.
• Admitted load is the subset the system accepts under the current policy generation.
• Attempt load is physical work, including retries, hedges, replay, fan-out, and background calls.
• Goodput is admitted logical operations that preserve correctness and finish before the 1.8-second end-to-end deadline.

A 200 response after the deadline is physical throughput but not Northstar goodput. A payment-authorized purchase whose response times out remains a correctness-sensitive ambiguous outcome; retrying it with a new identity can double charge. A shed recommendation or receipt-enrichment call is not a failed purchase. Aggregating those populations would make safe degradation look like an outage and duplicate attempts look like demand growth.

The request boundary begins when the authenticated edge accepts a checkout command. It includes admission, application queueing, inventory reservation, payment authorization, durable order commit, and the response. Recommendation, receipt enrichment, analytics, and synchronous fraud features with approved fallback are optional work. Client rendering and asynchronous fulfillment are outside the latency boundary, though fulfillment correctness remains a downstream invariant.

Normal objectives are:

• completion p99 at most 1,800 ms for admitted checkout operations;
• at least 8,100 deadline-valid successes/s at the modeled peak;
• no duplicate payment authorization or order commit;
• queue age below 250 ms in normal state and below 1,000 ms while protected;
• physical attempts no more than 1.08 per logical operation in normal state; and
• recovery to normal admission within twelve minutes of dependency restoration.

During protection, the team permits bounded rejection and removal of optional response fields. It does not permit silent correctness loss, unlimited queueing, or retries that outlive the caller’s deadline.

Reconstruct the incident one clock at a time

14:00–14:02: the dependency becomes the constraint

A payment-risk dependency’s p99 rises from 95 to 410 ms after a storage maintenance event. Median changes little. Application CPU remains 58%. The first misleading conclusion is “we have headroom.”

The relevant resource is not average application CPU. The dependency’s service demand and its bounded client connection pool now govern completion. In-flight calls grow by Little’s Law. More requests wait behind occupied connections, leaving less of the end-to-end deadline for inventory and payment.

Northstar’s application timeout is 350 ms, shorter than the new dependency tail, but it is not derived from the remaining request deadline. Some calls continue after the edge has abandoned the parent. Work that cannot influence a live response consumes connections and dependency capacity.

14:02–14:04: independent retries multiply attempts

The edge retries 28% of logical requests after its timeout. Within each application attempt, the service retry policy generates an additional dependency attempt for 22% of calls. These are not additive independent percentages at one layer. Their modeled composition is:

λ_attempt = 9,000 logical requests/s × (1 + 0.28) × (1 + 0.22)
          = 14,054.4 attempts/s

Physical attempt amplification is 1.5616×. The dependency slowdown has reduced effective attempt capacity to 9,200/s, so a system with unchanged logical demand receives 4,854.4 more attempts per second than it can complete.

The multiplication is a teaching simplification; a real request tree requires attempt counts by edge, service, dependency, branch, and terminal state. The diagnostic lesson survives: retries at multiple layers multiply, and a timeout says nothing about whether earlier work stopped or committed.

The current constraint is now both dependency capacity and the application queue feeding it. Raising an application thread limit would allow more waiting work to hold memory and connections. It would not increase useful dependency service.

14:04–14:06: queue age becomes a deadline detector

At a net 4,854.4 attempts/s, two minutes can add 582,528 queued attempts in the fixture. Northstar’s bounded queues reject before that count, but the arithmetic explains how quickly pressure arrives. Queue length alone is hard to compare when instance count changes. Age of oldest useful work reveals whether the system can still satisfy the deadline.

At 800 ms of application queue age, a request with a 1.8-second end-to-end deadline has already spent most of its risk and payment budget. Processing it may produce a response the caller has abandoned. Keeping it in the queue gives throughput graphs work to show while lowering deadline-valid goodput.

The command team marks work with an absolute deadline propagated from the edge. Each queue checks remaining slack before admission and before expensive stages. Expired work is cancelled end to end. Cancellation that only stops the edge response is not relief.

14:06–14:09: cache and autoscaling create new loops

Longer queues and more in-flight request objects increase heap occupancy. The application cache yields memory under pressure. Its hit ratio drops from 93% to 71%, increasing storage and dependency work per attempt. Garbage collection grows. Effective service capacity falls again.

The autoscaler sees sustained CPU and requests twenty instances. A warm instance can complete a modeled 420 attempts/s under the incident mix. A cold instance initially completes 180/s while loading code, connections, routing state, and cache entries. Twenty requested instances therefore represent 8,400 attempts/s of eventual warm capacity but only 3,600 attempts/s while cold—42.86% of the advertised warm contribution.

“Pod running” and “capacity ready” are different events. The cold instances also miss more caches and open connections concurrently, creating backend work at the exact moment the dependency is constrained. If readiness passes before representative warm-up, the router shifts live demand to replicas whose first operations are the most expensive.

Autoscaling is not inherently wrong. Its delay is wrong for the first mitigation. Detection window, scheduling, image pull, process startup, readiness, connection warm-up, cache warm-up, and load-balancer convergence all occur after overload begins. Capacity that arrives six minutes later cannot save work whose deadline is 1.8 seconds. It can support recovery after the system first reduces offered work.

14:09–14:12: failover exports the overload

The regional router treats health-check failures as lost capacity and moves 15% more traffic into the cell. That policy assumes the destination has independent warm reserve. It does not. The destination is already processing retry amplification with cold new instances.

The current constraint moves across layers: first payment-risk service time, then dependency connections, then application queue age and heap, then storage/cache misses, then ready regional capacity. “Root cause: dependency latency” is historically useful but operationally incomplete. At 14:10, restoring dependency latency alone may not drain old queues, cancel retries, rewarm caches, or return failed instances to service.

The incident commander rejects further automatic failover until destination admission and warm reserve are proven. Failure isolation must include work, not just hosts.

Draw the loops only after the chronology is aligned

Once the clocks are synchronized, the causal structure becomes visible:

dependency slowdown
  -> timeout
  -> retry
  -> physical attempt load
  -> queue age and in-flight memory
  -> fewer deadline-valid completions
  -> more perceived failures
  -> retry

queue/in-flight growth
  -> cache eviction and garbage collection
  -> more backend work and less service capacity
  -> queue growth

autoscaling request
  -> cold instances and connection/cache warm-up
  -> temporary backend work and low effective capacity
  -> pressure signal remains high
  -> more scaling pressure

health failures
  -> regional shift to remaining cells
  -> higher destination load
  -> more health failures

These are positive-feedback loops: their effects reinforce the initiating change. Several controls respond to lagging local signals and share no system-level attempt budget. Each can be reasonable in isolation while the combined loop is unstable.

Stabilize useful work in dependency order

At 14:12 the incident commander sets one objective: keep correct deadline-valid checkouts flowing while reducing work on the constrained dependency. The team logs every action with hypothesis, expected signal, safety boundary, owner, and rollback.

The order matters.

Suppress retries before adding them elsewhere

Retries are admitted work, not a client entitlement. One layer owns the retry budget for the whole logical operation. It retries only an idempotent or safely deduplicated action, only when the failure is classified as transient, only with enough remaining deadline, and only while a system-wide token budget remains. Backoff and jitter reduce correlation but do not create capacity.

Overload responses are explicit and non-retriable within the current budget window. Payment ambiguity uses a status lookup keyed by the original mutation identity, not a new authorization attempt. The service propagates cancellation and absolute deadline to the dependency.

In the protected policy, 8,600 admitted logical operations/s receive at most a 2% retry budget:

λ_protected = 8,600 first attempts/s + 172 retry attempts/s
            = 8,772 attempts/s

That fits the modeled 11,172 attempts/s recovery capacity and leaves 2,400 attempts/s to drain queued work. This is why suppressing retries and admission precede queue recovery.

Shed work by semantic value

Northstar disables recommendations, rich receipt rendering, synchronous analytics, and an optional fraud enrichment whose approved fallback is conservative. It does not shed inventory reservation, payment idempotency, durable order commit, or mandatory risk checks.

Feature shedding is a predesigned product state with response schemas and metrics. Deleting arbitrary calls during an incident risks correctness and creates untested code paths. A “degraded” 200 is counted as goodput only if it satisfies the declared checkout contract.

The team prefers cheap rejection before allocation over accepting work and abandoning it deep in the dependency. Admission is segmented: existing payment continuations and status lookups have reserved capacity; new low-value checkouts receive bounded overload responses first; no tenant can consume the entire retry pool.

Drain below service capacity

A queue drains only when useful service capacity exceeds new physical arrivals:

dQ/dt = λ_attempt - μ_useful < 0

With 180,000 attempts remaining, 11,172 attempts/s of recovery capacity, and 8,772 new attempts/s, the ideal drain rate is 2,400/s and the ideal drain time is 75 seconds. Actual recovery is longer because attempt costs vary, cancellation takes time, cache warm-up consumes capacity, and some old work must be discarded rather than served.

The controller uses age and remaining deadline, not FIFO purity. Expired work is removed. Correctness-sensitive continuations keep reservations. Background replay remains paused. Admission does not return to normal at the first zero-length sample; it holds through cache and dependency recovery and ramps with hysteresis.

Replace reactive thresholds with a load-state machine

Northstar’s old behavior had independent rules: autoscale at CPU 70%, retry certain errors twice, evict cache under heap pressure, fail over on health, and queue up to a fixed count. No rule knew the total state.

The replacement has four operational states.

Normal

Attempt amplification, queue age, dependency latency, heap, cache misses, and ready-capacity reserve are within bounds. Normal optional features run. Retry tokens are scarce and measured. Autoscaling can prepare capacity, but the service does not treat requested replicas as ready.

Pressured

Leading indicators cross a sustained warning surface: attempt amplification, slack at admission, oldest useful queue age, dependency saturation, or ready reserve. The service stops shadow/background work, tightens retry budgets, prewarms already requested capacity, and alerts before goodput collapses. It does not wait for CPU alone.

Protected

A hard gate binds. Admission caps physical work, optional features shed, retries suppress, expensive request classes receive explicit budgets, and failover requires destination authorization. The state aims to keep goodput above 7,800/s and correctness violations at zero. Rejection is an intended outcome, not an unclassified 500.

Recovering

The initiating dependency is healthy, but the system still has queue, cache, connection, and replica debt. New arrivals remain below useful capacity. Queues drain, old attempts reconcile, caches warm under controlled traffic, and instances prove readiness. The controller returns to Normal only after hold periods and a staged ramp. Any gate breach returns directly to Protected.

Transitions are generation-stamped and visible in traces. Configuration authority is centralized enough that retry, admission, failover, and feature policies cannot contradict each other during one incident. Manual command remains available, but it changes a recorded policy generation rather than issuing untracked one-off commands.

Re-run the incident as a controlled experiment

The team converts the chronology into an open-loop resilience test. The offered logical schedule is identical in baseline and protected trials. Both use the same request mix, key popularity, instance image, cache state, dependency slowdown, regional capacity, and failure timing. Logical identities make duplicate attempts and ambiguous commits observable.

The replay injects:

1. payment-risk p99 from 95 to 410 ms for nine minutes;
2. the original edge and service retry policies in the baseline, then the unified budget in the candidate;
3. memory pressure sufficient to exercise cache eviction;
4. twenty new instances with six minutes of cold behavior;
5. a proposed 15% regional shift; and
6. dependency restoration while queues and caches remain degraded.

The baseline reproduces attempt amplification, queue-age growth, cache/backend feedback, premature readiness, and a 29-minute recovery. The candidate admits 8,600 operations/s, caps retry attempts at 172/s, produces 8,170 modeled successes/s at a 95% protected success fraction, holds above the 7,800/s goodput floor, drains the declared 180,000-attempt recovery queue in an ideal 75 seconds, and returns to the normal state in nine minutes. Recovery improves by 68.97% relative to the fixture baseline.

Those numbers do not prove a production system. The production gate requires repeated trials with distributions of service demand, cache state, instance startup, key skew, network loss, and operator timing. It checks correctness and terminal-state reconciliation, not just rates.

The comparison packet reports:

• logical offered, admitted, rejected, attempt, completion, goodput, expired, and ambiguous populations;
• attempt amplification by layer and terminal result;
• queue age and remaining deadline by request class;
• dependency concurrency, connection wait, service time, and cancellation completion;
• requested, process-ready, traffic-ready, and warm capacity;
• cache hit/miss and backend work per logical operation;
• retry token consumption and suppression reason;
• state-machine transition, policy generation, and manual override;
• time to stabilize, drain, ramp, and return to Normal; and
• duplicate authorization/order invariants and reconciliation delay.

Closed-loop load generation is prohibited for this claim because rising latency would reduce offered traffic and hide backlog. Percentiles are not averaged across instances. Time series share a clock and event annotations. The baseline and candidate retain raw event and configuration evidence.

Turn the incident into durable design

The post-incident work is organized by control failure rather than by the team that happened to own each symptom.

Design changes

• one absolute deadline and cancellation context crosses edge, service, and dependency;
• one logical identity spans attempts, payment authorization, and order commit;
• one retry-budget owner limits attempts across layers and classes;
• admission protects each constrained boundary before expensive allocation;
• feature shedding is typed, prevalidated, and correctness-aware;
• queue capacity is bounded by age, memory, and deadline, not only item count;
• destination cells authorize failover based on warm reserve and current attempt load; and
• cache and connection warm-up are explicit readiness requirements.

Test changes

• every service has an open-loop overload curve showing throughput, goodput, queue age, and terminal outcomes beyond saturation;
• resilience trials combine slowdown, retry, cold scale-out, cache loss, health removal, and recovery rather than testing each alone;
• retry multiplication is statically inventoried and dynamically measured;
• cancellation tests prove downstream work stops;
• recovery tests begin from debt, not from a clean restart; and
• regional evacuation tests reserve destination capacity before shifting demand.

Telemetry changes

• dashboards align logical load, attempt load, goodput, queue age, ready capacity, and policy state;
• traces carry logical identity, attempt number, absolute deadline, admission class, retry token, shed features, and terminal result;
• autoscaling reports requested, scheduled, process-ready, route-ready, warm, and useful capacity separately;
• cache misses are translated into downstream work, not displayed only as a ratio; and
• the current constraint ledger is an incident artifact updated as the system changes phase.

Governance changes

Retry, timeout, queue, health, scaling, failover, and cache policy changes require a system-level review when they affect the same request path. The owner must state the normal, pressured, protected, and recovering behavior; the maximum work amplification; the degraded product contract; and the test that proves recovery.

Action items include an owner, due date, validation evidence, and retirement condition. “Tune autoscaling” is not an action. “Do not route checkout traffic to an instance until representative connections are established, required caches meet the declared warm criterion, and a canary proves 350 attempts/s at the incident mix” is testable.

Blameless analysis removes hindsight punishment so operators can report truth. It does not remove ownership of hazardous controls. The system allowed several teams to create work independently without one authority bounding the total.

Source transfer and implementation limits

Google SRE’s cascading-failure chapter supplies primary guidance on overload, retries, load shedding, cold behavior, and the need to test beyond capacity. Its examples do not establish Northstar’s rates or thresholds. Google SRE’s incident-management chapter supports explicit command, roles, and operational structure; it does not prescribe this fictional decision log.

AWS’s Builders’ Library guidance on timeouts, retries, backoff, and jitter supports bounded retries, idempotency awareness, backoff, and jitter. Jitter reduces synchronized retry bursts; it cannot make 14,054 attempts/s fit 9,200 attempts/s of capacity.

Kubernetes’ current Horizontal Pod Autoscaling documentation distinguishes startup/readiness behavior and documents initialization handling. Northstar’s six-minute warm-up and per-instance rates are fixture inputs, not Kubernetes defaults. Envoy’s overload-manager documentation demonstrates resource monitors, triggers, and overload actions as an implementation example. Northstar’s state machine is a system policy spanning more than one proxy.

Run the incident packet:

cd examples/performance-engineering-system-design-handbook/part-08/overload-incident
node analyze.mjs
node verify.mjs

The incident record

Initiating condition. Payment-risk p99 rose from 95 to 410 ms while logical checkout demand remained 9,000/s.

Amplifiers. Independent 28% edge and 22% service retry fractions produced 14,054.4 modeled physical attempts/s. Queueing raised in-flight memory, cache eviction raised backend work, cold scale-out supplied less than half its eventual capacity, health routing shifted more demand, and expired work continued to consume resources.

Stabilization. Suppress retries to a 2% unified budget, shed optional work, admit at most 8,600 logical operations/s, freeze unsafe failover, cancel expired work end to end, and hold physical arrivals at 8,772 attempts/s while queues drain below 11,172 attempts/s of recovery capacity.

Protected result. The fixture produces 8,170 successes/s, above the 7,800/s incident floor; drains 180,000 attempts in an ideal 75 seconds; and reduces full recovery from 29 to nine minutes. Correctness-sensitive continuations retain reservations and stable identities.

Rejected actions. Raising thread or queue limits stores more doomed work. Retrying at another layer multiplies attempts. Immediate regional failover exports pressure. Counting requested replicas as capacity ignores cold work. Removing correctness checks creates false goodput. Returning to full admission at the first green sample reopens the loop.

Residual risks. Attempt costs are heterogeneous; dependency and cache recovery can correlate; client retries outside the controlled edge may persist; readiness tests may not predict production key heat; and manual response time varies. The production safety margin must cover those uncertainties.

Revisit triggers. Attempt amplification exceeds 1.08 in Normal; queue age or remaining slack crosses its state gate; goodput falls below the protected floor; optional shedding changes the required product contract; cold capacity contributes less or later than tested; retry ownership changes; a new dependency enters the critical path; regional reserve falls; or a combined resilience replay cannot return to Normal within twelve minutes.

Applied work

Field exercise: choose the first action

At minute five, dashboards show application CPU 64%, payment-risk p99 520 ms, edge timeout 900 ms, oldest queue age 1.1 seconds, 1.7 physical attempts per logical request, and 94% of the configured maximum replicas requested. The application team proposes doubling the queue. The platform team proposes forcing maximum replicas immediately. Product proposes disabling receipt enrichment.

Choose the first two actions and state the evidence that would change your order.

Answer guide

First suppress retry amplification or bound it at the highest controlled layer; 1.7 attempts per logical request is direct evidence that the system is manufacturing load. Second shed receipt enrichment if it consumes the constrained path, then apply admission sufficient to make queue age fall. Doubling the queue preserves already doomed work: at 1.1 seconds, little of a 1.8-second deadline remains. Maximum requested replicas may help later, but does not describe route-ready warm capacity.

The order could change if receipt enrichment is proven to dominate dependency demand and can be removed instantly while retry-policy propagation is slow. It could also change for a correctness emergency requiring all new mutations to stop. The evidence must be current work per logical request, retry propagation delay, remaining deadline, dependency saturation, and useful ready capacity—not CPU alone.

Principal exercise: approve or reject regional failover

Cell A is protected and admits 7,000 of 9,000 logical operations/s. Its physical attempt rate is falling. Cell B reports 30% CPU reserve, but half its new replicas are cold, its cache miss ratio is twice normal, and client retries from A can arrive independently of the global router. A controller proposes shifting 2,000 logical operations/s from A to B.

Define the admission decision and the minimum trial.

Answer guide

Reject the unconditional shift. CPU reserve is not a destination work budget. Calculate B’s warm useful capacity under the incoming key and dependency mix, include its current logical and physical attempts, reserve failure headroom, and account for retries that bypass the router. B must issue an admission grant; A cannot infer it.

Canary a small stable cohort with one logical identity across regions. Hold A’s retry budget, prevent duplicate payment execution, and observe B’s attempt amplification, queue age, dependency saturation, cache/backend work, goodput, and warm-ready capacity. Increase only after a hold period. If B crosses Pressured, stop the shift before health routing creates a second cascade.

The recovery contract also needs failback. When A becomes healthy, drain or reconcile in-flight operations before moving ownership. A regional move that cannot explain ambiguous payments is not successful overload relief.

Final field card: design for the state after saturation

When a system slows under load, ask in order:

• Is offered logical demand rising, or only physical attempts?
• What counts as deadline-valid correct goodput?
• What is the current constraint now, not only the initiating failure?
• How old is the oldest useful queued work, and how much deadline remains?
• Which retries, hedges, fan-out, replay, or failover paths manufacture work?
• Does cancellation stop work at the constrained boundary?
• Which optional work can be shed without changing required correctness?
• Is new capacity requested, process-ready, route-ready, warm, or actually useful?
• Can caches, connections, health checks, and failover reinforce overload?
• What admission rate makes physical arrivals lower than useful recovery capacity?
• Which queues, caches, and replicas remain as recovery debt after the trigger clears?
• What state and gate prevent a premature return to full traffic?
• Can the combined scenario be replayed with fixed offered load and terminal-state evidence?

The durable performance method is visible in this incident. Define the useful outcome. Separate logical demand from physical work. Follow the critical path. Measure distributions and queues. Name state and authority. Price every retry, cache miss, cold start, and recovery action against a resource boundary. Test the failure path and the path back.

Performance engineering does not promise that a system will never saturate. It makes the behavior on both sides of saturation explainable and bounded: some work is admitted, some is rejected, essential invariants remain intact, operators can see the active constraint, and recovery reduces debt instead of creating another loop. That is the standard by which a design becomes defensible under growth, skew, failure, and change.

When the next design review or incident begins, turn that standard into working evidence. Appendix A keeps units and populations honest; the formula sheets and magnitude references test the first model; the design, experiment, load-test, and incident templates make assumptions and recovery gates reviewable. Use the appendices as field instruments, then revise the design when observed behavior disproves the model.