Performance Incident Response

At 02:14, Mercury API is receiving 8,000 logical operations per minute—the same demand it handled three minutes earlier. Yet clients are sending 13,760 attempts/min, p99 queue age is 3.1 seconds against a 750 ms completion deadline, CPU is 96%, and only 2,600 correct operations/min finish on time. The optional profile-enrichment dependency has slowed from 158 ms to 792 ms at p99. Immediate retries are converting delay into more work, which creates more delay, which triggers more retries.

The incident commander does not need the dependency’s root cause to make the first decision.

At 02:15 the team disables automatic retry for the affected read path, omits optional enrichment under the existing response contract, and rejects work that cannot finish before its remaining deadline. Attempts fall to 8,400/min. Goodput rises to 6,700/min even though dependency p99 is still 706 ms. Queue age drops 40% in one minute. The intervention removes 5,360 attempts/min and restores 4,100 useful completions/min before anyone explains why the dependency slowed.

That sequence defines the operating discipline for acute performance failure: bound impact, stop positive feedback, protect useful work and control paths, capture disappearing evidence, prove recovery, then pursue the deeper cause. Explanation matters, but explanation performed inside an accelerating loop can become another source of delay.

Part VII shifts the book’s question from “What evidence supports this design?” to “How does the system remain governable while its evidence, workload, software, and capacity are changing?” Incident response is the sharpest version of that problem. Every emergency action changes the experiment while people are still observing it.

Declare the incident boundary in operational terms

An alert is a clue, not the boundary. The boundary connects user impact to work and resources:

Affected capability: Mercury customer read API with optional enrichment
Population: region A interactive reads, all tenants, versions 7.18–7.19
Interval: 02:11 onward; five-minute SLO windows plus one-minute command view
Unit of work: one logical customer read; attempts reported separately
Success: correct response within 750 ms; enrichment may be omitted and labeled
User impact: deadline misses and incomplete optional profile data
Current constraint: enrichment wait plus retry-inflated ingress/CPU/queue
Known unaffected: authoritative writes and region B control plane
Unknown: dependency root cause and cross-region transfer safety

This record prevents four common errors. Attempt throughput cannot masquerade as demand. Successful latency cannot hide failed and expired operations. An optional field cannot be shed unless the response contract allows it. An unaffected control plane cannot be assumed merely because its dashboard still loads.

Start with user-visible impact—deadline goodput, error/timeout outcomes, freshness or quality, and affected population—then connect it to queue age, saturation, service demand, retries, and failures. CPU alone does not define the incident. A queue can be growing while CPU is moderate because a dependency, lock, connection pool, storage device, quota, or serialized authority is the constraint.

Assign explicit command roles early enough that work does not fork invisibly:

• an incident commander owns priorities, actions, and stop/continue decisions;
• an operations lead executes reversible production changes;
• an investigation lead ranks hypotheses and requests discriminating evidence;
• a communications lead records impact and stakeholder updates; and
• a scribe maintains the clock-aligned narrative and action ledger.

One person may hold multiple roles in a small incident, but one decision must still have one owner. Subject-matter experts advise; they do not each tune a subsystem independently.

Stabilization buys decision time

Stabilization reduces the rate at which the system loses useful work or enters less reversible states. It is not synonymous with rollback, failover, or adding capacity. Choose the action that interrupts the dominant loop with the lowest correctness and recovery risk.

Traffic reduction

Reduce offered work, not merely accepted connections. Admission can reject low-priority, expired, unauthenticated, over-quota, or excess work before it consumes the constrained resource. Coordinate with clients and upstreams: a fast rejection can increase load if every caller retries immediately.

Prefer bounded, explicit controls:

• lower per-tenant or per-class concurrency at the earliest authoritative gate;
• pause background exports, prefetch, reindexing, compaction, or recovery work that competes with the user path;
• shape upstream traffic with retry budgets and jitter rather than moving a queue one hop away; and
• protect a small probe class so recovery can be observed without reopening the flood.

Dropping all traffic may preserve a database but destroy essential operations. Continuing all traffic may destroy both. Rank classes by correctness, urgency, cost, and retry behavior.

Feature shedding and quality degradation

Shedding removes optional work while preserving the declared core result. Mercury can omit enrichment because the API contract distinguishes authoritative customer data from optional profile decoration. It must mark that omission and avoid caching an incomplete response as complete.

Unsafe “degradation” changes correctness silently: stale authorization, skipped fraud checks, missing durability, reduced model quality outside an agreed floor, or partial writes reported as success. Precompute degradation states during design. During an incident, the team should select a named state, not invent one under pressure.

Rollback

Rollback is attractive when a recent code or configuration change is temporally and causally plausible and the prior version remains compatible with current state. Check:

• whether the rollout changed schemas, data, caches, or protocols that the old version cannot read;
• whether rollback itself causes cold starts, cache misses, connection churn, or capacity loss;
• whether the suspected change is in Mercury, its dependency, client behavior, or infrastructure; and
• whether enough unaffected control capacity exists to execute and observe the rollback.

Mercury’s deployment finished at 01:48 and operated normally for 23 minutes. That does not exonerate it—a threshold or delayed path may be involved—but immediate fleet-wide rollback is weaker than cutting the observed retry loop. A canary rollback can remain a discriminating test after stabilization.

Failover

Failover transfers workload, state access, and failure risk. Before moving traffic, verify destination spare capacity under the transferred work, cache warmth, data freshness, write authority, idempotency, network path, quota, and shared dependencies. A destination that shares the slow dependency is not an independent escape. A region with 20% spare capacity cannot safely receive 50% more retry-amplified work.

Reads may tolerate a stale or reduced mode that writes cannot. Do not fail writes across regions until authority, fencing, duplicate effects, reconciliation, and data-loss boundaries are known. “The other region is green” describes its current load, not its post-failover state.

Capacity add

Capacity helps when service demand is understood, work is parallelizable, provisioning is fast relative to queue tolerance, and downstream capacity exists. It can fail when:

• retries or unbounded queues consume every new worker;
• startup and cache warm-up add load before capacity becomes useful;
• the constraint is serialized, locked, quota-limited, or downstream;
• health checks recycle warming instances; or
• adding replicas increases coordination, replication, or connection pressure.

For Mercury, one minute of peak work contains 13,760 attempts for 8,000 originals. Adding 50% frontend capacity without controlling retries may admit still more dependency work and delay collapse rather than stop it. Cut amplification first, then add capacity if the post-control service-demand model supports it.

Use a symptom-to-stabilization matrix, not a reflex

The matrix below is a command aid. “Prefer” means evaluate first; “avoid” means the action is high-risk until a condition is checked.

Every action gets an action-ledger entry: UTC timestamp, commander, executor, exact scope, predicted effect, correctness risk, measurement window, observed effect, rollback condition, and completion state. Without that record, three changes can land together and destroy both causal evidence and rollback safety.

Protect the tools needed to operate the service

Overload often consumes the same queues, authentication paths, network, storage, and CPU used by health checks, deploys, feature controls, telemetry, and operator access. A service can become impossible to recover even while data-plane capacity still exists.

Reserve bounded capacity for:

• health and readiness signals that distinguish process life from service overload;
• configuration and feature-control reads/writes;
• deployment, rollback, and traffic-management commands;
• incident telemetry, exemplars, and low-cost diagnostic queries;
• authentication and authorization for operators; and
• one synthetic or probe workload that validates useful completion.

Isolation must exist before the incident. A “priority” queue sharing the same exhausted worker pool is only a label. Test that control requests bypass or reserve the constrained resource, that telemetry collectors shed safely, and that dashboards have bounded query cost. Preserve a low-cardinality emergency view when high-dimensional analysis is too expensive.

Do not turn on exhaustive tracing, heap dumps, lock instrumentation, or debug logging fleet-wide during saturation. Estimate overhead and blast radius. Use a small cohort, bounded duration, sampling, and an automatic off switch. Evidence collection that worsens goodput or exhausts disk is another incident action, not a harmless observation.

Build one time-aligned narrative

Different systems report event time, observation time, scrape time, ingestion time, and display time. Hosts can have clock error; aggregation windows can make effects appear before causes. Keep original timestamps and observed timestamps, record clock source/uncertainty, and align changes to the smallest honest resolution.

Mercury’s minute record is:

These rows are simulated one-minute aggregates. They do not prove event order within each minute. The decisive pattern is stronger than simple correlation: logical demand stays flat; dependency delay rises before attempts; attempts, queue, CPU, and goodput worsen together; then attempt control improves queue and goodput before the dependency fully recovers. That supports the retry-feedback mechanism without explaining the dependency’s initiating fault.

Separate five causal classes on the same timeline:

1. Workload change: originals, mix, payload, keys, tenants, geography, deadlines, and client versions.
2. Dependency change: latency, errors, quotas, capacity, deployment, cache, and routing outside the immediate service.
3. Code change: binary, library, feature, query, allocation, or protocol behavior.
4. Configuration change: timeout, retry, queue, pool, limit, route, feature, or runtime setting.
5. Infrastructure change: host, CPU policy, network, storage, zone, scheduler, kernel, or control plane.

“No deployment” does not mean “no change.” Configuration, dependencies, certificates, data distributions, client versions, traffic routes, and infrastructure move independently. “Traffic is flat” can also be false if only request count is flat while payload, fan-out, cache misses, or attempts change.

Rank hypotheses by decision value

During stabilization, a hypothesis is useful when one low-risk check can change the next action. Rank by explanatory coverage, prior plausibility, evidence already present, check time, check risk, and actionability.

Do not equate the highest-ranked hypothesis with root cause. “Retry-amplified dependency slowdown” contains a trigger and an amplifier; the packet validates the amplifier more strongly than the trigger. The underlying dependency change may be code, load, data, configuration, or infrastructure.

Use traces and logs as correlated evidence, not a perfect global clock. OpenTelemetry’s log data model distinguishes source Timestamp from ObservedTimestamp and provides optional trace/span context. Preserve both when collection delay matters. Trace IDs connect execution context; they do not recover untraced client queues, dropped telemetry, unsampled failures, or clock accuracy.

Avoid tuning that destroys the experiment

High-risk tuning is attractive because it feels active. During uncertainty, prefer changes that directly interrupt an observed mechanism, are reversible, have a bounded scope, and produce a discriminating effect.

Avoid by default:

• increasing queue or timeout limits when queue age already exceeds useful deadlines;
• changing thread pools, GC, kernel, database memory, connection pools, and retry policy together;
• restarting the fleet without knowing whether cold state or reconnection will worsen load;
• forcing a query plan globally from one slow exemplar;
• failing traffic over before calculating destination work and authority; and
• deleting or truncating logs, profiles, queue state, or failed artifacts before capture.

If a risky action is the only plausible safety move, state the predicted mechanism and stop condition. Example: “Restart one wedged cohort because its progress counter is flat for five minutes while peers complete; abort if remaining-fleet utilization exceeds 75% or cold-miss demand rises 10%.” This is an experiment with operational authority, not a ritual.

Capture volatile evidence before the system erases it

Recovery drains queues, replaces processes, rotates logs, changes caches, rewrites autoscaler history, and makes profiles look normal. Capture enough state to reconstruct the transition without delaying stabilization.

Use this bounded checklist:

VOLATILE PERFORMANCE-INCIDENT EVIDENCE
[ ] incident boundary, affected/unaffected populations, success definition
[ ] UTC timeline; source and observed timestamps; clock uncertainty
[ ] logical originals, attempts, retries, goodput, errors, late work
[ ] queue depth/age by class; in-flight work; admission and limiter state
[ ] saturation and service demand at the current constraint
[ ] representative trace IDs for fast, slow, failed, retried, and shed work
[ ] deployment, dependency, configuration, feature, route, and infra change IDs
[ ] binary/config/schema/data-statistics versions and host/topology identity
[ ] bounded CPU/off-CPU/allocation/lock/plan/packet evidence where safe
[ ] screenshots or exported raw queries with windows, filters, and aggregation
[ ] every incident action, prediction, scope, observed effect, and rollback state
[ ] privacy/access classification and retention owner

Prefer raw query exports and immutable snapshots over screenshots alone; screenshots preserve a shared view and annotations but not exact samples. Keep the evidence population and missingness. A sampled trace packet should say how sampling treated errors and long requests. A profile should name process, cohort, duration, overhead, and workload mode.

Protect sensitive fields during emergency capture. Broad access does not authorize copying raw customer payloads into an incident document. Use stable pseudonymous identifiers, approved secure stores, least privilege, and an explicit deletion/retention plan.

Recovery must outrun new work

The first green minute is not recovery. Verify:

• deadline goodput and correctness across affected classes;
• queue age and depth falling, not merely stable at a harmful level;
• attempt amplification and late/orphan work returned to policy;
• resource headroom after control and telemetry traffic;
• dependency and cache state across warm/cold cohorts;
• control-plane health and rollback readiness;
• backlog age, duplicates, ordering, expiry, and poison items; and
• behavior when one mitigation is cautiously relaxed.

For a backlog B items, healthy completion capacity mu items/min, and new admitted work lambda items/min, a simple drain lower bound is:

T_drain = B / (mu − lambda), when mu > lambda

Mercury has 14,400 queued items and healthy capacity of 9,000 completions/min. At normal arrivals of 8,000/min, spare drain capacity is 1,000/min, so the lower bound is 14.4 minutes. If admission temporarily reduces new work to 7,600/min, spare capacity becomes 1,400/min and the bound is 10.286 minutes. Both estimates assume homogeneous valid work, no retries, no new failures, no ordering constraints, and constant capacity. Real drain can be slower.

If mu <= lambda, the backlog cannot drain. Adding workers may raise mu; shedding or admission may lower lambda. Expired work should usually be discarded before service, but only under the outcome contract. Durable writes may need ordered replay or reconciliation. Poison items need isolation so one failure cannot halt the drain.

Release mitigations in controlled steps. Re-enabling retries, enrichment, traffic, background work, and normal limits simultaneously recreates the original uncertainty. Keep a hold interval after each change, with stop boundaries on goodput, queue age, attempts, errors, and resource headroom.

Hand off a bounded causal question

Incident command ends when useful work is stable, backlog/recovery is controlled, mitigations have owners, and remaining uncertainty can be investigated without emergency authority. It does not end because the initiating alert cleared.

The deep-analysis handoff should contain:

Stable state: retry disabled, optional enrichment shed, deadline admission active
Affected boundary: Mercury region A interactive reads, 02:11–02:18+
Validated mechanism: dependency delay plus retry feedback amplified attempts to 1.72x
Not yet explained: initiating dependency slowdown; possible delayed rollout interaction
Evidence: raw timeline, change IDs, trace exemplars, queue snapshots, profiles, fixture export
Current risk: mitigation hides enrichment; retry contract differs from normal mode
Recovery: queue-age decline; 14,400-item backlog; 14.4 min lower bound at normal demand
Next discriminating work: dependency change review; compatible cohort rollback; shape-stratified traces
Do-not-do: re-enable retry before queue and dependency hold criteria; cross-region write failover
Owners/expiry: named owner for each mitigation and corrective-action review

This packet gives the optimization workflow a falsifiable question: which change increased enrichment service time for the affected shape, and how much of the user impact remains after retry amplification is removed? It prevents a post-incident team from optimizing the loudest metric while forgetting the mechanism that turned a slowdown into collapse.

Corrective action must span more than the failed component

A strong post-incident program repairs the initiating defect, the amplifier, the missing detection, and the governance gap.

Avoid “add dashboard” as the default corrective action. Detection without a safe control can shorten awareness and still leave the team helpless. Avoid “add capacity” when retry amplification or unbounded waiting remains. Avoid an action list so large that no owner can distinguish risk reduction from cleanup.

Tabletop: retry-amplified dependency slowdown

Run the companion packet before the exercise:

cd examples/performance-engineering-system-design-handbook/part-07/performance-incident-response
node analyze.mjs
node verify.mjs

Give participants only the 02:11 and 02:12 rows first. Ask them to declare the boundary, command roles, three hypotheses, and two safe checks. Then reveal 02:13 and 02:14. Require one stabilization action and an action-ledger entry with predicted effect and stop condition. Reveal 02:15 only after they commit.

The best emergency choices preserve core correctness while cutting work:

• Disable or tightly bound retries for the affected operation because attempts are the observed amplifier and the path has a defined logical identity.
• Shed optional enrichment because omission is inside the response contract and directly removes dependency work.
• Reject expired/insufficient-deadline work early because completing it cannot contribute to declared goodput.
• Reserve control and telemetry capacity so mitigations remain executable and observable.

Treat these as conditional choices:

After the final row, calculate 1.02× and 1.72× attempt amplification, 5,360 attempts/min removed, 4,100 good completions/min restored, 40% first-minute queue-age reduction, and both drain bounds. Then produce the volatile-evidence checklist, the five-class causal timeline, a ranked hypothesis table, and the handoff record.

A strong tabletop allows participants to disagree on the initiating cause while converging on safe loop control. If they need root-cause certainty before acting, the exercise has exposed the central response risk.

Ten-minute field reference

When acute performance impact is growing:

1. Name affected users, logical work, success, deadline, and current constraint.
2. Separate originals from attempts and latency from goodput.
3. Assign command, operations, investigation, communication, and scribe ownership.
4. Find positive feedback: retries, queues, health restarts, cache misses, failover, recovery traffic.
5. Select one bounded action that cuts the loop with explicit correctness and rollback limits.
6. Protect control-plane and low-cost telemetry capacity.
7. Record action time, prediction, scope, and observed result.
8. Capture volatile change, queue, trace, config, profile, and topology evidence safely.
9. Prove recovery through goodput, queue drain, attempt bounds, correctness, and headroom.
10. Hand off validated mechanisms, unresolved causes, mitigation expiry, and next discriminating tests.

The list is deliberately command-oriented. Detailed profiling, optimization, and architectural repair belong after the system can sustain them.

Evidence and transfer limits

• Google SRE’s incident-management guidance describes explicit incident roles, a recognized command structure, communication, and state tracking. Local organizations must adapt authority, escalation, security, and regulatory processes.
• Google SRE’s cascading-failure analysis explains positive-feedback overload, retry/backoff concerns, rollback checks, capacity limits, health-check interactions, and cold-state risk. Its system examples are not Mercury measurements.
• Google’s SRE Workbook chapter on canarying supports matched control/canary evaluation and change-aware rollback decisions. An emergency rollback still needs state and compatibility checks.
• OpenTelemetry’s log data model distinguishes event Timestamp, ObservedTimestamp, resource identity, and optional trace/span identifiers; the logging specification explains time, execution-context, and resource correlation. These fields enable correlation but do not guarantee complete sampling or synchronized clocks.
• RFC 9110 Retry-After defines how a server can indicate how long a user agent ought to wait before a follow-up request. It does not create a retry budget, jitter policy, idempotency contract, or compliance guarantee.
• All Mercury incident values are simulated teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-07/performance-incident-response/. Passing checks validates arithmetic and declared ordering in the packet, not a production root cause.

The decision rule is: during an acute performance incident, first stop positive feedback and protect useful work, correctness, control paths, and volatile evidence; establish detailed root cause after the system is stable enough to test explanations safely.

Recovery creates a disciplined starting point, not an optimization result. The next step is to turn the handoff’s bounded causal question into a reproducible baseline, rank changes by achievable impact, and prove that any gain survives the workload envelope without recreating the incident’s hidden cost.