Appendix H — Performance Incident Runbook

Latency falls after an emergency timeout change, and the incident channel celebrates. Yet valid checkouts continue to decline, attempts per checkout rise, and the oldest queue item is now older than the user deadline. One graph improved because the system abandoned work sooner. The service did not stabilize.

A performance incident runbook must distinguish visible motion from protected outcomes. Its job is to help a command team answer four questions in order: who is being harmed, which feedback or constraint is worsening the state, which bounded action protects the most valuable correct work, and what evidence proves durable recovery? Detailed root-cause analysis can wait; impact, amplification, and expiring evidence cannot.

The copyable artifact at examples/performance-engineering-system-design-handbook/appendices/performance-incident-runbook/performance-incident-runbook-template.md includes command, impact, feedback, action, evidence, timeline, recovery, and handoff tables. A filled Mercury slice shows a retry-amplified dependency slowdown. Its dependency-free verifier checks structure, not operational safety.

This appendix assumes an established incident policy, authorized emergency levers, known owners, and secure evidence handling. It does not define a universal severity scheme, grant production access, replace security or forensic procedure, or authorize responders to weaken correctness and privacy.

Establish command around user impact

Declare early enough that coordination can still reduce harm. Record the incident ID, start time, commander, operations lead, communications lead, scribe, approved change channel, and next update. Only the designated operations group should modify the system; observers and subject experts supply evidence through the command structure. A clear owner for each undelegated role prevents well-intended freelancing from turning one incident into several.

Begin with user populations and valuable operations, not host symptoms. For each affected journey, region, tenant class, and operation class, record the observation window and separate:

• offered demand from admitted work;
• original operations from attempts, retries, and hedges;
• completions from valid goodput—correct, timely, policy-compliant outcomes;
• terminal errors from abandoned, timed-out, or late effects; and
• latency from freshness, durability, and business or safety impact.

Label each statement observed, inferred, modeled, or unknown. A fleet-average p95 can hide a failed region, premium tenant, or write path. A server’s 200 response is not valid goodput if the authoritative effect was duplicated, lost, or completed after the caller abandoned it.

Write the protected invariants beside the impact board. Examples include no duplicate charge, durable accepted order, authorization enforcement, monotonic cursor, bounded data loss, and no unapproved sensitive capture. Emergency speed does not suspend them. If a mitigation trades functionality for availability, name which optional result may degrade and who approved that boundary.

Find the loop before chasing the component

Acute performance failures often move through phases. The first constraint may be a slow dependency; the next may be a saturated connection pool; retries then multiply arrivals; failover shifts work; cache eviction or cold capacity changes service demand; recovery work competes with foreground traffic. Asking “what is the bottleneck?” without a timestamp assumes a static system.

Maintain a small feedback-and-constraint board:

Positive feedback means the consequence reinforces its cause. A slow dependency causes timeouts; timeouts trigger retries; retries consume more dependency and caller capacity; queues age; more calls miss deadlines. Autoscaling can join the loop if cold instances increase cache misses or connection churn. Failover can move more demand than the receiving failure domain can safely serve.

Use one discriminating change when possible. If retry suppression reduces attempt rate and queue-age growth while original demand is steady, it supports the loop hypothesis. It does not prove why the dependency slowed. Preserve that distinction in the timeline.

Stabilize useful work with bounded actions

The best first action is not the cleverest diagnosis. It is the reversible action with a known mechanism, authorized blast radius, protected invariants, and observable stop condition. Common classes are:

For every action, log the observed state, proposed mechanism, expected user effect, correctness and blast-radius risk, approver, rollback or stop signal, and result. Avoid changing queue size, concurrency, retry count, timeout, autoscaling, and routing at once. The graphs may move, but causality and safe reversal disappear.

Protect control planes and evidence paths. Reserve access for incident operations, bound high-cardinality telemetry, and keep an out-of-band communication and change path. Do not restart, reschedule, flush, or fail over merely to obtain a clean dashboard. Such actions can erase the only state that distinguishes overload, deadlock, quota exhaustion, or dependency waiting.

Capture what the next state will destroy

Evidence has a half-life. Autoscaling changes topology; a restart erases process and queue state; failover changes locality; drain changes age distributions; cache refill changes service demand; configuration systems overwrite the previous value. Capture the smallest decisive record before the action when doing so does not materially increase harm.

Prioritize:

1. the command and decision log with synchronized timestamps;
2. offered, admitted, completed, valid-goodput, error, and latency populations;
3. queue age/depth, concurrency, pool occupancy, resource saturation, and progress;
4. retry, timeout, cancellation, hedge, duplicate, and dependency distributions;
5. topology, versions, configuration, feature flags, quotas, deploys, and failure events; and
6. bounded traces, profiles, logs, packet captures, or runtime state selected by the current hypothesis.

Each artifact needs scope, clock, query or command, tool/version, collection effect, access class, retention, owner, and immutable location or hash. Screenshots can orient a reader but usually omit buckets, labels, excluded intervals, and query revisions. Preserve machine-readable evidence and the query that selected it.

Volatile capture is not permission to improvise forensics. Collection changes a live system and may expose credentials or personal data. Follow preapproved security, privacy, legal, and evidence-custody rules. If collecting a profile could worsen user harm, record why it was deferred. If evidence was unavailable, retain the gap rather than replacing it with certainty.

Read the four diagnostic lanes together

The field checklist is compact because responders need retrieval, not a textbook.

Queue. Inspect oldest age before depth: a bounded queue can still contain work too old to matter. Compare arrival, admission, useful completion, and discard rates; priority mix; dead letters; retention; and net drain. Ask whether the unit is an original operation, attempt, message, byte, or batch.

Saturation. Pair utilization with progress. Check CPU runnable time and throttling, memory pressure and pauses, worker and connection pools, storage latency/space/IOPS, network loss/bandwidth, locks, quotas, and control-plane reachability. A resource at 100% can be healthy if useful throughput is stable; 40% fleet CPU can coexist with a saturated shard or pool.

Retry. Measure attempts per original by layer and reason. Inspect remaining deadlines, retry budgets, backoff and jitter, hedges, cancellation, duplicates, and late effects. A shorter timeout can reduce a latency graph while increasing attempts and lowering goodput. Never extend deadlines merely to hide queueing.

Dependency. Split latency and errors by dependency and request class. Inspect quotas, throttling, pools, fan-out, fallback, health-signal validity, ownership, and correlated changes. A dependency marked healthy may still violate the caller’s deadline or serve only its cheap operation class.

Keep correctness as a fifth lane through all four: valid-goodput denominator, durable acknowledgments, ordering, uniqueness, stale or partial results, and reconciliation lag.

Build a timeline that preserves uncertainty

Use UTC plus the clock source and known offset. A useful row contains user impact, demand and goodput, queue or current constraint, observation or change, decision owner, and evidence link. Record actual—not planned—times for detection, declaration, mitigation, restoration, and recovery.

Do not rewrite an early hypothesis after the outcome is known. “18:08 inferred retry amplification from attempts/original; 18:10 suppression began; 18:13 queue-age growth stopped” retains the reasoning chain. “Retries caused the incident at 18:08” claims more than the incident team knew and may be false: retries can amplify a failure without initiating it.

The timeline should explain why the constraint changes. That record becomes the bridge from acute command to Chapter 58’s deeper response method and Chapter 74’s causal reconstruction.

Prove recovery, not a crossing

Recovery requires both new work and retained work to remain safe. Define exit bands for:

• user outcome and valid-goodput populations;
• latency, freshness, and terminal errors;
• queue age, backlog, and net drain;
• retries, dependencies, and failure confirmation;
• resource saturation and control-loop stability;
• correctness, durability, and reconciliation; and
• repair, replay, rebalance, cache refill, or deferred maintenance.

If backlog is B useful operations, new arrivals are λ operations/s, and useful recovery completion capacity is X_r operations/s, then

T_drain = B / (X_r - λ)

is a first drain estimate only when X_r > λ, units match, service demand remains comparable, and no other constraint or priority policy invalidates the rate. Recovery that violates the foreground objective is not safe merely because the backlog shrinks.

Require a stability dwell longer than the relevant control delays. Include autoscaling, cache, election, compaction, repair, and retry horizons. Re-enable temporary features one bounded step at a time, with the same impact and stop signals. Close acute command only when exit conditions hold or accountable owners explicitly accept a named residual state.

Mercury: the graph that lied by omission

At 18:08 UTC, Mercury observed 4,050 original checkouts/s, 3,710 admitted/s, 3,210 valid base checkouts/s, 8.1% terminal errors, and base p99 of 1.8 s. Attempts had reached 6,900/s—1.70 per original. Enrichment dependency p99 was 900 ms, its connection pool was full, and oldest base queue age grew 180 ms each second.

The command team captured its counters, distributions, retry reasons, topology, configuration, deploy markers, and bounded trace sample. At 18:10 it disabled new optional enrichment and applied the already approved retry cap. It did not extend deadlines or add cold capacity. By 18:13, attempts fell to 4,180/s, valid goodput rose to 3,680/s, and queue-age growth approached zero.

That was evidence of stabilization, not root cause or recovery. Exit required ten continuous minutes with at least 99.95% valid goodput among admitted authorized checkouts, base p99 no more than 300 ms, oldest queue age below 100 ms and falling or empty, attempts/original below 1.05, no duplicate effects, dependency p99 below its incident band, and no new saturation. At 18:19 the dwell was incomplete, so command stayed open and enrichment remained disabled under a named owner.

Tabletop exercises

Choose the least dangerous first move. A zone failover has pushed the receiver to 82% CPU, oldest queue age grows 90 ms/s, and attempts/original is 1.7. Compare moving more traffic, adding cold instances, suppressing retries, and shedding optional work. State the mechanism, protected operation, stop signal, and evidence captured before acting. Reject any answer based only on lowering p99.

Preserve a disappearing state. A restart is likely to restore service but will erase worker stacks, in-memory queue ownership, and pool waiters. Decide which bounded capture is safe before restart, which evidence can be reconstructed, and which local security authority is required. If capture risks extending harm, log the trade rather than delaying automatically.

Refuse premature closure. Errors are normal and latency is inside target, but a 1.2-million-item backlog drains at 3,000/s while new work arrives at 2,800/s. Compute the first drain estimate, identify assumptions, and define a stability dwell. The 100-minute estimate is invalid if repair changes service demand or priority work starves.

Command card

1. Declare, assign command roles, and restrict changes to the operations path.
2. Bound user populations, valuable operations, valid goodput, and protected invariants.
3. Align demand, queue age, constraint, retries, dependencies, and changes on one clock.
4. Rank feedback loops; run the least risky discriminating check.
5. Apply one authorized, reversible stabilization action with a stop signal.
6. Capture expiring evidence with scope, collection effect, custody, and access controls.
7. Validate new work, retained work, correctness, control loops, and deferred recovery through a dwell.
8. Hand off the timeline, unknowns, temporary levers, residual risk, and corrective actions.

The decision rule is: during acute impact, first protect valuable correct work and stop confirmed positive feedback with the smallest authorized action; preserve decisive volatile evidence before state changes; declare recovery only after user outcomes, queues, dependencies, correctness, and recovery work remain inside explicit bands for the required dwell.

Primary references and transfer notes

• Google SRE, Managing Incidents motivates explicit command roles, a live incident record, coordinated operational work, handoff, and preserving evidence for later analysis. Adapt roles to local policy rather than copying organization names.
• Google SRE, Addressing Cascading Failures explains overload, long queues, missed deadlines, retries, load shedding, and cascading failure mechanisms. Its examples are mechanisms, not universal thresholds.
• Microsoft Azure Architecture Center, Retry Storm antipattern describes how frequent retries impede recovery and how bounded retries, backoff, throttling, and circuit breaking can limit amplification. Product examples do not define Mercury’s policy.
• NIST SP 800-86 explains that volatile collection decisions should be made quickly and that collection changes the system. It is computer-forensics guidance; local legal, security, and privacy policy controls its application.

Acute command should leave more than a timeline. Feed the observed operation mix, amplification, failed capacity, repair demand, and recovery rate into Appendix I so the next capacity commitment can turn red on the constraint that actually governed the incident.