On-Call, Runbooks, Incidents, and Postmortems

The Page That Arrived Too Late

At 02:17, customer support reports that recently renewed subscriptions still appear expired. The renewal API is healthy. Database CPU is ordinary. The overnight schema migration completed, and its backfill dashboard is green.

The page arrives four minutes later: the number of active subscriptions in the billing ledger no longer agrees with the entitlement index. That alert is more useful than a page for CPU or queue depth because it names the broken promise. Customers who paid may be denied service, and a completed job may have written incorrect facts.

The incident began before either signal. A migration added a normalized status column, application code started reading it, and a backfill populated it from legacy fields. Rows with a particular cancellation history were classified incorrectly. The job completed exactly as written. Its success metric measured processed rows, not preserved meaning.

This is what data-system on-call must control: not merely whether a component runs, but whether useful facts remain trustworthy while people diagnose, mitigate, repair, and communicate under pressure. The response is complete only after user harm is bounded, corrupted state is identified, the safe path is restored, and independent evidence shows that the facts are right again.

Readiness Is a Set of Usable Promises

The subscription incident can be handled safely only if several decisions already exist.

The path has an operating owner who can stop the backfill and approve a read-path change. The on-call rotation has read-only access to compare the ledger with the index, and a tested break-glass path for the migration controls. The dashboard shows renewal success, entitlement freshness, mismatch counts, backfill rate, database pressure, and the deployed application and schema versions. The alert links to a runbook written around the customer symptom. Support knows how to recognize the failure and where approved updates will appear. The team knows who can authorize degraded operation and who must join if payment or personal data may have been exposed.

Without those promises, the first responder spends the incident discovering ownership, requesting access, decoding infrastructure graphs, and asking whether a completed job can be stopped. A document that cannot answer those questions at 02:21 is inventory, not readiness.

Alerts deserve the same operational test. Page a person for user impact or a fast-moving risk that requires action: failed writes, stale customer-visible facts, an invariant drifting, a critical backup failing, a restore-readiness regression, or disk exhaustion approaching faster than the team can respond. Queue depth alone rarely provides a decision. Queue age against a freshness objective, its drain rate, the affected workflow, and the action available to the responder might.

Readiness therefore joins the work of the previous two chapters. An SLO supplies the promise and the trigger. A recovery plan supplies the safe return path when ordinary mitigation is no longer enough. On-call turns both into decisions that a tired engineer can actually make.

Open One Decision Channel

At 02:23, the responder declares an incident. The point of declaration is coordination, not ceremony. One channel becomes the source of truth; one person owns the next decision.

The incident commander maintains impact, severity, decision order, escalation, and risk acceptance. A subject expert investigates and proposes technical actions. A communications owner gives support and other stakeholders accurate updates. A scribe records times, observations, commands, rejected hypotheses, and decisions. A small team may combine roles, but it should still name the work. Diagnosis degrades quickly when the same person is also answering executives, approving a risky repair, and reconstructing the timeline from memory.

The first statement is deliberately narrow:

Renewed subscriptions are being denied access because the entitlement index disagrees with the billing ledger. The earliest known affected write is 01:48. We have stopped the status backfill and are measuring the affected set. Renewals continue, but entitlement updates are in a degraded path. The next update is at 02:35.

This statement separates impact from hypotheses. “Database problem” would hide the customer consequence. “The migration corrupted subscriptions” would claim more than the evidence yet supports.

Severity follows impact, sensitivity, correctness risk, and recovery uncertainty. A delayed internal report and an incorrect payment ledger may both involve the same queue, but they require different command posture. The latter may justify an early write freeze, closer business involvement, and a lower tolerance for speculative repair.

The First Thirty Minutes

The first response has four jobs: bound harm, preserve evidence, establish the affected set, and choose a reversible mitigation.

First, stop the process that can enlarge the incident. The backfill is paused at a recorded checkpoint. Its workers and version are preserved; no one reruns it. Recent deploys, migrations, feature flags, and batch jobs are recorded. The team does not yet know whether the migration, the new reader, the backfill logic, or their interaction is responsible, but it knows where change entered the system.

Second, inspect safely. Read-only queries compare renewed subscriptions in the ledger with their normalized status and indexed entitlement. The team samples by tenant and by cancellation history, then turns the pattern into a query for the full affected set. It checks whether new renewals are also wrong. The goal is a bound that can be revised: which facts, which customers, which versions, which time window.

Third, reduce user harm without destroying the evidence needed for repair. The team routes entitlement checks for recently renewed accounts to the billing ledger, a slower but authoritative path. It does not overwrite the index in place, delete suspicious rows, or replay the whole backfill. Each of those actions would mix repair with diagnosis and make the original affected set harder to prove.

Finally, verify the mitigation in customer terms. A known affected renewal now receives access. New renewal-to-entitlement latency remains inside the agreed degraded bound. Mismatch growth stops. Database load stays below the stop condition for the fallback path. A cleared alert alone would prove none of these things.

The incident timeline keeps observation and interpretation apart:

This is one of the few places where a table earns its space: responders need to compare time, evidence, and consequence without turning chronology into narrative guesswork.

The Runbook Governs Risk

A useful runbook begins with the page and the customer-visible behavior: “renewed subscriptions may remain inactive because the entitlement index is older or disagrees with the billing ledger.” It names the authoritative fact, the allowed freshness bound, and the systems that can serve or mutate it.

Its first checks are read-only: SLI and invariant queries, lag and checkpoint views, recent changes, sample workflows, lock state, and resource forecasts. Commands that mutate data are separated from diagnosis. Every risky action states who may approve it, what evidence must exist first, how to reverse or stop it, and what result would show that it helped.

The runbook must also say what not to do. Do not rerun a failed backfill until its idempotency and logic are established. Do not replay messages until duplicate handling and the source checkpoint are known. Do not delete unknown files from a database host under disk pressure. Do not promote a replica whose completeness is uncertain. Do not paste sensitive rows into an incident channel. Under stress, a forbidden action with a reason can prevent a second failure more effectively than another paragraph of explanation.

Stop conditions are the runbook’s sharpest controls. They tell the responder when to halt a migration, freeze writes, abandon a fallback that is overloading the source of truth, move to the recovery plan, preserve forensic evidence, or bring in security and privacy owners. A mitigation without a stop condition is an experiment allowed to run indefinitely in production.

The particular mechanism changes, but the governing questions transfer. Replica lag requires the team to distinguish stale reads from failed writes and to know when failover would lose or hide acknowledged facts. Lost messages require a replayable source, trustworthy checkpoints, duplicate handling, and reconciliation across every downstream copy. Disk pressure requires a time-to-exhaustion forecast and exact knowledge of which data is disposable or recovery-critical. A suspected exposure requires containment without evidence destruction and a controlled handoff that identifies data classes, subjects, access path, time window, and evidence location. Slow queries, lock storms, failed restores, and corrupt caches all return to the same discipline: define impact, inspect safely, contain reversibly, and verify the fact the user depends on.

Recovery Is Not Yet Proof

By 03:06, the team has reproduced the classifier error against a copy of the affected rows. The repair is a versioned transformation that selects only rows matching the proven defect, writes to a quarantine-backed staging table, and emits before-and-after identifiers for reconciliation. A dry run reports the expected cardinality and no rows outside the affected set.

The repair then proceeds in bounded batches. Each batch has a rate limit, checkpoint, invariant check, and stop condition. The entitlement index is rebuilt from corrected authoritative state rather than patched independently. That choice is slower, but it avoids creating two competing versions of the truth.

Service recovery is established when the ordinary read path can resume without violating its latency and freshness objectives. Data recovery needs separate proof. The team compares source and derived counts, checks every affected identifier, samples customer workflows, verifies that no new mismatch appears, and accounts for writes that arrived during the incident. Only then does it close customer impact.

This distinction matters beyond backfills. A restarted consumer can be healthy while messages remain missing. A caught-up replica can still have served stale decisions. A restored database can be online before derived stores are rebuilt or business totals reconcile. Green infrastructure is an observation. Restored trust is a claim that needs independent evidence.

A Postmortem Must Alter the System

The incident’s trigger was a classifier that mishandled one cancellation history. Stopping there would make the postmortem nearly useless. Normal work allowed that defect to reach customers because the migration combined a new schema, a read-path switch, and an unverified backfill; the job measured throughput rather than invariant preservation; the correctness alert was slower than support; and the runbook did not name the billing-ledger fallback.

The timeline is evidence for that explanation. It should include uncertainty, dead ends, changes, commands, decisions, communications, mitigation, and verification. Its purpose is to show how the system behaved, including its human controls, without relying on the memories of the people who happened to be awake.

Actions then need a theory of impact. “Be more careful” cannot be tested. “Add monitoring” does not say what should wake a human. Strong actions change a condition that shaped the incident:

• gate future backfills on the ledger-to-index invariant and stop automatically when the mismatch count grows;
• separate schema expansion, compatibility deployment, bounded backfill, verification, and read cutover;
• add cancellation-history cases to the production-shaped migration fixture;
• place the authoritative-read fallback and its load stop condition in the runbook;
• shorten detection by paging on a fast burn of the correctness SLO;
• automate affected-set capture and batch reconciliation so repair no longer depends on ad hoc queries.

Each action has an owner, due date, expected effect, and a way to verify completion. The incident review remains open until the high-risk controls have changed production or an accountable owner explicitly accepts the residual risk. A backlog of unreviewed postmortem tickets is not learning; it is a record of lessons the system is prepared to teach again.

Recurring manual work deserves particular suspicion. If responders repeatedly route reads, clear a known lock, rebuild an index, or reconcile the same mismatch by hand, the procedure should become a guarded operation with preview, approval, audit, rollback, and verification. Automate the safe repetition. Remove pages that never change a human decision. Keep judgment where consequences differ from case to case.

Write the Runbook From the Alert Outward

For a critical data path, write the runbook in the order a responder will need it:

1. Trigger and impact. Quote the alert or recognizable user report. Name the workflow, data promise, likely scope, and whether availability, freshness, correctness, durability, or exposure is at risk.
2. Authority and boundaries. Identify the source of truth, derived copies, allowed staleness, owners, relevant versions, and the limit beyond which the incident must escalate.
3. First safe checks. Give exact dashboards and read-only queries that establish affected users, regions, tenants, partitions, time window, recent changes, checkpoints, locks, and resource runway.
4. Containment. Offer the smallest reversible actions that stop growth or reduce harm. State approval, reversal, load limit, and stop condition beside each action.
5. Forbidden actions. Name tempting shortcuts that could lose facts, duplicate work, destroy evidence, expose data, or close the only recovery path.
6. Recovery and proof. Separate rollback, roll-forward, replay, restore, rebuild, and repair from the independent reconciliation or workflow evidence required to trust the result.
7. Command and communication. Name incident roles, escalation targets, update cadence, customer-facing owner, security or privacy handoff, and the approved source of truth.
8. Follow-through. Link the timeline, postmortem, runbook correction, drill, alert tuning, toil removal, and hardening actions.

Test the runbook with someone who did not write it. Give that person a customer-facing replica-lag scenario: the primary accepts writes, one replica is stale, and a recent migration is still running. They should be able to determine affected read paths, measure freshness by replica and tenant, choose a safe routing change, state when the primary fallback must stop, explain why failover may be unsafe, and prove that users see fresh enough data again.

Then add a failed migration and a bad backfill. If the exercise cannot reveal who stops the job, how incorrect rows are bounded, which facts remain authoritative, whether the system rolls back or forward, and what evidence closes the incident, the runbook is not ready. The drill has found the gap while the team still has time to fix it.