Appendix G: Data Incident Runbook Template

Purpose

Use this runbook when production data may be wrong, stale, missing, duplicated, unavailable, exposed to the wrong audience, or too slow for a promised user journey. The database can be healthy while a deletion pipeline, event consumer, search index, warehouse model, cache, backup, or export is causing harm.

Open a copy as soon as the symptom is credible. Do not wait to know the cause. The response must first stop new harm, then preserve the evidence needed to find the affected boundary and repair it safely. Record commands and decisions as they happen: an undocumented correction can become the next incident.

Replace every prompt below. Keep links to logs, queries, scripts, approvals, and evidence in the incident system rather than copying secrets or sensitive records into the runbook.

Data Incident: [Symptom, Not Assumed Cause]

Declaration

• Incident ID and channel:
• Detected at / declared at:
• Trigger: alert, user report, reconciliation failure, access anomaly, failed backup or restore, deployment, migration, schema change, backfill, or another concrete signal
• Observed symptom: what is wrong, missing, stale, duplicated, unavailable, exposed, or slow
• Critical user journey or decision affected:
• Current severity and reason:
• Incident commander:
• Technical owner of the affected invariant:
• Command runner / independent verifier:
• Communication owner / next update time:

Treat the incident as critical when source truth for a critical journey is corrupt or unavailable, data loss may be unrecoverable, or privacy, regulated, financial, or legal harm is plausible. Page the incident commander and the relevant security, privacy, legal, compliance, or finance owner immediately. A broad customer-visible error, duplicate side effects, a recovery target at risk, or important derived data outside its contract is normally high. Bounded, recoverable inconsistency may be medium; a cosmetic issue with no material decision impact may be low. Raise severity when scope is unknown and the credible worst case demands a faster response.

Contain Before Repairing

Ask first: Is new harm happening now? If bad writes are entering source truth, pause or fence the writer, disable the responsible feature, or reject the risky command path. If a consumer is duplicating effects, stop it without discarding offsets or idempotency state. If a derived answer is wrong, route critical reads to trustworthy authority, show its freshness, or disable the feature. Stop a harmful migration or backfill at its checkpoint. Revoke an unauthorized token, role, or export path while preserving the audit trail.

Choose the smallest reversible action that stops harm. State what remains available in degraded mode and what users must not trust. A stopped alert is evidence of containment only when the corrupting or exposing path is known to have stopped.

• Containment decision and approver:
• Action or exact procedure:
• Authority for data while contained:
• Features, writers, consumers, jobs, or deploys paused:
• User guidance or degraded mode:
• Evidence that new harm stopped:
• Conditions for resuming each paused path:

Establish the Chain of Truth

Locate the authority before changing records. Is the source of truth itself wrong, or are only derived stores wrong? Which invariant or data contract settles disagreement? If authority is uncertain, keep writes and repairs contained until its owner resolves the uncertainty.

Trace one affected fact through the write request, transaction or event, source record, log or stream position, transform, derived stores, and the user-visible answer. Compare it with one known-good fact. This trace often separates a corrupt source from a broken projection, stale cache, duplicated consumer, or misleading report without requiring a broad speculative repair.

• Source of truth and owner:
• Failed invariants or contract terms:
• Affected fact trace:
• Known-good comparison:
• First bad boundary:
• Recent relevant changes: deploy, schema, configuration, permissions, traffic, migration, backfill, model, or dependency
• Current cause hypothesis and contradicting evidence:

Preserve Expiring Evidence

Preserve evidence before a destructive repair, restore, redeploy, replay, or credential rotation changes it. Record storage location, retention deadline, access restriction, collector, and timestamp. Minimize copied customer data; use identifiers and approved evidence stores.

Capture representative bad record keys and versions, before-and-after values where permitted, event and tenant IDs, timestamps, requests, idempotency keys, transaction IDs, logs, deploy versions, and feature-flag state. For streams, record topics, partitions, offsets, consumer groups, checkpoints, and relevant dead-letter entries. For access incidents, preserve identities, roles, token identifiers, query or export IDs, approvals, and immutable audit logs through the security process. For recovery, record the backup or snapshot ID, restore point, log position, retention deadline, and last known successful drill.

• Evidence index and restricted location:
• Earliest expiry or overwrite time:
• Operator actions already taken:
• Missing evidence and consequence:

Bound the Impact

Investigate by boundary rather than extrapolating from the first report. Establish the earliest and latest affected times and whether the issue is still active. Identify affected tenants, accounts, regions, roles, cohorts, and data classes. Name every table, topic, file, index, cache, model, export, backup, log, and source version in the path. Then follow lineage outward to consumers, dashboards, alerts, support tools, customer features, and decisions that used the data.

• Affected time window / confidence:
• Users, tenants, regions, roles, or cohorts / count:
• Source and derived data objects:
• Operations and versions involved:
• Downstream consumers and decisions:
• Privacy, deletion, retention, contractual, financial, or legal impact:
• Known unaffected boundary and evidence:
• Unknowns that could change severity or repair:

Do not describe an absence of reports as an unaffected boundary. Support volume, one clean sample, or aggregate row parity cannot prove that every tenant, deletion, authorization rule, or business invariant is intact.

Choose a Repair From Authority

The invariant owner and incident commander approve the repair. Prefer a deterministic procedure that can be reviewed, repeated safely, stopped at a checkpoint, and audited afterward.

When source truth is healthy, replay a consumer or rebuild an index, cache, materialized view, warehouse model, or feature table from it. When source truth is corrupt or deleted, use the recovery plan to compare point-in-time restore with repair forward; record the approved recovery point and expected loss. Preserve append-only ledgers and audit histories with compensating entries rather than rewriting their past. Use a targeted correction only when the affected keys and correct values can be derived deterministically. An access incident also requires verified revocation, exposure scoping, and the security or privacy response; changing the data alone is not a repair.

• Selected repair and alternatives rejected:
• Input authority / recovery point:
• Affected-key or partition selection:
• Script, query, rebuild, replay, restore, or compensation procedure:
• Idempotency rule and progress checkpoint:
• Throttle and production load limits:
• Stop condition / safe response:
• Peer review and approval:
• Expected data loss, stale interval, or residual impact:

Test the procedure on an isolated copy or a bounded cohort when time and impact allow. Never let urgency turn an unbounded update into the first test.

Response Ledger

Use the ledger for actions that change containment, evidence, production data, access, or user impact. Link verbose command output elsewhere.

Prove Correctness Before Resuming

An alert returning to green is not correctness proof. Show that the harmful write, stale pipeline, exposure path, or corrupting process has stopped and that source truth is correct or restored to the approved point. Run the critical invariants across the entire affected boundary. Reconcile business totals and inspect representative user-visible records, including boundary and previously bad cases. Verify deletion, privacy, authorization, and audit behavior where relevant.

For rebuilt or replayed systems, compare counts by meaningful cohort, canonical checksums where semantics allow them, freshness, lag, and sampled answers against authority. Keep unrepaired derived systems explicitly stale or out of service. Observe errors, latency, freshness, and lag for a declared window long enough to expose recurrence before resuming paused paths in dependency order.

• Source-truth proof:
• Invariant and business reconciliation:
• Derived-system proof:
• Deletion, privacy, and access proof:
• User-journey checks:
• Observation window and monitoring evidence:
• Independent verifier / result:
• Residual uncertainty accepted by / reason:
• Resume decision, order, and approver:

Communicate What Is Known

Each incident update should state the current severity, affected boundary, user-visible consequence, containment, repair status, owner, remaining unknowns, and next update time. Tell dependent teams whether their data is wrong, stale, unavailable, or safe to consume. Give support the affected period, customer-safe guidance, and escalation path.

Route privacy, regulated, contractual, financial, or legal impact through the named security, privacy, compliance, finance, and legal owners. Customer messages must use the normal approval and communication path. Do not speculate about cause or disclose sensitive evidence merely to make an update sound complete.

• Latest internal update:
• Dependent-team instruction:
• Support guidance:
• Required specialist notifications and owners:
• Approved customer communication / next update:

Handoff and Follow-Through

End active response only when containment is stable, the approved repair is complete or safely owned, correctness evidence has been accepted, paused paths have an explicit state, communications are assigned, and every remaining risk has an owner and due date.

• Resolved / monitoring / handed off at:
• Final affected boundary and user impact:
• Repair and verification evidence:
• Paused or degraded paths still in effect:
• Residual repair, communication, or customer remediation:
• Owners and due dates:
• Review date and facilitator:

For every high or critical incident, strengthen the earliest reliable guard: invariant enforcement, monitoring, a regression test, repair tooling, access control, lineage, or a data contract. Update the ADR, migration runbook, or recovery plan whose assumption failed. Schedule a drill when recovery, rebuild, replay, credential access, or communication depended on luck or undocumented knowledge. The review is finished when it changes how the next incident is prevented or handled, not when it has retold this one.