Appendix H: Recovery Plan Template

Purpose

Use this plan when a production data system must recover from deletion, corruption, a failed migration, dependency loss, operator error, compromised access, or a site or region outage. Complete it before the incident and revise it with evidence from every drill and recovery.

A backup is only a possible recovery source. This plan must show which business facts can be recovered, to what point, in what time, by whom, and in what order. It must also define the proof required before users or downstream consumers can trust the restored system.

Keep commands, infrastructure definitions, validation queries, dashboards, and restricted evidence in their controlled systems. Link them here. Replace every prompt; an empty section is an unknown recovery dependency, not documentation.

Recovery Plan: [System and Business Capability]

Boundary and Authority

• Plan owner / last reviewed:
• Business capability at risk:
• Critical user journeys and decisions:
• Authoritative facts and source-of-truth systems:
• Included components: databases, object stores, queues, streams, outboxes, indexes, caches, warehouses, exports, configuration, jobs, and observability
• Excluded components / owning plan:
• Business owner: accountable for continuity and accepted impact
• Technical owner: accountable for restored correctness
• Recovery lead: accountable for executing this plan
• Independent verifier: must not be the sole restore operator
• Recovery authority: may approve the recovery point, return to service, failover or failback, and acceptance of data loss
• Security, privacy, legal, compliance, or finance contacts:
• Incident channel / communication owner:

State the declaration threshold. Name the symptoms or failure boundaries that activate this plan, who may activate it when the usual owner is unavailable, and which changes stop immediately. A plausible corruption event should fence risky writers, migrations, backfills, retention jobs, and destructive automation until the recovery lead decides otherwise.

Recovery Promises by Data Class

Do not assign one RPO or RTO to a whole architecture merely because its stores share an application name. Classify the facts by business consequence. Include configuration, keys, audit evidence, and deletion state: losing any of them can make otherwise intact data unsafe or unusable.

For each promise, record who approved it and the business harm when it is missed. A derived store may tolerate complete loss only if its authority is retained long enough and its measured rebuild plus verification fits the RTO. A replica is a recovery source only for failures it does not reproduce. A cache is disposable only if no unique state has leaked into it.

Recovery Sources

Describe each source precisely enough that an operator can select it without trusting a green backup badge.

[Backup, snapshot, log archive, object version, retained event log, or rebuild source]

• Protects these data classes:
• Location, account, region, and administrative boundary:
• Format and creation mechanism:
• Schedule or continuous capture:
• Expected completion / freshness:
• Success and missed-backup alert:
• Retention and earliest available point:
• Immutability and isolation: failures, credentials, regions, automation, and malicious actions this copy survives
• Encryption and key-recovery path: include key owner and rotation effects
• Restore or retrieval tooling / version:
• Capacity and transfer constraints:
• Latest verified recovery point:
• Latest successful drill using this source:
• Known failure it does not survive:
• Cost owner: storage, retrieval, transfer, temporary capacity, and rebuild

Repeat this block for every recovery source. Confirm that retention is long enough to discover slow corruption, replay every required consumer, and meet audit needs without silently defeating deletion and privacy obligations.

Choose the Recovery Point

The newest available point is not automatically safe. Bound the failure first: identify its earliest possible start, last known-good state, affected data classes, and whether corruption or unauthorized access can have propagated to replicas, logs, exports, and backups.

• Observed failure / trigger:
• Failure began no earlier than / evidence:
• Last known-good state / evidence:
• Candidate recovery points and available log positions:
• Chosen point / backup, snapshot, timestamp, offset, or version:
• Why this point is outside the failure:
• Expected lost or repeated work:
• Affected tenants, transactions, events, objects, and time window:
• Reconciliation or customer remediation required:
• Recovery authority approval:

Preserve backup metadata, logs, queue offsets, deploy and migration history, schema versions, feature flags, audit records, and representative corrupt data before restore or repair changes the evidence.

Conditions Before Execution

Do not begin a production restore until the operator can answer all of these:

• Is the damaging writer, consumer, migration, credential, or automation fenced? What proves it?

• Is the selected source readable, decryptable, compatible with the target engine and schema, and isolated from the suspected failure?

• Is there enough storage, compute, network throughput, archive-retrieval time, and quota to meet the target?

• Are identity, network policy, secrets, keys, DNS, service discovery, configuration, and observability available in the recovery environment?

• Are application versions compatible with the restored schema and recovery point?

• Are source logs, outboxes, event offsets, idempotency records, tombstones, and deletion history aligned for safe replay?

• Does the team have approved, time-bounded restore and production-write access, an audited break-glass path, and vendor support access if required?

• Is an isolated restore possible? If not, who accepted the production blast radius and what is the reversal path?

• Preflight evidence / approver:

• Unmet condition and decision: stop, degrade, escalate, or proceed with explicit risk acceptance

Ordered Return to Service

Tailor this sequence to the architecture. Keep a component paused until both its prerequisites and the state it will consume have been verified. A command succeeding is progress, not correctness proof.

1. Declare and contain. Open the incident record, assign authority, fence harmful paths, preserve evidence, freeze incompatible changes, and state what users and dependent teams must not trust.
2. Establish the recovery foundation. Restore or verify identity, audited access, secrets, encryption keys, network policy, configuration, target capacity, observability, and the compatible application and schema version.
3. Restore into isolation. Provision a clean target. Restore authoritative databases and objects to the approved point. Retain the damaged environment for evidence unless the incident authority approves otherwise.
4. Prove source truth. Check schema and versions, restore completeness, invariants, referential integrity, business totals, deletion and retention rules, authorization boundaries, and representative records. Stop on an unexplained mismatch.
5. Align ordered history. Restore or locate outboxes, transaction logs, topics, archives, checkpoints, offsets, deduplication state, and tombstones. Define the replay start and end before starting a consumer. Prevent repeated external side effects.
6. Repair or rebuild derived state. Rebuild indexes, read models, caches, analytical tables, feature data, and exports from verified authority. Mark them unavailable or stale until freshness, delete propagation, and sampled answers are proven.
7. Exercise critical journeys. Test reads and writes through business boundaries, including retries, authorization, sensitive data, downstream effects, and any workflow that spans more than one restored component.
8. Return traffic in stages. Resume dependencies in the approved order. Observe errors, latency, freshness, lag, reconciliation, capacity, and invariants during a declared hold period. Keep a fast path back to the verified state.
9. Reconcile and communicate. Account for the RPO window, queued or manual work, duplicate-risk side effects, delayed deletes, stale reports, and customer remediation. Record every residual gap with an owner and due date.

For an isolated tenant or object failure, a scoped extraction and deterministic repair may be safer than rolling the entire system back. Describe how the selected records are identified, how current valid changes are preserved, and how the repair is made idempotent and independently verified.

Recovery Execution Ledger

Use this ledger during a recovery or drill. Put verbose command output and sensitive evidence in approved storage.

Record actual start and finish times separately for data restoration, correctness proof, dependency return, and useful service. Those intervals show where the RTO was spent.

Proof Required Before Trust Returns

Define the exact checks before the incident. Service startup, record count parity, or an alert returning to green cannot prove recovered correctness by itself.

• Recovery-source identity and integrity:
• Schema, application, configuration, and key compatibility:
• Authoritative invariants and referential integrity:
• Business reconciliation by meaningful cohort:
• Expected-versus-restored count, checksum, or aggregate: include where these measures cannot detect semantic corruption
• Known-good, boundary, and previously affected records:
• Critical read, write, retry, and side-effect journeys:
• Authorization, tenant isolation, privacy, retention, deletion, and audit:
• Event replay, idempotency, offsets, lag, and duplicate-effect proof:
• Derived-store freshness and comparison with authority:
• Backup and restore monitoring after return:
• Observation window and stop thresholds:
• Independent verifier / evidence / result:
• Recovery authority sign-off / accepted uncertainty:

Continuity During Recovery

• Safe operating mode: read-only, reduced features, durable queue, manual intake, alternate region, static status, or complete stop
• Capabilities available / deliberately unavailable:
• How users see freshness, delay, or restricted behavior:
• How queued or manual records are secured and reconciled:
• Support, finance, operations, and dependent-team instructions:
• Customer and regulator communication approval path:
• Trigger for changing continuity mode:

Do not offer read-only access to untrusted data or accept queued work without a durable, tested reconciliation path. State when stopping work is safer than a degraded promise.

Drill and Review Record

Run a realistic drill at [cadence] and after major changes to schema, topology, data volume, keys, providers, dependencies, or recovery tooling. Vary the scenario: a successful full restore does not prove tenant extraction, point-in-time recovery, event replay, region failover, lost credentials, or recovery without the usual owner.

• Scenario and failure boundary:
• Participants / absent primary owner:
• Plan and tooling versions:
• Recovery source and chosen point:
• Data volume and environment:
• Target RPO / observed loss window:
• Target RTO / time to restore / time to prove / time to useful service:
• Continuity mode exercised:
• Checks performed and evidence location:
• Result: capability demonstrated, partially demonstrated, or not demonstrated; explain the boundary rather than recording “pass” alone
• Surprises, manual steps, access failures, and unsafe assumptions:
• Follow-up owners and due dates:
• Next drill scenario / date:

Known Gaps and Accepted Risk

For each gap, state the failure it exposes, the data classes and user journeys affected, the likely RPO or RTO miss, the current compensating control, and the evidence that control works. Name an owner, due date, and review trigger. Risk acceptance must identify the authority and expiry date; an unowned gap is an unknown recovery promise.

Close the plan only when another operator can locate the recovery sources, obtain audited access, select a safe point, execute the ordered return, and produce the required proof in a drill. The last drill is the current measure of capability. Everything else is intent.