Appendix F: Migration Runbook Template

Purpose

A migration runbook turns a proposed change into a sequence an execution team can stop safely. Use it when a change can alter source truth, derived data, schemas, indexes, partitions, queues, streams, files, access rules, retention, or recovery behavior.

Prepare the runbook before the execution window. The technical and operational owners must review it, as must any team whose service can be stopped, slowed, or corrupted by the work. At every irreversible boundary, the runbook should answer four questions: what must already be true, what evidence permits the next action, what condition stops the work, and which route returns users to a safe state.

Copy the headings below into the migration’s repository. Replace every prompt and remove examples that do not apply. Commands should be exact enough to review, but keep credentials and other secrets in the approved secret store.

Migration Runbook: [Migration Name]

Decision and Boundary

• Outcome: the user, operational, compliance, cost, or reliability result this migration must produce
• Scope: source and target systems, tables, streams, indexes, jobs, caches, dashboards, exports, and contracts affected
• Non-goals: adjacent changes deliberately excluded
• Source of truth during each phase: unchanged / dual-written / moved, with the authority that settles disagreement
• Invariants: links to the register entries that must survive
• Risk: Low / Medium / High / Critical, justified by correctness impact, blast radius, and reversibility
• Window: approved start, expected duration, freeze period, monitoring period, and last time rollback remains credible
• Owners: migration lead, command runner, independent verifier, incident commander, customer communicator, and cutover approver

State the irreversible boundary in one sentence. For example: After the old writer is retired, rollback requires replay rather than a traffic switch. If the team cannot locate that boundary, it does not yet understand the migration.

Risks That Can Stop the Work

Record only risks that change the plan. A useful row connects an early signal to a named stop decision and a recovery action; “monitor closely” is not a control.

Evidence Required Before Start

No blank item is an implied approval. For each item, link the evidence, name the owner who checked it, and record the time of approval.

• Restore evidence: affected source truth has a recent usable backup; the restore target, credentials, procedure, validation queries, expected time, and last successful drill are known.
• Rollback evidence: the traffic switch, feature flag, compatibility path, schema reversal, or replay procedure has been exercised outside production.
• Load evidence: query plans, lock behavior, batch size, throttling, write amplification, and replica impact have been tested against a credible load.
• Observation: migration progress, critical invariants, read and write latency, errors, freshness, lag, and cost are visible. Alerts reach the execution team without requiring someone to stare at a dashboard.
• People and access: all named owners are present; the command runner has time-bounded access to execute, pause, switch, and inspect; support and dependent teams know the window and expected symptoms.
• Change boundary: conflicting deploys, schema changes, and heavy jobs are frozen or explicitly accepted by the migration lead.

Record the preflight decision:

• Ready / not ready:
• Approver and time:
• Known uncertainty accepted for this window:

Execution Ledger

Use one row per action. Put the exact command, query, dashboard, or procedure link in the action or evidence cell. A row is not complete until the verifier records its result. The command runner does not improvise sequence during the window.

Add rows for schema expansion, consumer upgrades, final delta capture, writer switching, regional rollout, or other real boundaries. Do not copy a sample step merely to make the ledger look complete.

Backfill and Live-Write Plan

Name the stable chunk key—tenant, ID range, partition, event-time interval, or file prefix—and show that each chunk can be selected without an unbounded scan under production load. Record the initial chunk size and the lock duration, write amplification, target throughput, and verification time that justify it.

Define the throttle in rows, bytes, messages, or write IOPS per interval. Name the signals that automatically slow or pause it. Each chunk must be safe to retry without duplicate effects; record the deterministic upsert, version replacement, or other idempotency rule.

The durable progress marker should contain the input range, transform version, counts or checksum, start and finish times, result, and verifier. Bound retries for transient failure and quarantine deterministic failures instead of looping forever.

Finally, describe how writes and deletes that arrive during the backfill reach the target: dual-write, CDC, replay, or a final delta pass. Include tombstones, redactions, and policy deletes. A target that contains every insertion but misses required deletion is corrupt.

• Chunk and selection query:
• Initial size and throttle:
• Automatic pause signals:
• Idempotency rule:
• Progress store and resume command:
• Retry and quarantine path:
• Live-write and delete capture:

Proof Before Cutover

Counts establish quantity, not correctness. Choose independent evidence that can expose different failures:

• compare counts by meaningful cohort such as tenant, partition, state, and time range;
• checksum canonical fields where the transform is deterministic;
• run critical invariants, including tenant and authorization boundaries;
• compare source and target answers for representative shadow traffic;
• reconcile business totals such as balances, entitlements, or active subscriptions;
• measure freshness at the consumer-visible boundary; and
• prove deletion, redaction, and access behavior on the target.

For every method, record the query or tool, scope, expected result, tolerance, actual result, verifier, and evidence link. Explain every accepted mismatch; an unexplained difference does not become safe because its percentage is low.

Cutover Decision

Cutover can proceed only while preflight evidence is current, the verifier has accepted the proof, required owners are present, rollback remains possible, and the observation window is long enough to reveal likely harm. High-risk migrations should move through deliberately chosen cohorts rather than a single switch.

• Current proof accepted by:
• Rollback tested at:
• Rollback remains available until:
• Canary cohort and stages:
• Observation window per stage:
• Cutover approver and time:

Stop when a critical invariant fails, source truth becomes uncertain, rollback evidence is missing, or user-facing errors cross the declared threshold. The short distance to the final step is not evidence to continue.

Rollback or Repair

Rollback is a data decision as well as a command. Routing reads back is enough only when the old path still holds trustworthy, compatible data.

If the target read path is wrong while source truth is healthy, return reads to the source or previous derived store and repair the target offline. If target writes are corrupt but source writes remain correct, disable the target writer and replay from authority. If source truth itself may be corrupt, stop harmful writes, preserve evidence, and enter the incident and recovery plans. When an old application version cannot read the new schema, preserve or restore the compatibility layer; reverse the schema only when that operation has been proved safe.

• Rollback trigger and decision owner:
• Command or procedure:
• Authoritative data after rollback:
• Writes accepted during reversal:
• Reconciliation or replay required:
• User communication:
• Evidence that service is safe again:

If cleanup has already removed the old path, call the action a recovery or repair, not a rollback. Escalate through the incident plan and record the lost reversibility as a review trigger.

Acceptance and Cleanup

Run the acceptance checks immediately after cutover and again after the agreed monitoring window. Critical invariants must pass; read and write SLIs, freshness, lag, and cost must remain within their declared bounds; business totals must reconcile; and access, deletion, support, billing, privacy, and audit signals must show no unexplained harm.

Record whether the rollback window is still open, explicitly closed by an approver, or replaced by a recovery plan. Only then may cleanup begin. Remove old code paths, flags, migration credentials, staging data, queues, indexes, dashboards, alerts, and obsolete contracts in separately reviewable steps. Archive the evidence required for audit, incident review, and later migration work.

• Acceptance evidence and verifier:
• Monitoring window completed:
• Rollback window closed by and at:
• Cleanup owner, steps, and verification:
• Evidence archive:

Open a postmortem or migration review if a stop condition was reached, rollback was executed or found unavailable, a critical or high invariant failed, the migration materially burned a user-facing SLO, later evidence exposed a gap in verification, cleanup harmed a consumer, or operators needed undocumented knowledge to finish the work. The review should change the next runbook, not merely describe this one.