Appendix J: Drill Library

Claims Become Capabilities Only When Rehearsed

A backup policy is not a restore. An idempotency design is not a safe replay. A failover runbook is not a failover. Each remains a claim until the team creates the failure, follows the operating path, and preserves evidence of what happened.

Choose a drill from the system’s current risk, not from a generic rotation. A new source of truth needs restore, access, and invariant work. A migration needs compatibility and reversal work. A new derived store needs rebuild, freshness, replay, and deletion work. After an incident, rehearse the condition that would have exposed the weakness earlier.

Prefer an isolated environment with production-shaped data and controls. If a drill must touch production, name the approving authority, bound the affected tenants or paths, watch explicit abort signals, and prepare rollback before the disturbance begins.

Write the Drill Record Before Starting

Name the capability, system, and failure being tested. State the promise in measurable terms: an RPO or RTO, a freshness budget, an invariant, an access boundary, a replay property, a rollback window, or a deletion deadline. Then record:

• scope: systems, tenants, partitions, topics, indexes, dashboards, and owners;
• preconditions: approvals, test data, credentials, monitoring, abort signals, and rollback;
• steps: ordered actions with an owner for each handoff;
• proof: pass conditions chosen before anyone sees the result;
• evidence: commands, timestamps, logs, queries, traces, and decision records;
• outcome: pass, partial, or fail, without averaging away a broken invariant;
• follow-up: one owner, due date, severity, and retest trigger for every gap.

Time both machine recovery and human response. Include an on-call engineer who did not write the procedure. Undocumented knowledge is a finding, even when the system eventually recovers.

Recovery: Can the Team Recover Useful Truth?

Restore the latest scheduled backup

Restore into an isolated environment and run the business invariants from the recovery plan. Prove that the selected backup is accessible, the restore finishes within RTO, expected data falls within RPO, critical queries agree, and restored access is controlled. Keep the backup identifier, restore output, elapsed time, recovery point, and invariant report.

Recover without the original author

Give the documented plan and approved access to the current on-call engineer. Do not rescue the exercise with private instructions. The engineer should be able to select a safe recovery point, restore dependencies in order, validate source truth, and state the user-visible loss. Record every missing permission, ambiguous step, and undocumented decision.

Derived State: Can It Be Recreated Without New Damage?

Rebuild search from source truth

Create a new index rather than destroying the current rollback path. Rebuild from the authoritative source, then compare counts, sampled documents, deletions, access filters, and representative relevance queries. Preserve the source boundary, index version, rebuild logs, comparison queries, and cutover or rollback decision.

Replay one day of events

Choose a closed offset or time interval and replay it into a clean derived table. Inject at least one duplicate. Prove that checkpoints advance, retries do not duplicate effects, totals reconcile with source truth, and external side effects remain outside the replay boundary or are safely idempotent. Keep input offsets, output counts, duplicate evidence, and reconciliation results.

Rebuild a vector index after an embedding change

Write into a versioned index while the old index remains available. Prove that the intended corpus and model version are present, deletions and access filters survive the rebuild, and a fixed evaluation set meets its agreed threshold. Record the model and index versions, corpus counts, evaluation report, filter tests, and rollback decision.

Freshness and Consumption: Does Delay Become Visible?

Pause CDC

Stop change propagation long enough to cross the freshness budget. The alert should fire from observed lag or age, not merely from a dead process. Resume the path and prove that it catches up without losing deletes or creating duplicate effects. Keep the stop time, alert time, lag history, resumed offset, and final reconciliation.

Kill a consumer

Terminate one consumer while producers continue. Prove that ownership moves or the process restarts as designed, lag is visible, poison records do not create an endless crash loop, and freshness returns within target. Preserve process events, partition ownership, lag, retry or quarantine records, and recovery time.

Inject duplicate messages

Deliver repeated events, callbacks, or jobs across a realistic retry window. Prove that idempotency prevents repeated business effects while duplicate attempts remain observable. Keep event identifiers, idempotency keys, dedup metrics, and before-and-after ledger or state comparisons.

Simulate a replica ten minutes behind

Route selected reads to a replica whose lag exceeds the stated budget. Prove that correctness-critical paths avoid stale state or make bounded staleness visible, and that operators see the breach. Record the induced lag, read results by path, routing decision, and alert output.

Change: Can the System Move Forward and Back?

Run a migration with old and new application versions

Hold both versions in service through the compatibility window. Exercise old and new reads and writes before, during, and after the schema change, then perform the planned rollback. Prove that neither version misreads missing, new, or reinterpreted fields and that authority never becomes ambiguous. Keep the version matrix, deploy sequence, contract results, data checks, and rollback time.

Retire an apparently unused derived store

Observe consumers long enough to cover infrequent jobs, support tools, exports, and recovery paths. Disable reads before deleting data. Prove that no required consumer breaks and that a bounded rebuild or archive path exists if the observation was incomplete. Preserve access queries, owner notices, dark-read results, disablement evidence, and the final removal decision.

Boundaries: Do Protection and Deletion Reach Every Path?

Rotate data-system credentials

Rotate credentials for the database, stream, warehouse, or object store without sharing the new secret through an unapproved path. Prove that every service recovers, scheduled and rarely used jobs still authenticate, old credentials are rejected, and access events are auditable. Keep rotation timestamps, service health, job results, audit entries, and denial proof for the old credential.

Test deletion propagation

Delete or tombstone a traceable test record and follow it through cache, search, analytics, exports, and other derived paths. Treat backup retention according to the declared policy rather than pretending retained backups were rewritten. Prove that active surfaces remove or redact the record within SLO and that tombstones live long enough to reach offline consumers. Keep record identifiers, timestamps, downstream queries, and policy evidence.

Load and Cost: Does One Workload Harm the Rest?

Simulate a hot tenant

Concentrate realistic traffic on the largest tenant or key range. Prove that per-tenant signals reveal the skew, controls contain it at the intended boundary, and unrelated tenants retain their SLOs. Preserve the workload shape, partition distribution, throttling events, and SLO comparison.

Run an analytical-query-storm tabletop

Present a burst of expensive dashboards or ad hoc queries with a plausible cost and concurrency profile. The team should identify the workload owner, find the active queries, estimate exposure, cancel or isolate them, and explain which reports become stale. Keep the query set, cost estimate, decision log, control procedure, and communication draft.

Failover: Does Traffic Move Without Hiding the Consequences?

Fail over a regional read path

Withdraw or degrade one region’s read path within a bounded scope. Prove that traffic moves as designed, access controls and consistency expectations still hold, capacity in the receiving region is sufficient, and rollback is available. Record routing changes, regional error and latency signals, replica freshness, capacity evidence, and recovery time.

Cadence Follows Consequence

Critical source truth, regulated data, financial ledgers, and entitlements deserve a restore or recovery drill at least quarterly and after a major migration. High-volume streams and shared derived stores deserve quarterly replay, rebuild, or freshness work. A moderate-risk service-owned database may justify semiannual restore work, but its migration rollback should still be rehearsed before a risky change. Low-risk caches can wait longer only when they remain genuinely rebuildable and incidents do not argue otherwise.

The calendar is a floor, not the reason for the drill. Architecture changes, new sensitive data, incidents, ownership transfers, and untested runbook edits should all pull the next rehearsal forward.

Close on Evidence, Not Attendance

A drill is complete when the team can say whether the promise held, point to the exact evidence, identify every step that depended on undocumented knowledge, and assign each failed alert, permission, contract, runbook, or recovery path. High-risk gaps require a dated repeat of the drill. A meeting ended and a capability proved are different outcomes.