Data Migrations, Backfills, and Cutovers

One Tenant, Two Shards

A collaboration product needs to move one large enterprise tenant off an overloaded shard. Copying the tenant’s rows is straightforward. The tenant, however, does not stop collaborating while the copy runs. People create documents, revoke permissions, add comments, schedule exports, and correct billing details. Search, audit, and background jobs each keep their own derived view of those changes.

Suppose a permission is revoked after the permissions table has been copied but before the documents table finishes. The snapshot carries the old grant to the target. If the migration misses the later revocation, row counts can match perfectly while a user retains access they should have lost. The hard question is not whether the copy finishes. It is whether every fact has an unambiguous authority while two live systems contain different versions of it.

This is where a data migration departs from the schema change in the previous chapter. Adding a compatible column is mostly a problem of application and schema coexistence. Moving a tenant, changing a data model, or crossing stores creates a second live state and a routing decision. The migration must account for every write between the baseline copy and that decision.

A useful migration therefore has two planes. The data plane copies and replays facts. The control plane records boundaries, progress, lag, production pressure, validation, authority, and permission to advance. A script may implement one step. It cannot, by itself, make the whole change safe.

Draw a Boundary Through the Write Stream

Before copying anything, name the scope. For this tenant move, scope includes more than rows bearing a tenant key. It includes projects, documents, comments, permission grants, audit records, billing references, search documents, caches, exports, scheduled jobs, support tools, and the routing entry that tells all of them where the tenant lives. An omitted derivative can be more dangerous than an omitted base row because it may continue serving plausible but stale answers.

Next, choose a snapshot boundary that can be related to subsequent writes. A useful boundary declares the source, consistency level, included entities, exclusions, and a log or event position. “The rows the worker happened to see” is not a boundary. If tables must be captured at different times, the plan must explain how their versions form a coherent tenant state.

For the tenant move, let the source shard remain authoritative while a consistent snapshot is taken at log position L0. The target receives that baseline. A service outbox then supplies every tenant mutation committed after L0, including deletes and permission changes. Each event carries a stable event identity and source version, so replay at the target is idempotent.

Now reconsider the permission revocation. If it committed before L0, it belongs in the snapshot. If it committed after L0, it belongs in the outbox tail. If neither path contains it, the migration has a gap. If both contain it, the target’s version rule must make the duplicate harmless. The boundary turns an alarming race into a question the system can answer.

Transaction-log change data capture can provide this boundary when its retention, replay position, ordering, delete representation, and schema-change behavior are understood. A service outbox can provide it when the data write and outbox record commit atomically. An event log works only if it contains every mutation required to rebuild current state. An updated_at scan is weaker: clocks, timestamp resolution, backdated repairs, and deletes can all create blind spots. Application dual writes introduce partial failure and retry-order problems; use them only with a declared authority and reconciliation path.

Dual reads and shadow traffic solve a different problem. They ask whether the target answers as the source does before target answers become visible. They are valuable for permission predicates, pagination, search tokenization, serialization defaults, and query cost. They do not recover a write absent from both the snapshot and the tail.

Backfill in Units the Operator Can Stop

The backfill should be intentionally dull: bounded work, durable progress, retry-safe writes, and a prompt response to production distress.

Choose a chunk key that contains the invariants you need to reason about. Primary-key ranges are simple but may cut across tenants or hot partitions. Time ranges suit append-only data. Partition ranges align with storage ownership. For this move, table-and-key ranges work for independent records, while a document plus its permission grants may need to travel as one business-entity chunk so that validation can inspect their relationship.

Every chunk needs a stable identity. The target can upsert by source identity and source version, reject an older version than it already holds, and record a content hash after canonical transformation. A retry then repeats an intended state transition instead of incrementing a counter, duplicating an append, or overwriting a newer change with an older snapshot value.

Progress belongs in durable state, not only worker logs. For each attempt, record the migration and transform version, source range or entity key, snapshot boundary, rows scanned, written, skipped, and failed, retry count, last error, validation result, timestamps, and worker identity. That record should let a new operator answer three questions without reconstructing history from log lines: what is claimed complete, what can be retried, and what remains unproved?

The worker also needs a brake connected to the system it might injure. Concurrency is only one lever. Pause or slow work when source lock waits rise, replica or outbox lag consumes the allowed window, target write latency climbs, queues back up, storage or transaction logs grow too quickly, foreground error rate increases, or an operator asserts a stop. Restarting must resume from durable chunks rather than beginning a second, overlapping copy.

The measured capacity margin established in Chapter 29 now becomes an operating constraint. If the migration was budgeted five milliseconds of foreground latency and twenty seconds of replica lag, those limits should change worker behavior. A dashboard that turns red without changing the work rate is observation, not control.

Prove the Answers Users Depend On

Movement metrics show that work occurred. They do not establish correctness.

Row counts are a useful first alarm. Partition checksums add coverage when source and target can be reduced to the same canonical values. Both can still pass while permissions, relationships, units, null meaning, timestamp precision, or derived answers are wrong. A cross-store migration often changes representation deliberately, so byte equality may not even be meaningful.

Validation should climb from structure to business behavior. In the tenant move, compare project, document, and comment counts; active permission grants; audit-log sequence ranges; billing totals; export manifests; and the membership of scheduled jobs. Then oversample records that are likely to resist the transform: very large documents, old and recently edited records, deleted items, manual repairs, rare permission combinations, null-heavy rows, and entities implicated in earlier incidents.

Finally, compare served answers. Shadow reads should exercise busy project pages, permission-filtered lists, search results, admin views, and export preparation. A mismatch record needs the tenant and entity key, request fingerprint, source and target versions, old and new results, mismatch class, and enough context to reproduce the read. Sampling can protect hot paths, but high-consequence operations such as permission checks may justify complete comparison during the migration window.

The permission revocation is a required probe, not a colorful edge case. Inject or identify one that lands while its entity is being copied. Show its outbox position, its application at the target, the resulting permission answer, and the absence of the revoked user from search and exports. If the control plane cannot trace that write, it is not ready to approve cutover.

Cutover Is a Change of Authority

By cutover, the target may contain a complete snapshot and a nearly current tail. The remaining decision is when authority moves and what behavior remains reversible.

Write the gate before migration begins. For the tenant move, a filled gate might say:

scope: tenant acme; 12 base tables; search, cache, export, billing, audit, and jobs named
snapshot: complete at source log position L0; no undocumented exclusions
tail: applied through L84219; lag below 5 seconds for 30 minutes
backfill: 8,416 chunks complete; 0 failed; transform version 3
validation: structural checks clean; billing totals exact; permission probes clean;
            50,000 shadow reads with no unresolved mismatch
production budget: p99 latency increase below 5 ms; replica lag below 20 seconds
writers: source authoritative until route switch; outbox remains active through rollback window
fallback: target writes continue to be captured; source-read switch rehearsed
decision: migration owner and incident commander both present; support channel open

The values are examples, not universal thresholds. Their purpose is to make “ready” falsifiable. Scope must be closed. The tail must remain within a declared lag for long enough to show stability. Failed chunks and mismatches must be resolved or explicitly accepted by someone who owns the consequence. Production must still have its error and capacity budget. The fallback must have been exercised rather than merely described.

Some cutovers need a short freeze because a class of writes cannot be captured safely. Define the blocked operations, the permitted operations, the user-visible behavior, who may override it, and the maximum duration. A vague request for nobody to write is not an operational control.

Shift low-blast-radius behavior first: internal support reads, then one read cohort, then the tenant’s ordinary reads. Watch correctness and latency together. The tenant routing entry is the actual authority switch. Immediately before changing it, close the tail gap or freeze the relevant writers; immediately after changing it, ensure new writes are authoritative on the target.

Rollback is now asymmetric. Routing reads back to the source may be safe while target writes continue to be captured and repaired. Blindly reversing all writes may discard corrections accepted after cutover. The runbook must name the authority for each phase:

• before the route switch, the source wins and the target is repaired from snapshot plus tail;
• during the bounded fallback window, new target writes are retained even if reads temporarily return to source;
• after the rollback window closes, the target wins and late source access is treated as a defect.

If the system cannot capture writes in both directions during fallback, narrow the allowed operations or accept that fallback is read-only. Reversibility is a property of the write path, not of the feature-flag interface.

Keep the Old Path Until Silence Is Evidence

The first minutes after cutover expose the target to the combinations that rehearsals miss: late jobs, stale caches, support scripts, monthly exports, old credentials, retries, and rare clients. Keep the control plane alive.

Separate health from correctness. Health covers latency, errors, saturation, storage, and lag. Correctness covers business totals, permission results, mismatch classes, support reports, and unexpected reads from the source. A target can be healthy and wrong.

Reconciliation needs a rule for each mismatch class. If a source write committed before the authority switch but arrived late, replay it. If a correction committed on the target after the switch, preserve it. If both sides changed the same business fact, send the conflict to the named domain owner instead of letting timestamp order invent policy. Retain the source and target versions, first and last observation, attempted repair, result, and owner for each unresolved mismatch.

Make the old tenant copy read-only before deletion. Watch source access by job, credential, query fingerprint, and owner through at least the rollback window and every low-frequency cycle the migration promises to cover. Move or remove scheduled jobs, dashboards, exports, caches, routing exceptions, credentials, support tools, backup assumptions, and alerts. Retention and residency policy decide whether the old data is archived or deleted.

Silence becomes evidence only when the system can attribute it. “We saw no traffic” means little if batch readers share an unidentified account or query logs expire before the monthly export runs. This is the handoff to the next chapter: before an old data shape or API can disappear, its consumers must be discoverable and accountable.

How the Pattern Changes With the Move

The tenant move supplies one concrete architecture, but not every migration needs its exact machinery. A same-store backfill may inherit transaction semantics and avoid a separate CDC service, while still contending with locks, replicas, compaction, and foreground work. A cross-store move must account for differences in types, constraints, isolation, collation, and authorization. A data-model migration needs semantic comparisons because one source record may become several target records or an event history. A regional move adds routing, residency, encryption-key, and jurisdictional boundaries.

In each case, keep the invariant: every fact has a declared authority, every write after the snapshot has a path, every unit of work is safe to retry, validation tests meaning, cutover changes authority deliberately, and decommission waits for attributable silence.

Migration Control Record

For a real migration, keep one live record that an operator and approver can both use. It should contain:

• scope, owners, source and target, derivatives, exclusions, and the invariant at risk;
• snapshot consistency and position, change-capture mechanism, retention, lag, and replay recovery;
• chunk identity, transform version, idempotency rule, progress, retries, and stop behavior;
• resource budgets and the signals that automatically throttle or pause work;
• structural checks, business totals, edge samples, shadow comparisons, and unresolved mismatch classes;
• authority by phase, freeze behavior, route sequence, fallback limits, rollback window, and stop authority;
• reconciliation rules, source-access evidence, retention obligations, and decommission criteria.

Do not fill the record with green labels alone. Link each claim to a query, log position, comparison result, rehearsal, owner, or exception. The control plane earns a cutover by preserving the reasoning that makes the next state safe.

Migration Drill

Choose a tenant or customer data set that continues accepting writes. Design a move to another store or shard. State the snapshot boundary, change-capture path, chunk key, idempotency rule, progress record, throttle signals, validation evidence, cutover gate, authority change, fallback limit, reconciliation rules, and decommission evidence.

Then introduce one uncomfortable write—a delete, permission revocation, billing adjustment, or ownership change—while its entity is being copied. Trace it from source commit through capture, replay, validation, cutover, and possible fallback. If any phase can silently lose the write or leave its authority ambiguous, revise the migration before production does it for you.