Skip to content
PracticalHandbooksSoftware Handbooks
Library

Senior Engineering Interview Handbook / Chapter 105

Migrations and Backward Compatibility

A production-engineering chapter that follows one billing migration through its compatibility contract, backfill, dual running, cutover, changing rollback options, long-tail consumers, and cleanup.

The migration is already larger than the schema

The final diagram is clean because it has forgotten the old table, the mobile app that will not update for three weeks, the partner export that runs on the first of the month, and the records written before the new invariant existed. Production has not forgotten any of them.

Suppose a billing service stores an account’s plan as free text in plan_name. The business now needs a normalized plan_id that refers to a plan table. The destination is easy to draw. Accounts point to plan records; invoices, entitlements, and reports join through the identifier. The dangerous part is the journey, while old and new code are both serving traffic and only some records have moved.

A plausible one-line plan—add the column, run a script, deploy the new code, then delete the old column—contains no safe production state. It says nothing about an invoice created midway through the script, a legacy name that maps to two plans, a support query that still searches the label, or an old client that cannot send an identifier.

Migration planning begins by designing the overlap. Each intermediate state must say which representations may exist, which readers and writers are valid, what must remain true for customers, and what operators can do when evidence turns against the change.

Write the promise before the sequence

The billing migration is not principally a column migration. It is a promise that the account receives the same plan, price, entitlement, renewal behavior, tax treatment, and invoice history while the representation changes. Completeness—every row has a plan_id—does not prove that promise. A backfill can update every row successfully and still assign a small legacy cohort to the wrong plan.

That customer-facing promise becomes a set of invariants the migration can actually test:

  • one account resolves to one unambiguous billing plan at the time an invoice is produced;
  • plan assignment does not change merely because a record was migrated;
  • discounts, trials, cancellations, and grandfathered terms survive the new lookup path;
  • old writers cannot silently erase a newer plan decision;
  • unmappable records stop for review instead of receiving a guessed value;
  • every migrated value can be traced to the source value and mapping rule that produced it.

Now the dependency inventory has a purpose. The team searches for every writer and reader whose behavior can violate one of those invariants: checkout and account services, renewal jobs, invoice generation, entitlement checks, analytics, customer exports, support tools, imports, partner APIs, and manual repair scripts. Historical rows belong in the inventory too. So do operators, because pause, resume, reconciliation, and repair are part of the system during the migration.

Teams use backward compatible and forward compatible from different points of view. The label is less useful than the exact promise. Can an old reader handle output from a new writer? Can a new reader interpret old records and replayed events? Can both versions write without losing meaning? Put those sentences in the plan; do not let the word compatible stand in for them.

Expand until old behavior becomes boring

The first deploy adds the plan table and a nullable plan_id. Nothing reads the new field yet. Old code can continue to write plan_name, and a rollback is still an ordinary code revert because no production behavior depends on the new representation.

The next deploy teaches controlled writers to populate both fields when the mapping is unambiguous. If both values are stored in one database, one transaction can keep them together. If the migration crosses services or data stores, “dual write” is not an atomicity guarantee; the design needs an outbox, idempotent processing, reconciliation, or another explicit way to detect and repair a partial result.

New readers can then prefer plan_id while falling back to plan_name. That fallback is temporary scaffolding, but it has an important job: it lets the team exercise the new path before completeness is required. A mismatch metric can compare the plan resolved by each representation. The comparison must classify expected differences—such as an intentionally retired display label—so that real billing disagreements do not disappear into noise.

A schematic compatibility ladder begins with a baseline invariant, then expands the schema, runs old and new paths together, backfills, compares, cuts over, considers reversal, and cleans up while old clients continue to work.
The ladder is a memory aid, not a universal order. Each rung needs its own invariant, evidence, and reversal action; several rungs may overlap.

This expand-and-contract shape is useful because it separates support for the new representation from dependence on it. Expansion creates room. Population and comparison create evidence. Cutover spends that evidence. Contraction removes the old promise only after its consumers are gone.

A backfill is a live writer

Before the backfill starts, the team resolves obvious mappings and places ambiguous names in a review queue. Historical plans that no longer appear in the current catalog remain first-class records; “not sold anymore” does not mean “safe to reinterpret.” The mapping rules are versioned so that a later repair can explain which rule affected each account.

The job runs in bounded batches with checkpoints and a practiced kill control. Its write is conditional: update plan_id only if the source plan_name still matches the value the job examined and the target remains unset. Without that guard, a customer plan change racing the backfill could be overwritten by a decision made from stale data. A retry must be safe, and a stopped batch must leave enough evidence to distinguish work that committed from work that did not.

Row counts and progress bars answer whether the machinery is moving. They do not answer whether billing still means the same thing. The useful evidence is closer to the invariant: unresolved mappings, disagreements between old and new resolution, invoice and entitlement results by cohort, renewal and trial behavior, sampled grandfathered accounts, database load, replication lag, and support contacts. The team pauses on semantic disagreement even when the job itself is healthy.

Dual running has costs. Shadow reads consume capacity. Duplicate writes can diverge. Comparison reports can become noisy enough to normalize failure. That is why every temporary path needs an owner, a tolerance, and an exit condition. It is measurement equipment, not the future architecture.

The contract continues beyond the database

When account APIs begin returning plan_id, an optional field may look like a safe addition. It is safe only for clients that ignore unknown fields and do not infer a new meaning from its presence. A new plan status or enum member can break exhaustive client logic even though the response still parses. If the semantics change, preserve the old behavior or create an explicit version boundary; documentation and a new SDK announce a migration but do not prove that customers adopted it.

Requests need an overlap policy too. During the transition, the boundary may accept plan_name, plan_id, or both, then normalize them to one internal command. If both arrive and disagree, precedence must not be an accident. The service can reject the request or record the conflict for deliberate handling, depending on the product contract, but it should not silently choose whichever field the implementation happens to read first.

Events add history to the overlap. A consumer deployed today may replay an account event written last year. The new consumer therefore needs a defined interpretation for events without plan_id, and old consumers need a safe response to newly added data. Schema-registry checks can catch structural incompatibility; they cannot prove that plan_name and plan_id describe the same commercial promise. Replay representative history and compare the business result.

Mobile apps, partner SDKs, exports, dashboards, and customer cron jobs create the long tail. They do not share the service team’s deployment schedule. Observed version traffic, access logs, consumer registrations, query history, support evidence, and direct partner confirmation are stronger than a calendar date. An unknown consumer is not evidence that no consumer exists.

Rollback changes meaning as the work advances

Before any data changes, rollback can mean reverting code or disabling a flag. During the backfill, the safe response may be to pause, identify the last checkpoint, repair a cohort, and resume. Once reads cut over, the old path is a fallback only if it has remained current. After an invoice or partner action depends on the migrated value, restoring yesterday’s binary cannot restore yesterday’s world; recovery may require a forward fix, a compensating record, or manual repair.

There is therefore no single rollback plan. Each phase needs a trigger and an action:

  • what evidence stops further exposure;
  • whether new writes must be disabled, queued, or reconciled;
  • how partial work is located;
  • whether the old path still contains current data;
  • which changes can be reversed and which require compensation;
  • who has authority to cross the next irreversible boundary.

The point where code rollback stops being sufficient should influence the design before the first production write. It may justify audit records, snapshots, reversible transforms, delayed constraints, or a smaller initial cohort. “We can roll back” is credible only when the verb names an operation.

Cleanup is the last correctness gate

The migration has not finished when most reads use plan_id. It has finished when the old contract can be removed without surprising a live consumer and the temporary machinery can be deleted without losing required evidence.

For the billing service, that means all known writers have moved or received a documented exception; unmappable accounts are resolved; invoice, entitlement, support, analytics, and export paths use the new identifier; old API and SDK traffic has met its deprecation agreement; mismatch and error measures have remained within their bounds; and a named owner has removed fallback reads, duplicate writes, flags, queues, and migration-only dashboards.

A deadline alone is not evidence, and zero observed traffic has limits when instrumentation is incomplete. External consumers may require communication, an extended compatibility window, or a maintained adapter. The honest outcome may be that part of the old contract cannot yet be removed. Recording that exception is better engineering than declaring a false finish.

Rehearse the decision path

In a design or production interview, “I would roll it out gradually” leaves the hard reasoning hidden. Use one sheet of paper and make the overlap inspectable:

  1. State the customer or system invariant that must survive the change.
  2. Name old writers, old readers, new writers, new readers, historical data, long-tail consumers, and operators.
  3. Draw the intermediate states and mark which representations each allows.
  4. For every transition, name the evidence that permits it and the action that stops it.
  5. Mark where a code revert ceases to restore the old world.
  6. Define the adoption and correctness evidence required for cleanup.

Apply the exercise to a field split, an API enum change, or an event consumed by independently deployed services. Then change one fact: the source data cannot be recomputed, the oldest mobile version cannot ignore unknown fields, or one partner can migrate only once per quarter. If the plan survives only by repeating “expand and contract,” it is still a slogan. A credible answer changes the sequence, evidence, or duration to meet the new constraint.

Once the path is visible, a different question becomes possible: is the destination worth the cost and risk of getting there? That is the beginning of technical-debt and refactoring strategy.