Evolution and Migration

The new system is ready. The old world is not.

Suppose an application sends email during the request that creates an order, resets a password, or invites a teammate. Templates live in application code. User preferences sit in the main database. A provider timeout may fail the request, trigger a blind retry, or disappear into a log.

The target architecture seems straightforward: write a durable notification intent, let a notification service choose the template and provider, retry failed attempts, and retain an audit history. Drawing that system is the easy half of the design. The harder question is how it becomes real while old application versions, queued work, existing preferences, support tools, and provider side effects are still in motion.

A one-release replacement asks several unproven changes to succeed together. The new service must interpret every notification correctly, honor suppression rules, avoid duplicate sends, survive retries, and produce an audit trail. A code rollback cannot unsend an email. By the time the error is visible, the old world may already be impossible to restore.

Begin with the promise that must survive

For this migration, use three promises:

• a business event produces at most one intended notification;
• opt-outs and suppression rules are honored before provider delivery;
• every delivery attempt can be explained to an operator.

These promises are more useful than “zero downtime.” They determine the identity of an intent, the evidence needed during shadow operation, the stop condition for a canary, and the cases where rollback is unsafe.

Give each intent a stable idempotency key derived from the business event and notification purpose, not from a particular HTTP request. Record template version, recipient, suppression decision, and provider attempts separately. Then a request retry can find the same intent, while a provider retry remains a new attempt against that intent rather than a second business action.

The distinction also exposes a hidden compatibility problem. The old application may represent a password-reset email as a direct provider call; the new service represents it as an intent with a versioned template and policy decision. Supporting both shapes is not enough. The design must say which interpretation is authoritative at each stage.

Expand the old world before asking it to move

Mixed versions are ordinary production state. A rolling deploy leaves old and new application processes alive together. Mobile clients and browser tabs can outlast several server releases. Queues retain messages written under old schemas. Reports, exports, caches, and support tools often depend on contracts that are absent from the architecture diagram.

Begin with additive changes. Create intent and attempt records without requiring them. Add optional event fields before any consumer depends on them. Deploy readers that tolerate both old and new representations. If an event’s meaning changes, version the meaning and keep a translator or versioned handler; changing only the label to v2 does not make old data safe to replay.

Database changes follow the same logic:

1. Add the new tables, columns, or indexes.
2. Deploy code that can read both representations.
3. Populate historical state while live writes continue.
4. Move reads and then writes behind measured gates.
5. Remove the old representation only after rollback, retention, audit, and adoption needs have expired.

Renaming or deleting a column in the first step is a wager that every reader, job, dashboard, and export changes at the same instant. Expand first; contract only after the compatibility window has actually closed.

Keep authority singular while behavior overlaps

The first useful release does not send through the new service. The application continues to call the provider, but it also records the intent that the new platform would have received. The old sender is still authoritative; the intent is observational.

That is a dual write, but not a claim that two destinations are equally true. If the provider call succeeds and the intent write fails, the notification has still been sent under the current contract. The failure must be counted and repaired for comparison, but replaying it through the new sender would risk a duplicate. If the intent succeeds and the old send fails, the old path’s retry policy remains authoritative until cutover.

Where possible, avoid independent application writes altogether. Commit the business state and an outbox record in one transaction, then derive the intent from the outbox. Change-data capture can serve a similar transition when all writers cannot be changed, provided the design handles ordering, deletes, schema evolution, replay volume, and lag. In either case, derived delivery is only credible when retries are idempotent and divergence has an owner.

The temporary data path can be spoken as a trace:

business transaction
  -> authoritative old send decision
  -> durable mirror intent
  -> new service renders and checks policy
  -> comparison record; no provider call

This period is not wasted duplication. It is where the team discovers that the new preference service interprets a missing setting differently, that an old event lacks locale, or that two template names represent the same business purpose.

Shadow work must compare consequences

In shadow mode, the new service consumes real intents, selects templates, checks preferences, and records what it would send. It does not call the provider. Compare decisions that protect the invariant:

• whether a notification should exist at all;
• recipient and channel;
• template and locale;
• suppression reason;
• idempotency identity;
• render success and latency.

Aggregate match rate is too forgiving. A small mismatch rate may contain every password reset or every opted-out user. Partition differences by notification type, tenant, client version, locale, and policy outcome. Sample rendered content where privacy controls permit it, and retain enough identifiers to reproduce a disagreement without copying sensitive bodies into logs.

Shadow traffic proves that the new path can observe and decide. It cannot prove that provider calls, retries, or user-visible side effects behave under real consequence. That requires a canary.

Move consequence before moving all authority

Enable provider calls first for internal accounts, then for one low-risk notification type or a small tenant cohort. A useful canary has a blast-radius boundary and a named stop condition. For this system, stop on any confirmed opt-out violation or unexplained duplicate; pause when intent-to-attempt lag, provider error rate, or audit gaps cross their limits.

During the canary, routing must be exclusive. A selected intent is sent by the new service; the old path records that ownership and does not also call the provider. Feature flags that merely enable the new path without disabling the old one create the exact duplicate-send race the migration is meant to remove.

Authority moves by notification type or cohort:

before cutover
  old sender = authority
  new service = observer

canary cohort
  durable intent = authority
  new service = sender
  old sender = compatible fallback only where explicitly safe

after cutover
  durable intent = authority for all migrated types
  provider calls = retryable attempts derived from that intent

This phrasing matters in an interview because “we dual-write, shadow, and canary” otherwise sounds like a list of mechanisms. The movement of authority turns them into a migration design.

Treat the backfill as live system load

Support may need recent send history in the new audit view. That does not imply copying every historical log line. Choose the smallest history that serves an actual support, audit, or reconciliation need, and mark imported records so they cannot be mistaken for new delivery attempts.

Partition the backfill by time window or tenant. Use stable checkpoints, idempotent writes, bounded batches, rate limits, and a repair queue for records that cannot be translated. Expose progress and error counts by partition, not just a global percentage. Operators need a pause control, and the system needs a rule for live writes that arrive in a partition while it is being copied.

A backfill competes with user traffic for database I/O, replication bandwidth, cache capacity, and operator attention. Estimate its duration and maximum load; do not call it “background” as if that made the work free.

Reversal changes as the migration advances

Before the new service calls a provider, reversal is simple: stop the shadow consumer and leave the old sender untouched. During a canary, routing can return to the old sender only for notification types whose templates, preferences, and idempotency semantics it still understands.

After new-only notification types appear, ordinary rollback may be dishonest. The old path cannot send a message it cannot represent. After any provider call, no rollback can undo the external side effect. The safe response may be to pause one type, quarantine ambiguous intents, reconcile attempts, or roll forward with a repair.

State that boundary before cutover:

Code can roll back while contracts remain compatible.
Data may need restore, replay, or compensation.
External side effects may require quarantine and roll-forward repair.

Keep the old path, adapters, replay log, and comparison evidence until those recovery options are no longer needed. Cleanup is a migration phase because deleting old code and data removes ways to understand and repair failure.

Give the interviewer the transition, not the inventory

A complete answer does not need to enumerate every migration tool. It needs to make one transition inspectable. For the notification platform, a strong spoken answer could be:

The invariant is no duplicate intent, honored suppression, and an auditable
provider attempt. I would add durable intents and attempts first while the old
sender remains authoritative. The new service would shadow real intents and
compare recipient, template, locale, preference, and idempotency decisions.

I would canary one low-risk notification type with exclusive routing, stopping
on any opt-out violation or unexplained duplicate. At that boundary the intent
becomes authoritative for the canary cohort; provider calls are attempts
derived from it. Recent history can be backfilled in bounded, idempotent
batches for support, but imported records never trigger sends.

Rollback is safe only while the old path understands the active semantics.
After new-only types or external sends, I would pause, quarantine, reconcile,
or roll forward rather than promise that a deploy rollback restores the old
world. Old code and fields are removed last, after adoption, retention, audit,
and repair gates pass.

That answer gives the interviewer productive places to push: a mobile client that cannot upgrade, an outbox backlog, a preference mismatch concentrated in one locale, a provider without idempotency support, or a compliance rule that requires longer audit retention. Each pressure changes the transition rather than inviting another generic component.

Practice a migration that resists the recipe

Apply the same reasoning to a product catalog moving from one price_cents column to regional price books with currencies and scheduled changes. State which price must remain authoritative, then design additive schemas, compatible reads, live-write handling, a throttled backfill, shadow price comparisons, a regional canary, and the point after which the old column can no longer provide a truthful rollback.

Make the exercise harder: old mobile clients cache prices for a day, checkout must honor the displayed price for a bounded period, and one export consumer cannot read the new schema yet. If the plan only says “expand, migrate, contract,” it is not finished. Show where the displayed-price promise lives, how versions are translated, which path owns a write, what evidence permits cutover, and how an operator repairs divergence.

When the design starts to collapse into a deployment checklist, recover with:

Let me describe the temporary system: what remains compatible, where authority
lives now, how we compare old and new behavior, and which reversal is still
honest at each cutover.

The previous chapter asked whether an architecture’s economics can survive its workload. Migration asks whether the organization can reach that architecture without gambling existing users and data. Once that transition is credible, the remaining interview problem is making the reasoning easy to follow while the conversation changes direction. Driving the Conversation takes up that work.