Deployment, Rollout, and Online Migration

A zero-downtime migration is not one reversible command. It is a sequence of authority states.

Mercury must add a normalized customer-name field and a new search index across 1.2 billion rows while 18,000 writes/s continue. The old binary knows only display_name; the new binary can populate normalized_name. Reads must remain correct, foreground latency must stay inside its objective, and operators want binary rollback.

There are six legal operating states:

Only the first four preserve the verified rollback path. Crossing S3→S4 is an authority decision, not “turning traffic up.” A rollback button after irreversible new-format writes may restart old software successfully and still corrupt, hide, or reject valid state.

The rule for this chapter is: treat deployment and migration work as demand, reserve capacity for it, and preserve a tested rollback path until the new state is authoritative. Availability, correctness, latency, capacity, compatibility, and cleanup all belong to the result.

Use this workflow when software replacement has material cold behavior, old and new versions overlap, stored state changes shape or authority, production inputs are shadowed, or a backfill competes with foreground work. A binary with no state, no warm-up, and instant process replacement may need only a small subset; the same questions still expose whether that simplicity is real.

By the end, you should be able to:

• select a rollout pattern by the comparison and rollback boundary it creates;
• calculate a cold-capacity rollout rate from multiple constrained resources;
• build a mixed-version compatibility and authority matrix;
• sequence expand, dual write, backfill, verification, cutover, and contraction;
• specify partial-success repair without claiming exactly once; and
• govern a long migration with explicit success, abort, observation, and cleanup states.

Choose a rollout pattern for the risk being isolated

No rollout name guarantees safety. Each pattern creates a different comparison, capacity bill, and rollback boundary.

A canary answers whether a bounded population behaves acceptably; it does not by itself migrate stored state. Blue-green gives a clean binary traffic switch only when state, queues, identities, and external effects also have a reversible boundary. Shadow execution is safe only if effects are suppressed or written to an isolated namespace. Rolling replacement is cheap when instances warm quickly and dangerous when every new process creates a cache-miss and connection wave.

Use patterns together deliberately: expand the schema, canary compatible code, roll it through warm cohorts, backfill by phased key range, shadow new reads, cut authority, then contract. Name which question each stage answers.

Cold state is part of deployment capacity

A process can be alive but not useful at full rate. Startup may include image fetch, JIT or dynamic compilation, class loading, allocator initialization, configuration and secret reads, TLS establishment, connection-pool ramp, cache fill, local index load, model load, or topology discovery. At the same time, terminating instances redistribute connections and work.

Mercury has 60 warm instances, each modeled for 200 requests/s: 12,000 requests/s total. Foreground demand is 9,600 requests/s and the required reserve is 960 requests/s. The capacity allowed to disappear is therefore:

C_unavailable = 12,000 − 9,600 − 960 = 1,440 requests/s

At 200 requests/s per warm instance, capacity permits seven simultaneously warming replacements. Two other boundaries apply: a new instance adds 30 requests/s of cold backend load against 240 requests/s reserve, permitting eight; and it opens 60 connections over 30 seconds, or 2/s, against a 20/s reserve, permitting ten. The minimum is seven.

Warm-up takes 180 seconds, so the idealized continuous ceiling is:

R_start <= 7 × (60 / 180) = 2.333 instances/min

The runbook chooses two starts/min, one every 30 seconds, then adds observation holds. That is not universal safety. A correlated image pull, node drain, regional cache miss, or dependency loss can consume reserve. Recompute from current demand and stop conditions; never treat yesterday’s maxSurge as capacity evidence.

Health, readiness, warm capacity, draining, and termination differ

Define separate signals:

• startup: initialization is progressing within its allowed envelope;
• liveness: the process is irrecoverably stuck and restart is safer than waiting;
• readiness: the endpoint may receive its declared traffic class;
• warm capacity: the instance can sustain assigned load with guardrail headroom;
• draining: new assignments have stopped and accepted work has bounded ownership; and
• terminated: in-flight, queued, buffered, and durable effects reached a declared outcome.

Kubernetes distinguishes startup, liveness, and readiness probes. Its documentation warns that incorrect liveness can create cascading failure and notes readiness use for connections, file loads, and cache warm-up. A green readiness endpoint still cannot prove 200 requests/s of useful capacity unless it measures or gates the application-specific warm condition.

During deletion, stop new work before consuming the grace interval. Drain long requests, streams, sessions, leases, and queue ownership according to their protocol. Make client and load-balancer propagation time explicit. A preStop hook shares the termination budget; a hanging hook eventually loses to the grace deadline. Termination that abandons accepted work is a correctness event, not deployment housekeeping.

Compatibility is a matrix, not a version number

For every overlapping pair, ask whether it can read, write, ignore, and preserve the other’s state:

Compatibility spans API schemas, events, database rows, indexes, cache keys, RPC methods, configuration, and stored files. Unknown-field preservation matters: a read-modify-write client that silently drops unknown fields is not forward compatible. Defaults must mean the same thing across versions. Enum expansion, precision changes, ordering rules, and validation tightening can break mixed fleets without a parser error.

Define the compatibility window in releases and time. Observe old-version population rather than assuming rollout completion. Mobile, edge, batch, and disaster-recovery clients may outlive the server deployment.

Expand before migrating; contract after authority changes

Mercury’s sequence is:

1. Expand: add nullable normalized_name; add compatible code paths; create the new index without making it authoritative.
2. Dual-write: new writers update legacy and new representations under one logical operation identity; old remains authoritative.
3. Backfill: migrate historical rows by stable key range and checkpoint, fenced by authority epoch 41.
4. Verify: compare old and new reads by strata; reconcile missing, stale, and conflicting rows; prove index coverage.
5. Cut over: fence old writers, advance authority 41→42, make new reads/writes authoritative, and observe.
6. Contract: after the rollback window closes deliberately, remove fallback reads, legacy writes, old index/field where legal, flags, shadow work, and repair queues.

Expansion must be safe with old binaries. Contraction must wait until old readers, writers, jobs, restore images, and rollback procedures are absent or compatible. Schema presence is not authority. Backfill completion is not verification. New read success is not proof that all writes reached the new representation.

PostgreSQL 18’s CREATE INDEX CONCURRENTLY is a useful scoped example: it avoids locks that prevent concurrent inserts, updates, and deletes, but performs more work, takes longer, waits on transactions, and can leave an invalid index after failure. It cannot run inside a transaction block, only one concurrent build may run per table, and unique builds have additional semantics. “Concurrent” therefore means a particular lock behavior, not free, instantaneous, or failure-atomic migration.

Backfill has identity, ordering, and checkpoints

A safe backfill answers:

• Which stable key range is owned by this worker?
• Which source version or snapshot is read?
• What happens when foreground write and backfill race?
• Is the transformation deterministic and idempotent?
• Which epoch fences a stale worker after cutover?
• What checkpoint can resume without skipping or duplicating effects?
• How are poison rows isolated and audited?

Mercury uses 5-million-row checkpoints: 240 across 1.2 billion rows. A worker at epoch 41 may populate missing new representation but may not overwrite a newer dual write. At S4, epoch 42 fences epoch-41 workers. Checkpoint completion includes durable output and progress; writing progress before data can skip rows, while writing data before idempotent progress may repeat safely.

Read repair and lazy migration move cost onto foreground access. They are useful for a sparse cold tail, but they create latency variance and may never touch abandoned rows. Bound repair per request, make failures observable, prevent a hot key from repeatedly repairing, and retain a sweep for completeness when contraction requires it.

Dual writes require an outcome protocol

If old is authoritative during S2, four outcomes exist:

This matrix is not an exactly-once claim. Crashes can occur between effects and outcome recording. Use idempotency keys, monotonic versions, reconciliation, and explicit unknown outcomes. If acknowledging old-only success creates unbounded repair debt, fail the operation instead. If the new side is an index, a missing entry harms query completeness; if it is a derived cache, invalidation may be sufficient. The contract follows the state meaning.

Dual reads also cost capacity. Reading old and new for every request can double database, network, and deserialization work. Sample by representative strata, cap comparison concurrency, and ensure comparison does not change authoritative output. A mismatch metric needs stable categories: absent, stale version, unequal normalized value, ordering difference, read error, and comparison skipped.

Reserve migration capacity from foreground work

Mercury’s database write budget is 80,000 row-equivalents/s. The ledger reserves:

foreground writes                         18,000 rows/s
dual-write tax                           18,000 rows/s
new-index maintenance equivalent         12,000 rows/s
required failure/traffic reserve         10,000 rows/s
safe backfill remainder                  22,000 rows/s

The idealized lower bound is:

T = 1.2 × 10^9 rows / 22,000 rows/s = 54,545 seconds = 15.15 hours

Real completion takes longer because checkpoints, retries, skew, vacuum/compaction, replication, validation, throttling, and peak holds consume time. Throttle on the constrained resource and foreground outcomes, not only rows/s. A row can cost different CPU, WAL/log, index, storage, and replication work by shape.

Isolation can use a separate worker pool, queue, connection budget, storage bandwidth class, tenant/key range, or time window. It is real only at the bottleneck. A separate Kubernetes namespace sharing the same database write path is administrative separation, not capacity isolation.

Shadow traffic belongs in the same ledger. Copying 10% of reads may add 10% ingress but far more downstream work if caches are cold or effects fan out. Suppress email, payments, audit authority, mutation, and quota consumption unless the shadow contract intentionally tests them in an isolated namespace.

Success, abort, and observation windows are state-specific

Do not use “pods available” or “backfill 100%” as the sole success criterion. For each transition define:

S2 enter: compatible writer can produce both forms; repair queue bounded
S2 abort: correctness mismatch, repair age >10 min, write-budget breach
S3 enter: backfill checkpoints complete; invalid/poison rows accounted for
S3 success: sampled strata agree; old-only writers absent; new index valid
S4 enter: authority owner approves epoch 42; rollback boundary acknowledged
S4 abort/forward-fix: correctness, goodput, latency, or saturation breach
Hold: covers warm-up, peak, slow repair, background cycle, and failure probe

Before S4, abort can mean disable dual write, stop backfill, repair debt, and roll binaries back. After S4 has accepted new-only semantics, “abort” may mean stop expansion, preserve new authority, and forward-fix. Runbooks must not promise a forbidden direction.

Observation windows follow mechanisms. Three green minutes cannot validate a 210-second warm-up plus hourly cache expiry. A day may still miss a weekly compaction. Use leading indicators—queue age, repair rate, cold misses, connection establishment, mismatch—and lagging outcomes—goodput, p99, correctness, cost—without waiting indefinitely. Every hold has a named reason and exit decision.

Govern long-running migration as production software

Every migration needs one owner, deputy, state, start time, checkpoint, rate, remaining estimate with assumptions, error/repair age, resource ledger, last verified rollback point, next decision, and expiry. Page on harm and stalled safety signals, not merely slow progress.

Cleanup is a planned state transition:

• stop and delete backfill workers and credentials;
• drain or reconcile repair queues;
• remove shadow reads and comparison telemetry;
• remove old writers/read fallback and compatibility flags;
• drop obsolete indexes/fields only after retention and restore checks;
• reduce temporary capacity and quotas;
• update disaster recovery, restore, bootstrap, and replay paths; and
• preserve an evidence record of counts, mismatches, exceptions, and authority change.

A migration that runs forever accumulates compatibility branches, double writes, duplicated cost, ambiguous authority, and operator fear. The final deletion is often where the permanent performance gain appears.

Migration runbook

ONLINE MIGRATION RUNBOOK
Capability/population/correctness/SLO boundary: ______________________
Old and new representations; authority owner: ______________________
Version/API/event/schema/cache compatibility matrix: _______________
States S0-S5 and entry/exit evidence: _______________________________
Last binary rollback state / last data rollback state: ______________
Foreground capacity + reserve + migration/shadow/repair ledgers: ____
Backfill key ownership, epoch, transform, checkpoint, poison policy: _
Dual-write outcomes, idempotency, unknown result, repair ownership: __
Read comparison strata, sample, mismatch taxonomy: __________________
Rollout pattern, cohort assignment, cold-start model, safe rate: _____
Startup/readiness/warm-capacity/drain/termination signals: __________
Success, abort/forward-fix, hold windows, escalation authority: ______
Current state/rate/ETA assumptions/errors/repair age: _______________
Contraction inventory, owner, earliest safe date: ___________________

Applied work: plan and rate the change

Use this field check at every state transition:

1. Which version and representation are authoritative now?
2. Can every overlapping reader and writer preserve state it does not understand?
3. Which exact operation closes binary or data rollback?
4. How much warm capacity, downstream cold-load capacity, and connection budget remains?
5. Are readiness and liveness separate from measured useful warm capacity?
6. Who owns accepted work while an instance drains or terminates?
7. Can every dual-write partial success be found and reconciled by logical identity?
8. Is backfill fenced, idempotent, checkpointed, skew-aware, and isolated at the actual constraint?
9. Do shadow, repair, index, replication, and validation work appear in the capacity ledger?
10. Do success and abort criteria cover correctness, goodput, latency, saturation, and repair age?
11. Does the hold window cover warm-up, peak load, background cycles, and failure recovery?
12. Which old code, state, index, flag, queue, credential, and temporary capacity will contraction remove?

Run the packet:

cd examples/performance-engineering-system-design-handbook/part-07/deployment-migration
node analyze.mjs
node verify.mjs

Then produce two artifacts.

First, complete the migration runbook for the 1.2-billion-row schema and index change. Explain why S0–S3 preserve rollback, which write makes S4 irreversible, how epoch 41→42 fences stale workers, how 240 checkpoints resume, and how each dual-write partial success reconciles. Recalculate the 22,000 rows/s remainder and challenge the 15.15-hour lower bound under skew, retries, and index-build interference.

Second, set the cold-start rollout rate. Derive all three concurrent-warming limits—seven from warm capacity, eight from backend cold load, ten from connection establishment—then choose the minimum. Convert the 180-second warm-up into the 2.333 starts/min modeled ceiling and defend the two starts/min operational rate, holds, and abort criteria. If you choose a slower rate, name the uncertainty it buys down.

A strong plan defines “zero downtime” as no planned loss of correct, in-objective service for the declared population. It does not mean zero errors anywhere, free shadow traffic, instantaneous rollback, or uninterrupted progress.

Evidence and transfer limits

• Kubernetes Deployment documentation defines rolling updates and maxUnavailable/maxSurge; probe guidance distinguishes startup, liveness, and readiness and warns about cascading restart failure. Controller availability is not Mercury’s warm-capacity proof.
• Kubernetes container lifecycle hooks describes preStop and termination-grace behavior. Application protocols still own accepted-work outcomes and drain correctness.
• PostgreSQL 18 CREATE INDEX documents concurrent-build locks, extra scans/work, transaction and per-table restrictions, and invalid-index failure states. Other engines and versions differ.
• Google SRE’s canarying guidance supports partial, time-limited evaluation against a control. It does not solve state compatibility or choose Mercury’s rollout rate.
• All Mercury values are simulated teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-07/deployment-migration/. Passing checks validates arithmetic and declared states, not a production controller or availability guarantee.

The decision rule is: model rollout and migration work explicitly, isolate it from foreground demand, and preserve a verified rollback path until new state is authoritative. Once change is safe under expected load, the harder operating question is what happens when load exceeds every planned boundary and the system must degrade and recover intentionally.