Geo-distribution and the Physics of Distance

Ledgerline’s architecture review begins with a diagram that appears balanced: three regions, identical service stacks, arrows in both directions, and “active-active” above the frame. The diagram cannot answer the first operational question: where may an inventory reservation become authoritative?

The workload makes symmetry impossible:

• catalog reads originate worldwide and tolerate 90 seconds of bounded staleness;
• checkout has a 180 ms server-side deadline for the measured population;
• a stock unit may be promised to only one order;
• payment and order records must remain in an allowed residency area;
• a buyer can browse in Nairobi, sign in from Paris, and complete an order whose stock is owned in Frankfurt; and
• loss of one serving region must not turn a regional incident into an unbounded global write conflict.

Ledgerline rejects the symmetric diagram. Catalog data can be copied near readers. Inventory authority cannot be inferred from the nearest endpoint. Each stock partition receives a home region and a fenced authority epoch. Checkout runs close to that authority or pays one explicit wide-area round trip. Regional data planes keep serving from locally applied policy when the global placement control plane is unavailable.

That decision is not “single region” versus “global.” It places each operation according to the invariant it must preserve.

The governing rule is: place state and coordination close to the operations whose invariants require them; use replication and routing to localize everything else.

Distance is a budget floor, not a deployment setting

A region selector cannot repeal propagation. A packet travels through fiber routes, optical equipment, routers, congestion-control state, encryption, queues, proxies, and application stacks. Geographic distance supplies a lower bound; the path and its operating state add to it. A cloud-region label supplies neither a latency guarantee nor a stable user route.

Model the path before choosing topology. For operation o from population g, record:

L_o,g = client + edge + route + regional queue/service
      + required wide-area coordination
      + dependency work + response

Use a distribution for each term. A single “region latency” number hides access-network variance, peering changes, loss recovery, connection setup, and cross-region queueing. Separate at least:

• warm and cold connection paths;
• read, write, and conflict paths;
• successful, rejected, pending, and failed outcomes;
• normal, evacuated, and recovering modes;
• user geography and network class; and
• local processing from wide-area waiting.

Ledgerline uses a modeled one-way propagation-and-route allowance of 76 ms from its Nairobi checkout edge to the inventory home. It is a teaching input, not a public-network measurement. One round trip consumes 152 ms:

checkout budget                         180 ms
one modeled inter-region round trip   -152 ms
remaining queue, service, and reply     28 ms

The path is already tight. A second serial coordination round trip would consume 304 ms before local work. The design must combine validation, authority, and commit into one bounded exchange, move execution nearer the authority, relax the objective, or change the invariant. Adding another replica does not change this arithmetic.

Bandwidth-delay product matters on bulk replication paths. A high-capacity link with a long round trip needs enough in-flight data and parallelism to fill it. Loss, retransmission, per-flow limits, encryption, compression, storage apply rate, and receiver backpressure may become the constraint before nominal link bandwidth. Measure useful applied bytes per second, lag age, and retransmitted work—not merely bytes accepted by a sender.

The latency budget should therefore include uncertainty. Use recent, population-scoped route measurements for planning; test from real access networks; preserve worst credible failover paths; and keep the physical/model floor distinct from observed tails.

Route evidence also has a lifetime. A one-week probe can establish neither a permanent bound nor the experience of populations it did not sample. Preserve timestamped raw observations, connection state, resolver answer, source network, destination, protocol, payload, success criteria, and path mode. Compare normal and evacuated routes with the same measurement boundary. If an edge change lowers median latency while increasing cross-region loss recovery for one mobile network, an aggregate regional percentile can hide the regression. The planning model should use ranges and an explicit remeasurement trigger: carrier change, new edge, region move, encryption/protocol change, or an observed route distribution outside the assumed envelope.

Start with operations, state, and invariants

“Where should the database live?” is too coarse. A system contains operations with different authority, freshness, privacy, and latency needs. Build a placement table before drawing regions:

Four placement questions follow.

1. Which state makes the decision authoritative? A cache, routing hint, session cookie, or replica can direct work without becoming authority.
2. Which operations require coordination? The answer comes from invariants, not from the marketing name of a database.
3. What may be stale or conflicting? Name the field, version, duration, and repair rule. “Eventually consistent” is not a user contract.
4. Where is processing legally and operationally allowed? Residency can constrain storage, logs, backups, support access, key material, derived data, and transit—not only the primary row.

Regional partitioning is often the durable answer when authority aligns with a stable key: merchant, tenant, account, document, or inventory pool. It converts global coordination into a routing problem for most operations. The cost is mobility. Moving an authority partition requires snapshot/catch-up, a fenced epoch, client rerouting, mixed-version behavior, and rollback rules.

Avoid deriving authority from current user location. People travel; mobile networks exit in surprising countries; corporate proxies relocate traffic; and IP geolocation is imperfect. Location can select a nearby read path. Authority needs a stable, inspectable assignment.

Residency is also not synonymous with “region preference.” Record the policy source, applicable data class, allowed processing and replication locations, encryption/key requirements, backup treatment, retention, deletion, and audit evidence. When policy is uncertain, route conservatively and obtain a policy decision; do not encode a legal conclusion into latency logic.

Three topologies answer different failure questions

Active-passive, active-active, and home-region designs are mechanism families, not maturity levels.

An active-passive service can still serve local replicated reads. An active-active service can still serialize one invariant through a leader. A home-region system may have many active regions while maintaining one authority per key. State the write and read path rather than relying on the label.

Active-active removes a remote hop only when the write can complete locally. If replicas must synchronously agree before acknowledgment, geography remains on the commit path. If each region accepts independently, the application inherits concurrent versions and must define detection, merge, rejection, compensation, or user-visible ambiguity.

Some operations commute: adding an item to a grow-only set under a bounded model, recording a uniquely identified observation, or increasing independent counters within an escrow allocation. Others do not: promise the final stock unit, apply a non-repeatable payment, choose one username, or enforce a global spending cap. “Last timestamp wins” is a conflict policy that discards one outcome; it is not neutral resolution, and Chapter 33’s clock limits still apply.

Ledgerline selects a hybrid:

• catalog and product media replicate outward and serve locally;
• stock partitions and merchant orders have a home region with fenced epochs;
• checkout is routed to the relevant home, with a signed operation identity across retries;
• order projections replicate to viewing regions with version and freshness;
• analytics facts ingest regionally and reconcile by stable event identity; and
• the placement directory is globally distributed but not consulted synchronously for every already-routed request.

Read locality and write locality are separate choices

Local reads improve latency only when the local copy has the fields, version, indexes, and freshness required by the operation. For each read, specify:

• source of authority and replica role;
• staleness bound and how age is computed;
• version/session requirement;
• behavior when the replica is behind or its lag is unknown;
• whether the client can wait, read home, accept stale data, or fail; and
• whether the response exposes source region, version, and freshness.

A replicated catalog page may tolerate 90 seconds. A “did my order complete?” view after checkout may require read-your-writes. Ledgerline carries the committed order version in the session. The receiving region can serve the local projection only after it has applied at least that version; otherwise it reads home within the remaining deadline or reports pending. Sticky routing can reduce misses, but the version requirement—not the cookie—preserves the session guarantee.

Write locality has three broad forms:

1. route to authority: simple semantics, wide-area latency for remote writers;
2. move authority toward a stable demand center: better locality after a governed migration, but movement has state and fencing cost; or
3. accept locally and reconcile: lower immediate latency, but only legal with explicit conflict semantics.

Beware multi-key operations whose keys have different homes. Options include co-partitioning related state, coordinating across homes, reserving rights through escrow, decomposing into a saga with visible intermediate states, or rejecting a cross-home shape. Hiding it behind an aggregation service preserves the same physics.

Cross-region reads also consume capacity at the home. A fail-open policy that sends every lagging local read to authority can overload the write region precisely during replication impairment. Bound home fallbacks by traffic class, reserve capacity, and degrade low-value views first.

Route on useful-work health, not map proximity

Global traffic steering chooses an ingress; it does not prove the selected region can complete the operation. Inputs may include resolver answers, anycast reachability, client-side measurements, edge policy, service discovery, load-balancer health, residency, session state, authority assignment, and capacity.

Separate these decisions:

user -> nearest healthy edge
edge -> allowed regional data plane
data plane -> operation's authority or eligible replica
authority directory -> versioned home assignment

Each layer needs a fallback and an evidence age. A low network round trip to a region with a saturated stock shard is not healthy for checkout. A region that can serve catalog reads may be unable to accept payment. Health is operation- and class-specific.

DNS can participate in steering, but cached resource records live for their TTL and resolvers may apply additional behavior. Existing connections, connection pools, mobile clients, intermediaries, and stale-serving resolver policies extend convergence. A shorter TTL increases refresh and authoritative-query load and still does not revoke established sessions. Treat DNS as a coarse routing mechanism, not an instant evacuation barrier.

Prefer explicit traffic epochs or signed routing assignments for authority-sensitive clients. During a move, the old authority rejects new writes after epoch e+1 becomes active, and storage validates the epoch. A client routed by stale metadata can retry against the new home using the same operation identity. The old region cannot commit merely because DNS still points there.

Session affinity is a performance hint when it preserves warm caches, local projections, or connection state. It becomes dangerous when correctness depends on it. Clients change networks, cookies expire, a region disappears, and load balancers rebalance. Put durable session state and monotonic-version evidence behind the affinity layer.

Mobility therefore requires a session contract:

• stable user and operation identity;
• last-observed authoritative version where monotonicity matters;
• acceptable stale behavior while a new region catches up;
• an authority lookup that survives regional loss;
• bounded home fallback; and
• privacy controls on what routing/session metadata crosses regions.

Replication lag is an operation constraint

Lag is not one scalar. Track at least sender commit-to-ship delay, transport delay, receiver queue age, apply delay, index/materialization delay, and visibility delay. A byte offset can be current while an index is unusable. A median can be low while one hot partition is minutes behind.

Useful views include:

• commit-to-visible lag distribution by partition and destination;
• oldest unapplied authoritative version;
• bytes and service demand queued at each stage;
• apply goodput versus incoming replication rate;
• conflict count and resolution age;
• session reads forced to home;
• state missing after an evacuation point; and
• recovery time under foreground traffic.

For asynchronous replication, define acknowledgment meaning and recovery point. “Replicated” might mean buffered in a sender, durable in a regional log, stored at a destination, applied to authority, or visible through all derived indexes. The failover plan must use the same boundary as the user promise.

Conflict resolution belongs to the data type and invariant. Options include:

• reject concurrent changes and ask the caller to retry from a new version;
• deterministic merge with preserved intent and provenance;
• application reconciliation by a durable workflow;
• commutative/monotonic state under a proven model;
• compensation after both effects become visible; or
• single authority so the conflict cannot be admitted.

Record lost intent. A field-wise merge can create a state no user submitted. A timestamp winner can erase a paid action. An automatic reconciliation job needs identity, deadlines, retry budgets, observability, and a terminal manual path.

Replication bandwidth is recurring service demand

Cross-region cost includes logical payload, protocol framing, metadata, indexes, compression, encryption, retransmission, repair, snapshots, and fan-out. Measure applied useful state per byte as well as billable transfer.

Ledgerline models 18,000 writes/s, 1.4 KiB of logical replication material per write, 35% protocol/metadata amplification, and one destination:

18,000 writes/s × 1.4 KiB/write × 1.35
  = 34,836,480 B/s
  = 34.84 MB/s decimal
  = 33.22 MiB/s binary

Over one day, that is about 2.74 TiB to one destination before retransmission, snapshots, read repair, derived indexes, or a second replica destination. Fan-out to two destinations roughly doubles sender egress unless the transport topology relays or deduplicates it; relaying moves failure and recovery load elsewhere.

Bandwidth planning must cover burst and repair modes. If a destination is disconnected for 30 minutes, the retained log grows while new work continues. Catch-up rate must exceed arrival rate without starving foreground storage or network. A 2× transport rate is irrelevant if receiver apply capacity is only 1.05×.

Security and residency change the calculation. Encryption may add CPU; per-region keys change recovery dependencies; payload minimization may avoid transferring restricted fields; and redaction or tokenization can create another derived-data pipeline with its own lag.

The reproducible fixture at examples/performance-engineering-system-design-handbook/part-04/geo-distribution/ performs the unit conversions and exposes the assumptions. Its output is modeled teaching evidence, not a provider price or observed route.

Evacuation is a sequence of proofs

A regional evacuation is not “set traffic weight to zero.” It changes load, authority, state freshness, network paths, dependency populations, and user semantics. Use gates:

1. Declare the operation scope. Catalog reads, checkout writes, order views, and back-office jobs may take different paths.
2. Freeze dangerous change. Stop global rollout, resharding, bulk backfill, and automatic failback where they compete with recovery.
3. Classify source state. Failed, isolated, degraded, or merely unreachable from one observer are different.
4. Prove destination state. Identify the last durable/applied version, missing interval, conflict exposure, keys, configuration epoch, and indexes.
5. Prove destination capacity. Include shifted foreground, local demand, retries, connection churn, cache coldness, replication catch-up, and uncertainty.
6. Fence authority. Advance the authority epoch through a surviving quorum or declared recovery procedure; storage rejects the old epoch.
7. Shift a bounded canary. Use real useful work and compare invariants, goodput, queue age, and tails.
8. Ramp by class. Required writes and critical reads receive reserved capacity; optional work remains shed until stability holds.
9. Reconcile ambiguity. Retrieve operations by stable identity; do not regenerate writes.
10. Keep failback manual and staged. Rebuild and validate the old region before it can receive ordinary traffic or authority.

Chapter 34 protected 8,400 work/s after shedding optional work. With 20% modeled uncertainty headroom, a receiving boundary needs:

8,400 work/s × 1.20 = 10,080 work/s

If measured safe capacity is 10,400 work/s, only 320 work/s remains beyond that planning requirement. That slender margin must cover distribution skew, control traffic, probes, and modeling error. A nominal 12,000-request/s claim from the normal request mix is not evidence.

Capacity must exist at every constrained dependency: identity, keys/secrets, storage, queues, egress, observability, and operator control. Reserved compute in the destination is useless if the replicated database or regional key service cannot accept the shifted mix.

State the consequences before activating:

write RPO boundary: last destination-durable authority record
unknown outcomes: query by original operation key
local read freshness: version/age shown; home fallback disabled if unsafe
session guarantee: monotonic to recorded version or explicit pending
conflict policy: no dual stock authority; non-authoritative facts reconcile by ID
latency objective: evacuated checkout uses measured alternate path budget
recovery objective: protected goodput and oldest backlog age

Failback is a migration, not the reverse arrow

The original region may return with stale state, expired credentials, cold caches, old placement metadata, incomplete logs, and software/configuration drift. Sending traffic back immediately creates two recoveries at once.

A safe failback:

• rebuilds the region from an identified authority point;
• replays and verifies state through all required indexes and projections;
• reconciles or tombstones pre-failure ambiguous operations;
• applies current schema, policy, credentials, keys, and authority epochs;
• proves useful work with shadow and probation traffic;
• warms bounded caches without overwhelming authority;
• restores replication in the intended direction;
• ramps stateless/local reads before authority-sensitive writes;
• transfers authority with a new fenced epoch only if placement should move back; and
• preserves rollback to the current authority until stability windows pass.

Sometimes the correct failback decision is not to return authority. The incident may reveal that demand moved, the old placement violated latency, or the new home is operationally safer. “Primary region” is a decision record, not an ancestral right.

Measure failback separately from failover. Success includes no lost accepted effects, monotonic sessions, bounded lag, protected goodput, stable controllers, drained recovery work, and restored disaster reserve.

Keep the global control plane off the regional fast path

Global control planes manage placement, routing policy, certificates, schema, quotas, and rollout. They are tempting central authorities because a single view simplifies management. If every regional request synchronously depends on that view, a control-plane impairment becomes a global data-plane outage.

Regional data planes should operate from versioned, locally applied policy for a bounded interval. The contract names:

• last-known-good configuration and expiry policy;
• which changes require fresh global approval;
• local admission and safety behavior while disconnected;
• monotonic configuration/authority epochs;
• reconciliation after reconnect;
• emergency revocation path and its availability assumptions; and
• observability that does not require the failed global path.

Fail closed where stale policy could violate security, residency, or money invariants. Fail with bounded last-known-good behavior where availability is safe. Make the split per decision rather than declaring the whole control plane “fail open.”

Avoid one globally writable emergency switch without blast-radius controls. Use scoped targets, two-step intent and activation, staged propagation, dry-run evidence, velocity limits, signed changes, independent recovery access, and an audit trail. The tool used to evacuate a region must not accidentally evacuate every region.

Test one failure while preserving recovery

Regional resilience needs rehearsals, but a test that simultaneously removes traffic, control, identity, telemetry, and rollback only proves that unbounded correlated failure is destructive.

Progress from bounded experiments:

• block one application’s inter-region replication while preserving an observation path;
• delay or drop a percentage of cross-region calls to expose hidden coordination;
• make one destination apply slowly while transport remains healthy;
• withdraw a regional route for one traffic class;
• deny fresh global configuration while regional last-known-good state remains;
• promote a fenced shadow authority for a disposable partition;
• evacuate a small tenant/cell with verified destination capacity;
• kill the source after acknowledgment at each replication boundary; and
• fail the failback canary to prove rollback.

Define aborts before injection: invariant violation, queue-age slope, protected-goodput loss, conflict growth, destination reserve consumption, observability loss, or inability to fence. Keep a recovery path outside the fault domain under test.

Collect an evidence packet: route distributions from user populations, authority versions, replication-stage lag, receiver demand, conflict records, operation outcomes, routing epochs, DNS/connection convergence, controller actions, and time to stable reserve. A green region dashboard without operation-level evidence is insufficient.

Geo-design worksheet

Use one row per operation, not one per service:

Reject the design if an invariant-sensitive operation has no authoritative home, an asynchronous path has no lag/freshness boundary, or evacuation has no capacity and fencing proof.

Geography drills

Place six operations. For catalog browse, inventory reservation, price quote, order view, analytics ingest, and account deletion, select authority and replica locations. Name the invariant, residency rule, session behavior, and failure result. More than one topology may be valid; assumptions must make it valid.

Spend the latency budget. Reproduce the 152 ms modeled round trip and 28 ms remainder. Add TLS/connection setup, p99 queueing, and one dependency. Decide whether to move execution, reuse a connection, weaken an objective, or reject the topology. Do not subtract percentiles as though independently composable without labeling the approximation.

Price replication. Reproduce 34.84 MB/s, 33.22 MiB/s, and about 2.74 TiB/day. Add two destinations, 8% retransmission, a 30-minute outage, and 1.6× receiver catch-up. Identify sender, network, receiver, and cost constraints.

Design session mobility. A buyer commits version 918 in Frankfurt and immediately reads through Singapore. Specify token contents, local projection test, bounded home fallback, privacy controls, and the response if neither path can satisfy the remaining deadline.

Evacuate without dual authority. Use 8,400 work/s protected demand and a 10,400-work/s receiving boundary. Add 20% uncertainty, a hot tenant, and cold caches. Define authority epochs, canary size, stop conditions, and unknown-operation recovery.

Red-team the control plane. Assume the global configuration store is partitioned, one region has a stale traffic map, and the emergency tool can target all regions. Decide which regional operations continue, which fail closed, and how operators regain scoped control.

Durable rules for geo-distribution

1. Derive placement from operations, state, invariants, residency, and failure behavior—not regional symmetry.
2. Treat distance and route distributions as budget floors; replicas do not erase coordination latency.
3. Separate read locality, write authority, session guarantees, and routing hints.
4. Use active-passive, active-active, and home-region labels only after specifying actual read, write, and conflict paths.
5. Make asynchronous replication’s acknowledgment, lag, visibility, RPO, and conflict semantics explicit.
6. Keep authority stable across user mobility; carry versions when sessions require monotonic views.
7. Treat DNS and global steering as convergent mechanisms, not instantaneous correctness barriers.
8. Price logical bytes, amplification, destinations, catch-up, apply capacity, and repair.
9. Evacuate by proving destination state, capacity, dependencies, and fencing before traffic ramp.
10. Fail back through resynchronization, probation, and a new authority decision.
11. Let regional data planes use bounded locally applied policy when the global control plane is impaired.
12. Inject regional faults with scoped aborts and an independent recovery path.

Regional traffic steering and reserved capacity are not instantaneous. Signals arrive late, new instances warm slowly, state moves, and multiple controllers react to the same demand. The next problem is therefore temporal: controlling capacity without turning delayed feedback into oscillation.

Evidence and transfer limits

• The Spanner paper documents one globally distributed design that pays for synchronous replication and external consistency while exposing clock uncertainty. It is an implementation study, not a command to coordinate every Ledgerline operation globally.
• Amazon’s primary Dynamo paper explains an availability-oriented design with object versioning and application-assisted conflict resolution. Its workload and trade-offs do not make conflict acceptance safe for stock or payment invariants.
• The primary Bayou conflict-resolution paper presents application-specific detection and merge in weakly connected replicas. Its mobile/disconnected model is evidence that conflict policy is application work, not a universal merge recipe.
• RFC 2181 clarifies DNS TTL and authoritative/cached data semantics; RFC 8767 permits bounded stale DNS serving under specified resolver failure conditions. Neither makes DNS an authority-fencing or instant-session mechanism.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/geo-distribution/ reproduces the 152 ms modeled round trip, byte-unit conversions, daily replication volume, and 10,080-work/s evacuation requirement. All inputs are explicit teaching assumptions and must be replaced with measured routes, workload distributions, capacity tests, and current cost data for a real design.