Case Study: Multi-Region Ordering and Payment

Ledgerline’s regional-failure review begins with an attractive requirement: a customer should be able to browse, add to a cart, and place an order against any healthy region, even if the nearest region is partitioned. The first architecture proposal translates that into “every region accepts every write locally.” It promises low latency and active-active availability. It also creates three inventory owners, three possible order records, and three opportunities to submit the same payment.

The review rejects the proposal before choosing a database. The phrase global experience says where customers are; it does not say which operations may commute, which observations may be stale, or which state may have multiple writers. The design must first answer a narrower question: what invariant would be violated if two regions acted independently during a partition?

For Ledgerline, browsing can tolerate a boundedly stale catalog. A cart is provisional and mergeable. Inventory cannot promise the same scarce unit twice. One logical checkout must not create two orders. A payment retry must not create a second economic effect. Those semantics lead to different placement and failure choices even though the operations appear in one user journey.

The design lesson is: global experience does not require globally coordinated execution for every operation; invariants determine where coordination belongs. The numbers below are deterministic modeled and simulated teaching evidence for a fictional platform, not measurements of a named database, payment provider, or network.

Start with an invariant ledger, not a region map

Ledgerline serves the Americas, Europe, and Asia Pacific. At the modeled peak it receives 2,200 logical checkouts/s: 1,100, 660, and 440 respectively. Transport retries raise offered attempts to 2,376/s. Browse traffic is much larger, but browse reads do not belong in the checkout coordination budget.

The state ledger separates five semantics.

This table prevents a common category error. A cart line that says “quantity 2” is customer intent, not an inventory reservation. A replicated payment status is an observation, not permission to submit another charge. A locally cached product page is not the authority for price at checkout. The UI may look continuous while the underlying state has different owners.

The invariant ledger also makes degradation honest. During a partition, Ledgerline may keep browsing and cart edits available while refusing checkout for an exhausted regional token pool. That is a smaller availability surface than “all writes everywhere,” but it preserves the economic promise.

Compare three coordination shapes

The team evaluates three plausible designs against the same workload and failure states.

Global synchronous writes

Inventory, order, and payment metadata participate in a globally synchronous transaction or equivalent coordination path. The benefit is a simple global ownership story and flexible inventory. The cost is that ordinary checkout inherits wide-area latency and quorum availability. In the teaching model, p99 checkout is 392 ms. The 96–176 ms inter-region round trips are not constants of nature; they are scoped inputs representing the measured test topology. The important point is structural: a critical path that requires multiple distant coordination phases pays distance even when the customer and inventory are local.

A system with externally consistent transactions can provide strong ordering across regions. Google documents that Spanner’s default serializable mode provides external consistency, including real-time transaction ordering across servers and regions. That is a meaningful implementation capability, but it does not make coordination free or prove that every Ledgerline operation needs the guarantee. The design still chooses transaction scope. See Spanner’s external-consistency documentation.

Customer home-region ownership

Each customer has one writable home. Other regions forward checkout there. This avoids concurrent order writers and makes idempotency lookup straightforward. It performs well when customer, inventory, and payment route align. It performs poorly when customers travel, a home region fails, or the required inventory is owned elsewhere. It also turns home reassignment into an authority transfer that must be fenced. The model yields 244 ms p99, but the number hides a placement tail: Asia-Pacific customers homed in the Americas pay a long remote path even when Asia-Pacific has stock.

Partition inventory, then colocate order authority

Ledgerline divides sellable launch inventory into regional token pools: 5,700 Americas, 3,300 Europe, 2,100 Asia Pacific, plus a 900-unit central safety reserve not offered concurrently. A successful reservation consumes a token at one owner. The order authority follows that token. Payment remains a separate idempotent effect tied to the resulting order identity.

This design yields 171 ms modeled p99 for the declared mix, 221 ms less than the global-synchronous alternative. It does not create capacity; a region can sell out while another retains stock. Rebalancing tokens is an explicit fenced transfer, not an eventually consistent counter update. The safety reserve exists precisely because using 100% of stock as independently writable regional allocations would leave no controlled response to skew or evacuation.

The selected design trades some global pooling efficiency for a bounded coordination domain. That trade is defensible for a scarce launch item because oversell cost is high. A commodity item with abundant replenishment and cheap compensation might use a different allocation policy.

The checkout saga names every ambiguity

The workflow cannot rely on one atomic transaction across Ledgerline and an external payment boundary. Instead it uses a saga with narrow local transactions and durable transitions:

client                         order authority                 payment boundary
  | POST /checkout (key K)            |                              |
  |---------------------------------->|                              |
  |                            look up K                              |
  |                            reserve token T                       |
  |                            create order O:PENDING_PAYMENT         |
  |                            append payment command (O,K)           |
  |                                   |---- submit intent (O,K) ---->|
  |                                   |<--- accepted/declined/? ------|
  |                            record terminal or UNKNOWN             |
  |<---- O + status or ambiguity -----|                              |
  | GET /orders/O/status              |---- query intent O ---------->|
  |<---- authoritative status --------|<--- provider status ----------|

The checkout idempotency key names one logical customer intent. Ledgerline stores the key, a hash of relevant request parameters, the order identity, and the durable result. A retry with the same key and different basket or amount fails rather than mutating the first operation. Concurrent attempts race at the same authority; one creates the order and the others observe its result.

Stripe’s API documentation is one concrete example of server-side idempotency: it describes retaining the first result for a key and rejecting reuse with different parameters. Ledgerline is not specified as Stripe and does not inherit Stripe’s retention or error behavior; the source supports the general need to define key identity, parameter equality, and result retention explicitly. See Stripe’s idempotent-request reference.

Idempotency is not exactly-once transport. The fixture sends 108,000 attempts for 100,000 logical checkouts. The system produces 100,000 orders and 100,000 payment intents; 8,000 duplicate attempts are recognized. That result depends on durable key scope and retention outliving the retry horizon. If keys expire while old clients can retry, duplicate effects become possible again.

The saga has three compensations, none described as time reversal:

• If reservation succeeds but order creation fails in the same authority transaction, the token remains unconsumed because both changes abort together.
• If the payment is definitively declined, the order enters PAYMENT_DECLINED and a durable release command returns the inventory token. The release is itself idempotent.
• If payment submission times out, the order enters PAYMENT_UNKNOWN. Ledgerline does not submit a new payment with a new key and does not release inventory immediately. It queries by order identity until it learns a terminal result or reaches an operator-owned exception policy.

The last state is user-visible ambiguity. The correct response may be “We are confirming your payment; do not retry with a new checkout.” Returning a generic failure would invite a second logical intent. Claiming success would be equally unsafe. The status resource is part of the API contract, not merely an internal recovery tool.

Distance, replication, and reserve are one capacity model

Ledgerline budgets synchronous latency separately from asynchronous traffic. In the selected design, a normal checkout performs local admission, local reservation and order commit, then a payment interaction. Remote catalog, status, and analytics propagation do not sit on the success critical path.

The teaching model uses p99 inter-region RTTs of 96 ms between Americas and Europe, 148 ms between Americas and Asia Pacific, and 176 ms between Europe and Asia Pacific. These inputs must be measured from the actual client and service placement. They cannot be inferred from geography alone, and adding p99 values does not produce an end-to-end p99.

Each logical checkout emits a 1,433.6-byte order event to two replica destinations:

2,200 checkouts/s × 1,433.6 bytes/checkout × 2 destinations
  = 6,307,840 bytes/s

Analytics adds 1,408,000 bytes/s and catalog changes add 850,000 bytes/s, for 8,565,840 bytes/s of declared application payload. This is not wire capacity: protocol, encryption, batching, retries, replication acknowledgments, and recovery replay add overhead. The model exists to catch order-of-magnitude omissions and to ensure a recovery link is not sized only for steady state.

Failover capacity is also not “autoscaling will handle it.” Europe normally contributes 660 logical checkouts/s. The Americas reserve 850/s for a declared Europe evacuation, leaving a 190/s reserve margin. During the exercise the target reaches 1,760/s against a tested safe capacity of 1,900/s, leaving 140/s headroom. Both margins are intentionally small enough to demand an operator decision: if baseline or retry amplification grows, the evacuation plan is no longer valid even if every instance is healthy.

Regional capacity is reserved across the whole path—ingress, order authority, database write units, payment connections, queues, and replication—not only HTTP workers. A warm service tier backed by a cold connection pool or an unprovisioned payment quota does not constitute failover capacity.

Evacuation is an authority transfer, not a routing change

The Europe partition test separates three states:

1. uncertain: Europe and the control plane disagree about reachability; Europe retains generation 41 and may have committed work not yet replicated;
2. fenced: generation 41 can no longer acquire the write resource or publish acceptable commands; operators reconcile the last durable checkpoint; and
3. transferred: the Americas activate generation 42 for the declared token subset and admit evacuated traffic within reserve.

Traffic routing changes only after fencing evidence. DNS health or load-balancer reachability is insufficient because an isolated region may still reach inventory or payment dependencies. The design rejects dual writers even if that temporarily reduces checkout availability.

During the fixture, 317 delayed generation-41 messages arrive after transfer. Consumers reject them by authority generation rather than arrival time. A timestamp alone would be unsafe under clock error and message delay. The messages remain in an audit stream so operators can determine whether a command represented already-committed work, an obsolete retry, or a reconciliation defect.

Failback repeats the same discipline in reverse. Europe comes back as a read-only replica, catches up, verifies order and payment projections, and passes shadow traffic. A new transfer generation is issued only after the Americas owner is fenced for the returning subset. “Primary healthy” is not a failback condition.

Amazon’s Builders’ Library describes minimizing correlated failures and workload isolation as operational techniques; the durable lesson for Ledgerline is that redundant regions must not share an unexamined control or capacity failure. The exact cell and shuffle-sharding mechanisms are implementation choices, not correctness proofs. See Minimizing correlated failures in distributed systems.

Test the states that happy-path checkout conceals

The validation packet uses fixed simulated observations. Every run reconciles logical checkouts, transport attempts, inventory tokens, orders, payment intents, messages, and terminal states.

The packet records 1,240 client outcomes resolved by status lookup. That is not an error count to erase from a dashboard. It measures how often the ambiguity contract matters. Operators track age and population of PAYMENT_UNKNOWN; a growing queue can consume reserved stock and become a capacity failure even when no duplicate charge occurs.

Run the packet:

cd examples/performance-engineering-system-design-handbook/part-08/multi-region-ordering-payment
node analyze.mjs
node verify.mjs

It reproduces regional rates, retry attempts, inventory conservation, replication payload, failover margins, latency-alternative differences, fencing generation, and duplicate-effect assertions.

The compact design record

Decision. Keep catalog reads and carts region-local; allocate inventory tokens to one regional owner; colocate order authority with the consumed token; submit payment intent with a durable order-scoped idempotency key; replicate status asynchronously; evacuate only after fencing and within reserved path capacity.

Rejected. Global synchronous coordination for all commerce state spends distance on operations whose semantics do not require it. Customer-home ownership couples travel and regional loss to a distant writer. Independent regional counters cannot protect scarce inventory during partition. Creating a new payment request after timeout converts ambiguity into duplicate-effect risk.

Evidence. ledgerline-checkout-v1, 2,200 logical checkouts/s, 8% retry attempts, declared regional mix and WAN inputs, deterministic duplicate/delay/partition/evacuation/failback simulations, and reconciled effect counts.

Rollback. Routing and local-read changes can roll back independently. Token movement and authority transfer require forward reconciliation; once generation 42 owns a subset, restoring generation 41 is not a configuration rollback.

Revisit. Regional demand changes by 15%; retry attempts exceed 1.12 per logical checkout; evacuation headroom falls below 10%; ambiguity age exceeds the reservation policy; transfer reconciliation differs by any order or token; replication recovery cannot meet its objective; or product policy changes oversell and stale-price costs.

Design exercise: change the inventory economics

Assume 80% of items are abundant and cheap to compensate, but 20% are scarce launch items. Customers value checkout availability more than exact regional allocation for abundant items. Redesign authority, reservation, payment ambiguity, evacuation, and user messaging. State which operations remain locally available during partition and which evidence would justify a different choice.

Answer guide

A strong design does not apply the scarce-item token policy universally. Abundant items may use a bounded oversell budget, asynchronous reservation, or one home authority with a compensating fulfillment path. The design must quantify the expected oversell population and compensation cost rather than calling the item “eventually consistent.” Scarce items retain single-owner tokens or globally coordinated reservation.

Payment idempotency remains necessary for both classes because inventory economics do not make duplicate charges acceptable. The order can contain lines with different reservation states, but the customer contract must explain partial fulfillment and cancellation. Evacuation capacity must model the increased local admission of abundant-item orders plus the still-fenced scarce path. A valid alternative could choose stronger global coordination for scarce stock if its measured latency and availability meet the product objective.

Field review card

When designing multi-region commerce:

• Separate browse, cart, reservation, order, and payment semantics.
• Name the invariant and authoritative writer for each state transition.
• Treat a cart as intent, not reserved stock.
• Put stale, replicated, local, and asynchronous behavior in the contract.
• Compare global coordination, home ownership, and partitioned authority under the same failure states.
• Count logical operations and transport attempts separately.
• Give idempotency keys scope, parameter equality, durability, and retention.
• Represent payment timeout as ambiguity, not automatic failure.
• Describe compensation as a new idempotent action, never time reversal.
• Budget WAN latency and replication bytes with explicit boundaries and units.
• Reserve failover capacity across every constrained dependency.
• Fence the old writer before routing writes to the new one.
• Reject delayed messages by generation, not arrival order.
• Reconcile before failback and test recovery as a state transition.
• Record what the model cannot establish about a real network or provider.

Ledgerline’s global experience comes from local reads, local provisional state, and carefully bounded authority—not from pretending distance and partitions have disappeared. The next case removes correctness coordination from center stage and shows a different trap: a cache can report excellent aggregate efficiency while one key and one cold transition destroy the origin.