Replication, Consensus, Sharding, and Multi-Region Reality

One Seat Has to Belong Somewhere

At 09:00, tickets for a stadium concert go on sale. A buyer in Nairobi and another in London both ask for seat A-14. The catalog page may be a few seconds stale; the reservation result may not. Somewhere in the system, one actor must have authority to say which request won.

Adding copies does not remove that decision. A replica can be behind. Two regions can lose contact. A shard can move while a write is in flight. A newly promoted leader can have a different view from the process still accepting traffic. Distribution makes the system larger by dividing and copying authority, and every such move creates a failure boundary.

The useful design question is therefore not “should we distribute the database?” It is: which fact needs to be owned, copied, agreed, or recovered somewhere else, and what must happen when those places cannot communicate?

Name the Need Before the Topology

“Scale” hides several different problems. Read scale means the authoritative write path is adequate but readers need more capacity or lower latency. Read replicas, caches, or derived views may help, with a freshness cost. Write scale means one writer, partition, log, or storage node cannot absorb the workload with acceptable latency and headroom. Sharding may help, while making cross-partition queries and invariants harder.

Availability is different again. The requirement should name the failure the service must survive: a process, node, zone, region, maintenance event, or operator mistake. It should also name the operations that may stop. The ticket catalog might stay available during a regional failure while new reservations pause. That is a coherent product decision, not an infrastructure embarrassment.

Geographic latency asks which users are too far from the authoritative path. Nearby read copies may solve that problem without allowing writes everywhere. Residency asks where data and its backups, support access, and deletion workflows are permitted to exist. Disaster recovery asks how much committed work may be lost and how long restoration or promotion may take. These motives can coexist, but they do not imply the same design.

A measurable sentence is a better starting point than a topology: “Catalog reads should complete within 100 ms in three regions; reservation writes may cross to the event’s home region and must never sell one seat twice.” Now the architecture has work to do.

Draw the Authority Map

The ticketing service does not have one consistency requirement. Its reservation, availability count, confirmation, and analytics record are different facts with different owners. Putting them in an authority map prevents the strongest rule from burdening every read and the weakest rule from leaking into a sale.

Only the seat reservation needs agreement on the sale path. The availability count needs visible freshness. Confirmation needs an idempotent boundary. Analytics needs replay and deduplication. “Replicate the ticket database globally” says nothing useful until these contracts are separated.

A Copy Needs a Contract

Suppose the seat’s home shard has a leader and two followers in separate failure domains. The leader accepts reservation writes and sends its log to the followers. This arrangement gives the write path a clear owner, but it does not by itself say when a write is safe to acknowledge or which replica may answer a read.

With synchronous replication, the leader waits for the configured confirmation before returning success. The buyer pays at least the communication delay, and a slow or unreachable required replica can stall the write. In return, the acknowledged reservation has reached more than one place under the system’s stated durability rule. With asynchronous replication, the leader can respond before followers catch up. That reduces latency and lets the leader continue through follower delay, but promotion of a stale follower can lose an acknowledged reservation unless failover eligibility and the recovery-point objective explicitly permit or prevent it.

Read semantics require their own choice. A regional follower may serve the event page because a stale count is tolerable. It should not silently serve the post-purchase view if the product promises that a buyer will immediately see the reservation. That path can read from the authority, wait for a known replication position, or carry a session token that proves the required version is visible. “Reads go to replicas” is not a contract.

Multi-leader replication permits writes in more than one place. It helps only when concurrent changes can be kept separate or resolved according to a business rule. Two regions can merge independent profile edits if the system has field, version, or conflict semantics. They cannot both sell A-14 and repair the meaning of “one sale” with last-write-wins. A timestamp can select a stored value; it cannot refund the losing buyer, reverse fulfillment, and restore trust.

Leaderless designs send reads and writes to several replicas and use thresholds to decide when an operation has enough responses. Overlapping read and write quorums can be part of a sound design, but arithmetic alone does not settle concurrent writes, stale responses, failed writes, membership changes, or repair. The full contract includes version comparison, conflict handling, read repair or anti-entropy, tombstones, slow-replica behavior, and what the client observes when the threshold cannot be reached.

Replication makes copies. Authority, acknowledgment, read freshness, promotion, and repair determine what those copies mean.

Agreement Has a Scope

For the seat shard, a consensus protocol can make the reservation log behave as one ordered history while some members fail. Trace the mechanism rather than treating “consensus” as a reliability adjective. A leader receives the command, appends it to its log, and replicates it to voting members. When the protocol’s commit rule is satisfied, the entry becomes safe to apply and acknowledge. If the leader is cut off from a quorum, it must not keep committing merely because it still has clients. A new leader can be elected only from the side able to satisfy the protocol, and fencing must stop the old authority from returning later as a second writer.

This is where safety becomes visible as unavailability. During an ambiguous partition, reservation writes may stop while catalog reads continue. Refusing a sale is the correct result if the alternative is selling the same seat twice. The product still needs to decide what the buyer sees, whether requests queue, how long clients retry, and who intervenes when quorum does not return.

Consensus is useful for replicated logs, leader election, membership, strongly consistent metadata, leases, and other decisions that require one agreed order. It does not make every operation globally transactional, repair external side effects, or erase network delay. Nor must every fact pay for it. The reservation can agree within its home shard while the regional catalog remains an asynchronous projection.

Quorum placement is consequently a product decision. Keeping voting members within one region bounds normal write latency but makes region loss a recovery event. Spreading them across regions may preserve a live quorum through a regional failure, but every coordinated write pays wide-area latency and a bad partition can make the unavailable side surprising. The design must say which failure it optimizes for and which users fund that choice on every request.

A Shard Key Places Both Load and Rules

As sales grow, the ticket service may outgrow one write group. Sharding divides the keyspace so different owners can process work independently. The shard key decides more than where bytes live: it decides which reads stay local, which invariants can be enforced by one authority, what becomes hot, how much a move disturbs, and which customers share a failure.

Hashing by seat ID can spread point writes well, but a query for every seat in an event may fan out. Range partitioning by event preserves event locality, but the most popular sale can overload one range. A composite placement such as event plus seat block can distribute a large event while keeping each seat’s reservation rule inside one shard. It also makes an operation across several adjacent seats a cross-shard problem unless blocks are chosen with that workflow in mind.

The right key follows the workload’s hardest local fact. If support mostly retrieves all data for one tenant, tenant locality has value. If billing scans time windows, time placement may help, though an append-only current range can become the hottest partition. If one global counter receives every write, no hash function removes the fact that the invariant itself is global.

Before sharding, test cheaper explanations for the pressure: a missing index, an expensive query, unbounded retention, analytical work on the transactional path, avoidable write amplification, inadequate capacity, or a burst that a queue can smooth. Early sharding buys routing bugs, fan-out, cross-shard transactions, uneven load, tenant moves, and new recovery procedures before evidence requires them.

When a shard must move, ownership transfer is a protocol, not a copy command. Routing needs a version or epoch. The old owner must be fenced from new writes, the new owner must catch up to a known position, and clients must converge on the new route. The move needs throttling because copying a hot range consumes the same network, disk, and CPU serving customers. Interrupt it deliberately before trusting it in production.

Geography Does Not Remove Ownership

The ticket service now has users on three continents. The least surprising design may keep each event in a home region, serve catalog projections nearby, and route reservations to the home authority. Users pay cross-region latency for the scarce fact; cheap reads stay local. Region loss can pause sales for affected events without turning every healthy region into a competing seller.

An active-passive design uses one authoritative region and prepares another for promotion. Its credibility lies in the unglamorous details: how fresh the standby is, who may promote it, how the old region is fenced, whether dependencies and credentials work in the recovery region, how clients reroute, and how the system eventually fails back. The promised recovery point and recovery time belong to the workflow, not merely to the database service.

Active-active means more than running application servers in two regions. For every mutable fact, the design must choose one of a small number of truths: authority stays regional; writes coordinate across regions; capacity is reserved or escrowed so regions cannot overspend it; conflicts are acceptable and repairable; or the operation is unavailable during partition. A follow-the-sun writer still transfers ownership and fences its predecessor. It does not make authority disappear.

Clock skew makes casual conflict rules especially dangerous. A wall-clock timestamp from another machine or region is not proof that one business action should defeat another. Use an ordering or version rule whose scope is explicit, then retain enough history to inspect and repair conflicting actions. Chapter 10 follows that problem into retries, duplicate delivery, and late events.

Rehearse the Failure, Not the Diagram

Return to A-14. Assume its home shard has three voting replicas across zones, catalog projections in three regions, and an asynchronous disaster-recovery copy. Before launch, run a drill that makes each promise observable.

First, reserve seats under normal load and record which log position makes a reservation acknowledged, when each read path exposes it, and what support sees during lag. Then isolate the leader from the other voters. The old leader must stop committing; a new leader must either be elected safely or writes must pause. Drive traffic at both sides of the partition and prove fencing prevents two authorities.

Next, delay one follower and direct catalog and post-purchase reads toward it. The catalog may be stale within its stated budget. The buyer’s confirmed reservation may not disappear behind an unlabeled replica read. Lag, quorum health, routing decisions, and stale-read fallbacks must be visible without reconstructing the incident from raw logs.

Now remove the home region. Record the newest reservation present in the recovery copy before anyone promotes it. If the design permits losing recent acknowledged writes, the observed loss must fit the declared recovery point and have a reconciliation path. If it promises no such loss, promotion must refuse a candidate that cannot prove the required position. Exercise DNS or service routing, secrets, queues, object storage, third-party dependencies, support tools, rollback, and failback; a healthy database in an unusable region is not recovery.

Finally, run a flash sale that makes one event or seat block hot. Watch per-shard queueing, tail latency, throttling, and neighbors that share the shard. Start a split or move, interrupt it, and resume it. A rebalance that works only when the system is quiet is not a production mechanism.

These failures also reveal blast radius. Cells and tenant partitions can isolate customers only if routing, deployments, credentials, metadata, and migrations respect the same boundaries. One global control plane or bad schema rollout can reconnect every supposedly independent cell into a single incident.

Write the Distribution Justification

The architecture review should leave behind a short argument, not a claim that a product “handles” distribution. It should answer:

1. Which measured need cannot be met by an index, query or model change, capacity increase, cache, archive, queue, read replica, or single-region failover design?
2. For each important fact, who may accept a write, which copies may serve it, what freshness is promised, and which conflicts can be repaired?
3. Which requests pay coordination latency, and what continues, pauses, queues, or fails when a quorum, shard, link, or region is unavailable?
4. Why does the shard or regional ownership rule keep the important invariant local, and how are hot keys, moves, splits, merges, promotion, and failback handled?
5. Which drill proves the failure behavior, which signals expose lag and divergence, who owns the response, and how can the distributed path later be shrunk or retired?

The document is complete only when its costs are concrete enough to reject the design. If distribution is still justified after that, the team knows what it is choosing.

Exercise

Take one workload proposed for sharding or active-active regions. Write its motive as a measured sentence, then propose three less-distributed alternatives. If the proposal survives, make an authority map for four facts and choose one shard key or regional ownership rule.

Now write the failure drill as a sequence an operator could run. Include a stale replica, loss of quorum, ambiguous old authority, a hot shard, an interrupted move, and promotion of a recovery copy. For each step, state the expected user-visible result and the evidence that distinguishes safety from accidental success.

Distribution does not abolish the single source of truth. It makes truth a protocol: ownership, copies, agreement, placement, fencing, and recovery. The design is ready when the team can say where each fact belongs, what stops when communication fails, and how it will know the promise survived.