Consistency, Coordination, and Transaction Boundaries

“Make reservations eventually consistent so the checkout path stays fast” sounds like a performance decision. It is not yet a coherent design.

Ledgerline has eight remaining units of capacity for an event. Two transactions start from the same snapshot:

transaction A                         transaction B
-------------                         -------------
read remaining capacity = 8           read remaining capacity = 8
insert reservation A for 6            insert reservation B for 6
update allocation row A                update allocation row B
commit                                commit

The writes touch disjoint rows, so a snapshot-isolation implementation may see no direct write/write conflict. Both transactions commit. Twelve units are now reserved against eight available. Every row can be valid, every replica can eventually converge, and the business invariant can still be false by four.

The useful statement is not “we need strong consistency.” It is:

[ \sum \text{active reservation quantity for event } e \leq \text{sellable capacity for } e ]

Now the design can ask which operations might jointly violate the bound, where the capacity authority lives, which anomaly must be excluded, and how much coordination that exclusion costs. The answer might be a single-owner conditional update, a serializable transaction, preallocated escrow rights, or a changed reservation shape. A saga after overselling does not make the invariant true; it defines a business response to having broken it.

This is the method for any consistency discussion: pay coordination cost only where an explicit invariant requires it, and never weaken semantics without naming the anomaly the application must absorb.

Invariants turn preferences into testable histories

An invariant is a predicate that must hold over allowed states or histories. Examples differ in the coordination they require:

• an idempotency key maps to at most one logical operation within its scope;
• a username is unique within one tenant;
• an account balance never falls below a contractual floor;
• allocated inventory does not exceed sellable inventory;
• a workflow never captures payment before an order has durable identity;
• a document’s set of tags converges after replicas receive the same updates; and
• a user who completed a write does not read an older version in the same session.

State when the invariant must hold. A uniqueness rule required at acknowledgment is different from a duplicate-detection job that may repair later. State where it holds: one key, a partition, several rows, multiple services, or a business process spanning an external provider. State the failure behavior: reject, wait, reserve rights, return unknown, compensate, or permit a named temporary violation.

Correctness is not automatically delivered by serializability. A serializable database can faithfully serialize two transactions that never check the real business predicate. Conversely, not every invariant requires global serial execution. Operations on disjoint tenants may coordinate only within tenant. Add-only updates whose merge preserves the invariant may proceed independently. The application model and operation set determine the boundary.

The reservation review records four distinct rules:

This table prevents a convenient database label from substituting for domain reasoning.

Consistency labels answer different questions

Linearizability orders operations on an object

Linearizability makes each operation on a concurrent object appear to take effect at one instant between invocation and response, while respecting the real-time order of non-overlapping operations. If one client completes set(x, 2) before another begins read(x), a linearizable object cannot return the prior value.

The object boundary matters. Linearizable reads of two separate counters do not automatically provide an atomic snapshot across them. A per-key linearizable store does not make a multi-key reservation transaction serializable. “Strong consistency” often hides this boundary; replace it with the exact object and history rule.

Serializability orders transactions

Serializability requires committed transactions to have an outcome equivalent to some serial order. It protects multi-object relationships when transactions actually read and write the predicate-relevant state. Plain serializability does not necessarily require that order to match wall-clock completion. Strict serializability, also called external consistency in some contexts, adds a real-time constraint across transactions.

The cost can include coordination, validation, locks, aborts, version retention, and retry. Read-only snapshots may avoid blocking writes in multiversion designs while still observing a consistent transaction timestamp. The implementation mechanism and workload determine cost; the label alone does not predict latency.

Snapshot isolation stabilizes a view but permits write skew

Snapshot isolation commonly lets a transaction read from a stable snapshot and rejects some direct write/write conflicts. It can provide excellent concurrency, yet transactions that read a shared predicate and write disjoint items may both commit. The reservation trace is write skew.

Treat isolation anomalies as production behaviors:

Labels vary across products and versions. Verify the permitted histories and retry contract of the implementation in use.

Causal and session consistency preserve selected relationships

Causal consistency preserves the order of operations that are causally related while allowing concurrent independent operations to appear in different orders. It can support collaboration, feeds, or replicated state where causality matters more than one global order. The system needs causal metadata or an equivalent protocol, and metadata can grow or be compressed with trade-offs.

Session guarantees narrow the promise to a client’s observed history. Read-your-writes prevents a completed session write from disappearing on the next read. Monotonic reads prevent the session from moving backward. Monotonic writes and writes-follow-reads preserve other causal relationships. These guarantees can retain much of a weakly consistent system’s locality, but they are end-to-end properties, not mere connection affinity.

Eventual convergence says little about intermediate truth

Eventual convergence says replicas reach the same state once updates stop and communication succeeds under the merge rules. It does not specify which state is correct for an arbitrary business invariant, how stale a read may be, whether a session moves backward, or whether a temporary decision can cause an irreversible external effect.

Ask four questions whenever “eventual” appears: converge to what function of concurrent updates, after which delivery assumptions, within what operational objective, and which intermediate anomalies may users or downstream systems observe?

Coordination buys exclusion, order, or knowledge

Coordination is communication that constrains concurrent actions or establishes what other participants know before progress. It appears as a leader round trip, quorum, lock wait, validation, lease renewal, prepare/commit exchange, rights transfer, or failure recovery.

Its cost has more than one dimension:

• Latency: at least the slowest required participant or quorum path, plus durable work and queueing.
• Availability: if required participants or a quorum cannot communicate, the operation may wait or fail rather than violate the invariant.
• Contention: one hot predicate can serialize unrelated-looking row writes.
• Capacity: protocol messages, logs, versions, retries, locks, and validation consume resources.
• Recovery: uncertain coordinators, expired leases, in-doubt transactions, and old terms require resolution.
• Complexity: the protocol and application must agree on identity, retries, visibility, and error semantics.

Coordination is not inherently bad. It is the correct price for excluding a history the business cannot accept. The engineering task is to keep the coordinated scope no larger than the invariant.

The fixture models a two-participant path: 2 ms of client/application work, parallel participant round trips of 8 ms and 13 ms, 4 ms of prepare durability, another parallel 8/13 ms commit round, and 2 ms to return. The total is:

[ 2 + \max(8,13) + 4 + \max(8,13) + 2 = 34\ \text{ms} ]

The maxima reflect parallel participants; summing 8 and 13 would overstate each round. The 34 ms is still only a modeled no-contention path. Queue wait, lock conflict, log batching, retransmission, leader changes, and abort/retry can dominate its tail. Conversely, a protocol may overlap durable work or return under a different commit rule. Measure the actual path.

Consensus establishes one replicated decision sequence

In practical state-machine replication, a leader proposes entries in a term, replicas persist or acknowledge them, a quorum establishes commitment under the protocol’s rules, and state machines apply the same committed order. Terms or ballots distinguish leadership generations. Quorum intersection prevents two conflicting committed prefixes under the protocol assumptions.

Consensus can decide replicated log entries even when a minority of nodes fail. It does not define the application’s transaction boundary, make arbitrary operations commutative, or guarantee that an external payment effect happened exactly once. Those require a state machine command, idempotent effect boundary, or transaction/process protocol.

Read behavior needs equal care. A leader that has lost authority must not serve a supposedly current read. Implementations use quorum confirmation, read-index protocols, leases with assumptions, or reads at committed timestamps. A follower can serve an older snapshot if the contract permits it. “The log uses consensus” is not a complete read guarantee.

Consensus latency depends on placement, batching, stable-storage policy, leader load, and failure state. A three-voter group across nearby zones behaves differently from a group spanning continents. Under leader loss, election and catch-up may pause progress even though safety remains intact. Chapter 27’s recovery and bandwidth budgets still apply.

Leases need clocks; effects need fences

A lease grants authority for a bounded interval. It can let a current holder act without a fresh quorum round on every operation. Safety depends on how expiry is measured, the bound on clock uncertainty or renewal timing, and what happens during pauses and partitions.

Even a carefully designed lease should carry a monotonically increasing fencing token to downstream resources. Suppose worker 41 pauses beyond its lease, worker 42 obtains the next lease, and worker 41 resumes. If the storage layer accepts both, exclusivity is already lost. If it stores the greatest token seen and rejects 41 after seeing 42, the old worker is fenced.

Failure detection alone cannot grant safe exclusivity. “We stopped hearing heartbeats” means the old worker may be unreachable, slow, or partitioned—not that it cannot still write. The authority protocol and effect boundary must reject old epochs.

Concurrency control chooses when conflicts pay

Pessimistic control reserves before work

Locks or comparable reservations prevent conflicting operations from proceeding together. They fit high-conflict, expensive-to-retry work and invariants whose predicate can be locked precisely. Costs include wait, deadlock handling, lock memory, convoying, and vulnerability to slow holders.

Lock the invariant, not an incidental row. Locking reservation row A does not protect a capacity predicate spanning A and B. A counter or event-capacity row can provide the necessary common conflict point, but it can become hot. Predicate/range locks or serializable mechanisms may cover a broader condition at greater cost.

Keep lock acquisition order explicit, propagate deadlines, and never hold a database lock across an unbounded external call. If a client times out, cancellation must release or resolve ownership without leaving an in-doubt effect.

Optimistic control validates after speculative work

Optimistic concurrency control reads versions, computes, then validates that relevant state has not changed before commit. It excels when conflicts are rare and speculative work is cheap. Under contention, aborts multiply CPU, database reads, network calls, and tail latency.

Retry the entire logical transaction from a new snapshot, not the final statement in isolation. Reuse the logical idempotency identity, respect the remaining deadline, apply jitter or admission if conflicts synchronize, and record abort reasons. An optimistic protocol is not conflict-free; it chooses abort rather than wait.

Hot-record arithmetic identifies a shape limit

If one invariant authority holds a serialized critical section for (S) seconds, its service ceiling is approximately (1/S) before queueing and variance. The fixture uses 3 ms:

[ \mu = \frac{1}{0.003\ \text{s}} \approx 333.3\ \text{transactions/s} ]

At 220 transactions/s, modeled utilization is 0.66. Under an M/M/1 teaching approximation, mean time in the station is (S/(1-\rho) \approx 8.82) ms. At 900/s, utilization is 2.7; arrivals exceed service capacity and no steady queue exists.

The approximation is not a database prediction. It proves that “add more clients” cannot repair one 3 ms serialized authority at 900/s. Options include reduce hold time, batch compatible operations, partition the invariant, allocate escrow rights, reserve coarser inventory, admit fewer requests, or change the product contract. Sharding rows while preserving one global capacity bound merely moves coordination.

Atomic commit and consensus solve related but different problems

A distributed transaction may need all participants to commit or all to abort. In classic two-phase commit, a coordinator asks participants to prepare; each durable yes promises it can later commit. If all vote yes, the coordinator records commit and informs them. If the coordinator fails after participants prepare, those participants may remain in doubt until the decision is recovered.

The atomicity property is valuable, but availability and recovery cost are real. Locks and versions may remain held. Presumed-abort/commit variants, coordinator replication, timeouts, and consensus-backed logs change failure behavior. A participant cannot safely decide abort merely because it has not heard from a slow coordinator after voting yes.

Consensus chooses a value among replicas despite some failures. Atomic commit preserves one outcome across resource managers, each of which may vote no. The problems can be composed and optimized, but “use consensus” does not erase participant prepare state, and “two-phase commit” is not faultless consensus.

Keep external side effects outside an atomic database claim unless the provider participates in the protocol—which most do not. An outbox can atomically record intent with local state; a worker then delivers the external effect with idempotency and reconciliation. This yields a well-defined durable process, not a magical cross-provider transaction.

Sagas make process state explicit; compensation is a new action

A saga decomposes a long business process into local transactions with persistent progress and compensating actions. It can avoid holding distributed locks across minutes or external systems. It also exposes intermediate states and needs idempotent steps, ordering, retry ownership, timeouts, and operator reconciliation.

Compensation is not rollback. Releasing a reservation after sending a confirmation email cannot unsend the email. Refunding a captured payment is a second ledger event with fee, timing, and user consequences. Shipping a physical item may be irreversible. Classify each step:

Use a saga when temporary intermediate states are allowed and every exposed effect has an acceptable forward or compensating path. Do not use it to evade an invariant that must hold before acknowledgment.

Escrow rights can move the coordination boundary

The reservation bound can be decomposed when the system allocates nonduplicable rights. From 100 units, regions receive 40, 35, and 25 rights. They consume 38, 30, and 18 locally, leaving 2, 5, and 7. Total consumption is 86 and remaining rights are 14; the global bound holds without coordinating every local reservation because no region can spend rights it does not own.

Rights creation, transfer, revocation, loss recovery, and rebalancing still coordinate. A region with no rights may reject while another has spare capacity. The trade is per-operation global coordination for allocation skew and transfer complexity. Epochs and durable ownership prevent duplicated rights during failover.

Escrow fits divisible invariants. It does not solve one unique username or one seat that cannot be subdivided. It also does not help if a downstream side effect ignores the rights ledger.

Applied decision: reserve locally, reconcile the process globally

Ledgerline selects escrow for the high-rate capacity decrement, but not for the entire checkout. Each event’s authoritative capacity service creates signed, epoch-scoped rights. A regional reservation transaction consumes rights and creates the reservation record atomically under one local owner. It acknowledges only after both changes are durable. A region without sufficient rights rejects or requests a transfer; it never borrows optimistically from an unconfirmed global total.

The design keeps three other boundaries distinct. Reservation IDs use the Chapter 25 idempotency record, so a timed-out client can retrieve the same result. Rights transfers use a fenced authority epoch and cannot make the same rights spendable in both regions. Payment capture runs after the reservation commits through a durable outbox; the process state records capture_pending, captured, release_pending, or reconciliation_required. A provider timeout produces unknown outcome and an authoritative query, not an immediate new capture.

This design accepts stranded capacity: one region may reject while another holds unused rights. It pays transfer coordination less frequently than per-reservation coordination. Revisit it when transfer rate, rejection due to stranded rights, or skew becomes materially larger than the latency and availability saved. If one event receives 900 attempts/s and local consumption still serializes for 3 ms, escrow alone is insufficient; split rights among more independent owners, batch compatible reservations, reduce hold time, or shed attempts before the authority queue.

The rejected alternatives remain useful. One globally serializable transaction is simpler and may be correct at lower rates or when cross-region latency is acceptable. Snapshot isolation without a common conflict point is rejected because the exact write-skew history is legal. A saga that cancels four excess reservations is rejected because the product promised confirmed capacity. A mergeable counter is rejected for the same reason: convergence after oversell does not restore the acknowledgment invariant.

Mergeable state works only when the merge preserves meaning

Conflict-free replicated data types (CRDTs) define state or operations so replicas that receive the same updates converge deterministically under stated delivery assumptions. Sets, counters, registers, and sequences can have carefully defined merge semantics. This is powerful for disconnected and multi-writer operation.

Convergence is not arbitrary invariant preservation. A grow-only counter can merge concurrent increments, but a hard upper bound can still be exceeded. A last-writer-wins register converges by discarding one concurrent value according to its order rule; that may be unacceptable for orders or permissions. Deletion and membership have subtle observed-remove semantics.

Use mergeable state when concurrent operations commute or their deterministic resolution matches the domain, and when intermediate states are safe. Record metadata growth, tombstone/causal-context cleanup, privacy deletion, and recovery behavior. If correctness requires excluding a concurrent history, coordinate that invariant or redesign it with rights.

Session semantics survive routing only with evidence

Chapter 27 introduced a (partition, epoch, commit_position) token. The same idea applies across consistency levels. A read-your-writes request presents a minimum version. An eligible replica serves it, waits within the remaining deadline, forwards it, or rejects with a machine-actionable reason. A monotonic-read token advances after every response.

Sticky routing without a token is a placement preference. It fails when the replica restarts, the partition moves, or the user changes device. A token without retained history can also fail: the system may be unable to serve an old snapshot after garbage collection. Define token lifetime, privacy, size, signature, and fallback.

Session consistency does not preserve a global capacity invariant among clients. It improves each session’s view. Keep view guarantees distinct from transaction exclusion.

Operational evidence must expose the semantic path

Collect evidence at the invariant boundary:

• transaction attempts, commits, aborts, and retries by reason;
• lock wait/hold distributions and predicate or key scope;
• validation conflicts and wasted speculative work;
• quorum/consensus round and stable-log latency;
• lease age, renewal failures, term/epoch changes, and fenced writes;
• prepared/in-doubt transaction count and oldest age;
• saga state age, compensation attempts, and unreconciled external effects;
• session-token fallback/wait/failure rates;
• rights allocation, consumption, stranded capacity, and transfers; and
• invariant audit results derived independently of the write path.

Do not infer correctness from a low error rate. Run history-based tests for targeted anomalies, fault injection around prepare/commit and lease expiry, and reconciliation audits against authoritative external systems. Load tests must include conflict concentration, not uniform keys that avoid the hot invariant.

Rollout is a mixed-semantics interval. If an old client omits an idempotency key or session token, the server needs a compatibility rule. If one code version checks capacity and another does not, serializable storage cannot repair the absent predicate. Expand the schema and protocol, observe both paths, then retire the weak path with evidence.

Consistency selection worksheet

operation / user journey: _______________________________________
unit of work and success: _______________________________________

invariants:
  predicate: ____________________________________________________
  scope (object / transaction / session / process): _____________
  must hold at: __________________________________________________
  named violation/anomaly: ______________________________________
  user-visible consequence: _____________________________________

concurrency and state:
  operations that may race: _____________________________________
  authoritative state and owner: ________________________________
  read/write set or predicate: __________________________________
  hot-key rate and critical-section demand: _____________________

minimum mechanism:
  consistency/isolation history required: _______________________
  optimistic / pessimistic / rights / merge / saga: _____________
  consensus or atomic-commit role: ______________________________
  lease and fence boundary: _____________________________________
  session token and fallback: ___________________________________

failure and performance:
  required participants/failure domains: ________________________
  no-contention path and tail contributors: _____________________
  abort/wait/unknown/compensation behavior: ______________________
  overload and recovery policy: _________________________________

evidence:
  anomaly test: __________________________________________________
  contention/abort distributions: _______________________________
  invariant audit: ______________________________________________
  transfer limit and revisit trigger: ___________________________

Field questions

• Can every invariant be written as a predicate over named state or history?
• Does the proposed isolation level exclude the exact anomaly, in this implementation?
• Which rows or keys appear disjoint but participate in one predicate?
• Is a lease fenced at the resource that accepts the effect?
• Will optimistic retries still fit the original deadline and retry budget?
• Can the invariant be partitioned or represented as nonduplicable rights?
• Does the merge rule preserve business meaning, not merely convergence?
• Which saga effects are irreversible or only socially compensatable?
• Can an in-doubt transaction block recovery or schema change?
• Which test produces the forbidden history deliberately?

Anomaly drill: preserve eight without inventing global serialization

Start with the two six-unit reservations against eight units. Produce three valid designs:

1. one conditional capacity mutation at the event owner;
2. a serializable transaction whose read/write predicate covers all active reservations; and
3. escrow rights split across two regions.

For each, state the acknowledgment point, failure behavior, p99 latency contributors, overload policy, retry identity, and audit. Then add two adversarial facts: traffic rises to 900 attempts/s while the serialized section remains 3 ms, and payment capture cannot participate in the database transaction. Decide whether to batch, repartition rights, admit less, or alter the product. Place payment in a durable process with an explicit unknown/compensation path.

Find the bad answer: “use a CRDT counter and cancel excess reservations later.” It converges, but it changes the invariant from never acknowledge excess capacity to temporarily oversell and choose losers. That can be a business choice only when named and accepted.

Durable decision rules

1. Write invariants and forbidden histories before choosing a consistency label.
2. Match the guarantee boundary: object linearizability, transaction serializability, session order, and business-process state are not interchangeable.
3. Coordinate only operations that can jointly violate the invariant; partition authority or allocate rights when the proof permits it.
4. Treat optimistic aborts, pessimistic waits, atomic-commit uncertainty, and compensation as workload with deadlines and ownership.
5. Fence leases and leadership generations at the downstream effect boundary.
6. Use mergeable state only when its merge semantics preserve the domain, not merely replica convergence.
7. Test the forbidden history under skew, failure, recovery, and mixed versions; monitor the invariant independently of the implementation path.

The next design boundary is the interface. Chapter 29 turns these decisions into API and data contracts so clients cannot accidentally request unbounded work, lose completion identity, or assume stronger semantics than the service provides.

Evidence and transfer limits

• Herlihy and Wing, “Linearizability: A Correctness Condition for Concurrent Objects”, defines the real-time object-history condition. It should not be stretched into a claim about multi-object transactions without a transaction model.
• Berenson et al., “A Critique of ANSI SQL Isolation Levels”, defines snapshot isolation and exposes limitations of phenomenon-based isolation labels. PostgreSQL 18 transaction-isolation documentation provides current implementation-specific behavior and retry guidance, including serialization anomalies under its repeatable-read/snapshot mode.
• Ongaro and Ousterhout, the Raft paper, is primary protocol evidence for replicated logs, leaders, terms, quorums, and commit. Consensus does not by itself define the application’s distributed transaction.
• Gray and Lamport, “Consensus on Transaction Commit”, analyzes the relationship between classic two-phase commit and Paxos-style commit rather than treating them as synonyms.
• Terry et al., “Session Guarantees for Weakly Consistent Replicated Data”, introduces read-your-writes, monotonic reads, monotonic writes, and writes-follow-reads as scoped client guarantees.
• Bailis et al., “Coordination Avoidance in Database Systems”, formalizes invariant confluence: whether a given invariant and operation set can execute without coordination. Applying the framework still requires a correct application model.
• Shapiro et al., “Conflict-Free Replicated Data Types”, states convergence conditions for state- and operation-based replicated types. Convergence is not evidence for arbitrary cross-object invariants.
• Google Spanner’s external-consistency documentation describes one implementation’s transaction ordering and timestamp mechanism. Its clock infrastructure and database behavior are not generic properties of “distributed SQL.”
• The executable fixture in examples/performance-engineering-system-design-handbook/part-03/consistency-coordination/ reproduces the oversell by four, 34 ms modeled coordination path, 333.3 transactions/s serialized ceiling, 0.66 versus 2.7 utilization cases, and 100-right escrow conservation. These are teaching calculations, not observed database performance or a proof for a production reservation service.