Distributed Systems Fundamentals

The timeout after the last item

An online store has one camera left. A shopper in Nairobi presses Reserve. The inventory service accepts command c42, but the response does not arrive. The button stops spinning and offers Try again.

At that moment, several histories fit what the shopper can see. The request may have disappeared before reaching the service. It may still be waiting in a queue. It may have reserved the camera and lost only the response. If the client repeats the request, the system must act safely without knowing which history occurred.

This uncertainty is the beginning of distributed-systems reasoning. A timeout does not report what happened elsewhere; it reports what one observer failed to learn in time. More machines, regions, and replicas create more observers with incomplete views. Correctness comes from deciding which of them may act, what order their actions have, what a successful response proves, and how the system recovers when those views disagree.

The reservation gives us one operation to follow. Its invariant is exact: confirmed reservations for a SKU must never exceed stock. Browse counts may lag. Shopping carts may merge after reconnecting. The final reservation may not invent a second camera.

Give the invariant one place to become true

Suppose both the Nairobi and London regions read available = 1, decrement locally, and replicate afterward. Each region can make a valid decision from its own stale state. Together they sell the same item twice. Replication will eventually reveal the negative inventory, but convergence cannot withdraw a promise already made to a customer.

The reservation needs an owner: the component allowed to put mutations for this SKU into an authoritative order. A practical choice is to partition inventory by sku_id and route every reservation command for one SKU to the same logical owner.

owner key: sku_id = camera-17
command key: c42

state before:  available=1, reserved=0, version=901
decision:      reserve 1 when available >= 1
state after:   available=0, reserved=1, version=902
result:        reservation r88 belongs to command c42

The command key is part of the state transition. A repeat of c42 returns reservation r88; it does not decrement again. Reusing c42 with different arguments is rejected. The server retains that association for at least as long as the client, gateway, or operator may retry the command.

Ownership also gives ordering a useful scope. Versions 901 and 902 order mutations for camera-17; they need not order a profile edit, another SKU, or every sale in the company. Per-key order allows unrelated inventory to move in parallel and prevents a global sequencer from becoming a needless latency and availability boundary.

This is why a timestamp alone is not enough. Physical clocks are valuable for logs, expiry, monitoring, and user-visible time, but clocks on two machines can drift and messages can arrive late. The authority that owns a decision must establish its logical order through a version, log position, or term. “Timestamp A is smaller” does not prove that independent write A caused write B or that every observer saw them in that order.

Some products cannot assign one owner. A collaborative document may accept concurrent edits and merge them; an offline profile editor may use versions and an explicit conflict rule. That is still a choice about the invariant. Where conflicting decisions are harmless or repairable, coordination can be avoided. Where they spend money, grant access, or promise scarce inventory, the system needs one authoritative decision or a protocol that makes several participants act as one.

Replication changes the meaning of “reserved”

One owner can still lose its disk or machine. The service therefore stores its ordered log on three replicas. The owner proposes version 902, waits for two replicas to persist it, and only then may report that the reservation is durable.

N = 3 replicas
W = 2 acknowledgements required for write success

replica A: version 902  acknowledged
replica B: version 902  acknowledged
replica C: version 901  delayed

The third replica is not decorative. It must catch up before it serves a read that promises recency or before it is allowed to become an authoritative owner. Replication creates both a failure tolerance and a read contract.

A read from replica C can legitimately return available = 1 if the product permits stale browse data. It cannot safely authorize another reservation. After a successful reservation, the shopper may also expect a refresh to show their own result. The system can satisfy that with a leader read, a quorum read, a session token carrying version 902, or by routing the client until a replica has caught up. The mechanism matters less than naming the promise: linearizable reads, read-your-writes, bounded staleness, and eventual convergence are different user experiences.

Quorum arithmetic helps explain the overlap. With N = 3, W = 2, and a read that consults R = 2 replicas, R + W > N; the read set must intersect the successful write set. The coordinator still has work to do. It must compare authoritative versions, reject conflicting histories, and repair or exclude stale replicas. The overlap does not by itself create a linearizable system, settle a read racing with a write, or make an arbitrary “latest timestamp wins” rule safe.

The word quorum should therefore lead to concrete questions. Which nodes count? What does an acknowledgement mean? How is the newest valid version recognized? Can a replacement node join before catching up? What happens to the replica that did not acknowledge? Numbers without those rules describe a vote, not a correctness argument.

A partition turns architecture into product behavior

Now the Nairobi owner cannot communicate with two replicas. Waiting longer may not reveal whether they crashed, the network dropped their packets, or the owner itself is isolated. This is partial failure: some components remain healthy enough to act, but no observer can establish a complete picture.

The isolated owner has only one copy of the state, so it cannot reach the write quorum. For the reservation operation, it stops accepting new work. It might return Unable to confirm inventory, or record an explicitly pending request that will be decided only after quorum service returns. It must not show Reserved merely because accepting locally feels more available.

Other operations can continue. The store can serve cached product pages, accept cart edits, and record analytics locally because temporary disagreement there does not violate the scarce-stock invariant. Availability is chosen per operation and per failure, not stamped on the whole application.

This is the useful reading of CAP. While a network partition separates nodes, an operation that requires one linearizable answer cannot also guarantee a successful response from every reachable side. The design must withhold some answers or permit divergent ones. “Choose consistency or availability” is too coarse until it says which operation, which invariant, what response the user receives, and how later reconciliation works.

Sometimes accepting on both sides is the correct product decision. Reaction counts can merge; telemetry can upload later; comments can receive a stable display order after replication. Conflict then becomes part of the data model. The system needs a merge, rejection, compensation, or user-visible resolution rule. “Eventually consistent” describes neither the rule nor the harm while convergence is pending.

Election is not protection from an old leader

The two connected replicas elect B as the new owner. B has the majority log, including command c42 and version 902, and begins serving with term 18. The client retries c42; B replays the stored result and returns reservation r88.

Meanwhile, A may still be running. It did not receive a reliable message saying that it lost leadership; it only lost contact. A process pause can make this worse: A freezes while its lease expires, then resumes with its old state and continues as if no time passed.

Leader election chooses a new leader. It does not reach backward in time and disable the old one. Every effect that relies on exclusive ownership needs a way to reject stale authority.

The usual protection is a monotonically increasing term or fencing token. B’s writes carry term 18. Storage, job executors, and any other protected resource remember the highest accepted term and reject A’s later writes from term 17.

write(reservation=r88, term=18) -> accepted
write(reservation=r91, term=17) -> rejected as stale leader

A lease can limit how long an owner believes it is valid, but safe leases depend on timing assumptions and do not fence a paused process from a downstream system by themselves. The downstream check is what turns a new term into protection. External effects deserve special care: a payment provider or email system that cannot validate the fence needs its own idempotency record, intermediary, or reconciliation boundary.

The failover is now a trace rather than a claim:

1. A accepts c42 in term 17 and replicates it to a majority.
2. The response is lost, so the client does not know that c42 committed.
3. A becomes isolated; the majority elects B in term 18.
4. B recovers the committed log and recognizes the retry of c42.
5. B returns the existing result instead of reserving again.
6. Any late write from A carries term 17 and is fenced out.

Command identity handles the repeated request. The replicated log preserves the committed decision. The election restores an owner. Fencing prevents the previous owner from acting. None of the four is a synonym for the others.

Consensus belongs around the decisions that must not fork

The election and replicated log need nodes to agree on terms, membership, and an ordered sequence of committed entries. That is consensus territory. Protocols such as Raft and Paxos are ways for a group to preserve one agreed history despite crashes and delayed messages, provided enough members can communicate and the protocol’s assumptions hold.

Consensus protects safety; it does not abolish partial failure. A three-member group can keep making progress after one member fails, but not when no side can form a majority. It cannot make a remote payment atomic merely because a local log entry committed. It cannot promise one physical delivery of a message. Those boundaries still require transaction design, idempotency, or repair.

Agreement is expensive enough that its scope should remain small. Cluster membership, leader terms, shard assignments, schema versions, lock ownership, and the ordered decisions for a scarce resource are compact coordination targets. Routing every large payload, derived view, and approximate counter through one consensus group increases latency and blast radius without protecting a corresponding invariant. Often consensus assigns a partition to one owner, then that owner processes many local decisions without a cluster-wide vote for each unrelated key.

Distributed locks illustrate the same boundary. A lock record can say who currently owns a resource, but clients that act on the lock need a fencing token. Otherwise an expired client may wake up and perform work after a new client acquired the same lock. Mutual exclusion in the coordination service is useful only if the protected resource can distinguish current authority from stale authority.

Cross-owner work needs an honest boundary

After inventory reservation, checkout may need to authorize payment and create fulfillment work. These systems do not share one failure-free commit. If payment succeeds and the reservation later expires, or inventory commits and payment fails, the workflow can occupy an intermediate state.

A robust design makes that state durable and visible. It might reserve stock for a bounded period, authorize payment with an idempotency key, confirm the order after both facts are present, and release or compensate when the process cannot finish. An audit record ties every attempt to the reservation and payment references.

The aim is not to make every service pretend to be one database. It is to choose which transition owns the customer promise and to make every partial outcome recoverable. A status such as Payment received; confirming stock is an honest contract. A generic success screen backed by two unrelated best-effort writes is not.

Repair is part of the write path

Replica C still holds version 901. A payment provider may have accepted an authorization whose response was lost. A reservation could remain pending after its worker crashed. Correctness therefore continues after the online request.

The system needs mechanisms that find and resolve disagreement: log replay to bring a replica forward, read repair or anti-entropy between copies, reconciliation between reservations and payments, expiry for abandoned holds, compensation for a workflow that cannot complete, and an audit trail that records command ID, version, term, and outcome.

These mechanisms need observable age and ownership. Replica lag, rejected stale-term writes, duplicate command attempts, unresolved payment references, expired reservations, and reconciliation backlog reveal different broken boundaries. One generic “cluster healthy” light cannot say whether the store has made conflicting promises.

Repair policy also determines what history the system is allowed to erase. Automatically copying the value with the newest wall-clock timestamp can hide a concurrent write rather than resolve it. Repair must use the same ownership, version, and conflict rules as the online path, or route ambiguity to a person or compensating workflow.

Follow the uncertainty through one operation

In a system-design discussion, distributed-systems vocabulary is useful only when it explains a consequence. Start with the operation most capable of corrupting money, inventory, access, or legal state. State its invariant and smallest ownership key. Then follow one command through its authoritative order, acknowledgement rule, client timeout, partition behavior, retry, failover term, and repair path.

Pressure-test each success word. Does accepted mean queued, locally written, committed to a quorum, visible to recency-sensitive reads, or complete across an external provider? Ask which observer can still hold an older view and whether that view is allowed to make a decision.

The camera reservation can now survive an unpleasant history: its response is lost, its first leader becomes isolated, the client retries through a new leader, and a stale replica later returns. The invariant holds because one owner orders the decision, a majority preserves it, the command identity absorbs the retry, the new term fences the former leader, and repair restores the lagging copy. The design does not eliminate uncertainty. It confines the decisions that uncertainty is allowed to change.

The next chapter follows one of the deliberately weaker paths: a search index that may lag its source of truth. The same questions remain, but the product contract changes. Search can often serve a stale document and repair it later; the inventory owner could not safely sell a stale camera.