Consistency, Idempotency, and Transactions

A timeout has divided checkout into two realities

Return to the ticket sale from the previous chapter. One section owner has accepted a hold on seat 12. The buyer presses Pay. The server creates order ord_742, asks the payment provider to charge the card, and then the mobile connection disappears.

The buyer sees a spinner and a retry button. The payment provider may already have accepted the charge. The order service may or may not have recorded that fact. A replica may still say that no order exists. No component has necessarily misbehaved; the system simply cannot compress those different observations into one instantaneous event.

What should happen when the buyer presses the button again?

“Use strong consistency” is too broad to answer. The design needs several more precise promises:

• seat 12 has at most one buyer;
• one checkout attempt creates at most one logical charge;
• after the service acknowledges an order, that buyer does not read an older order state;
• a paid order eventually issues a ticket or reaches a visible repair state;
• every movement of money remains reconstructible after retries and corrections.

Each promise has an owner and a failure boundary. Consistency design is the work of finding those boundaries, then making repetition and uncertainty ordinary parts of the state machine.

Begin with the state that must survive repetition

Give the checkout attempt an identity before doing anything that may repeat. The client creates an idempotency key, checkout_8f31, and sends it with the order parameters. The service scopes the key to the buyer or tenant so another caller cannot discover or collide with the result.

For this one-seat order, keep the initial order aggregate with the section owner that holds the seat. The first local transaction can then make a small set of facts true together:

transaction on the section/order owner:
  verify seat 12 is held by this buyer and the hold has not expired
  insert order ord_742 in pending_payment
  insert idempotency record (buyer_19, checkout_8f31)
      with request fingerprint, order ID, and current response
  append outbox event payment_requested(ord_742, checkout_8f31)
commit

A uniqueness constraint on (buyer_id, idempotency_key) selects one winner if two requests arrive concurrently. The seat owner also needs a conditional write or uniqueness rule that prevents a second order from claiming the same hold. These database constraints matter more than a hopeful if not exists check in application code: two processes can both pass that check before either writes.

The transaction deliberately stops before the payment call. Its job is to protect facts owned by this order aggregate. It does not send email, update search, publish analytics, or ask an outside provider to share the database commit. Putting a network call inside the transaction would hold locks while waiting and still leave an ambiguous outcome if the provider committed just before the process died.

The outbox row is the durable handoff. A publisher can crash after the commit, restart, and still discover that payment is required. Publishing the event twice is acceptable because the next boundary is designed for repetition too.

An idempotency key is a concurrency protocol

When the mobile client retries checkout_8f31, the service looks up the stored attempt. If the request fingerprint matches, it returns ord_742 and the latest known status. It does not create a second order or call the provider merely because the original HTTP response was lost.

If the same key arrives with a different seat, price, currency, or basket, the service rejects the request as a conflict. Silently treating different parameters as the same operation would replay a result the caller did not ask for. The fingerprint therefore covers the fields whose change would alter the meaning of the operation, using a canonical representation rather than raw JSON byte order.

A useful idempotency record carries more than a cached response:

• caller scope and key;
• request fingerprint;
• stable resource identity, such as the order or charge ID;
• a state such as started, pending, committed, failed, or unknown;
• the response or status that later callers may safely observe;
• creation and expiry times matched to the business retry window.

The expiry policy is part of correctness. Deleting checkout keys after five minutes is unsafe if clients or support tools may legitimately retry yesterday’s attempt. Keeping every key forever may be unnecessary. Durable natural identities are often better for long-lived inputs: a payment callback can be deduplicated by provider event ID, and a ledger posting by its source event ID.

Idempotency also has a liveness decision. A second request may find the first in started while its worker is dead. It can return pending, take over work under a lease, or ask reconciliation to determine the external outcome. It must not assume that “still running” means “safe to execute again.” A lease needs an expiry and, where stale workers can still write, a fencing token or version check.

This is why attaching a random header to an API does not make the operation idempotent. The key has to participate in the same atomic decision as the state it protects.

Reads need a promise of their own

Suppose the order transaction commits at version 418, but the buyer’s next request reaches a replica that has applied only through version 412. The write was atomic and the retry path is safe, yet the status page says “order not found.” From the buyer’s perspective, checkout is still broken.

For this path, the write response can carry ord_742 and a version token. The next status read either goes to the authoritative owner or waits until its replica has applied at least version 418. This is read-after-write for the buyer. Public event pages do not need the same promise; they may show a slightly stale seat map as long as reservation returns to the seat owner before accepting a hold.

Monotonic reads protect a related expectation. Once the buyer has seen order version 421, later requests in that session should not show version 419. The service can route the session to a sufficiently current replica, include the last-seen version in requests, or fall back to the owner when a replica is behind.

Neither guarantee requires every reader to use the primary forever. The design should name who needs freshness, which operation establishes the required version, how long the service will wait, and what it shows if the version is not available. A truthful pending status is often safer than a fast stale answer that prompts another purchase.

Follow the payment across the boundary

After the local commit, an outbox worker sends the payment request using a stable provider idempotency key derived from checkout_8f31, not a fresh key on each delivery. The provider may return success, decline, or time out. A timeout is an observation about the response path, not evidence that no charge occurred.

Model that uncertainty directly:

pending_payment
  -> paid              provider success or verified callback
  -> declined          authoritative decline
  -> payment_unknown   timeout with no authoritative outcome

payment_unknown
  -> paid              reconciliation finds a successful charge
  -> declined          provider confirms no charge
  -> manual_review     automatic resolution budget is exhausted

Provider callbacks may repeat or arrive out of order. Store each provider event under its natural ID and apply it in a transaction with the legal order-state transition. If the event has already been applied, acknowledge it without changing the order. If a late event proposes an impossible transition, retain it for audit and reconciliation rather than forcing the state machine forward.

The same discipline applies downstream. The ticket-issuing consumer records a stable issuance identity before producing the durable ticket. Notification and analytics consumers deduplicate their own effects. Reconciliation compares orders, provider charges, tickets, and outbox progress to find facts that failed to cross a boundary.

No single database transaction covers this whole history. Correctness comes from a chain of smaller atomic decisions joined by durable identities and repairable state transitions.

“Exactly once” hides the paths you need to design

A broker may offer a narrow exactly-once processing mode. A database may commit one transaction exactly once from its own perspective. Neither can prevent a mobile client from resubmitting, a provider from repeating a callback, an operator from replaying a dead-letter queue, or a backfill from encountering an old event.

The useful top-level promise is therefore one logical effect, not one physical execution:

One checkout attempt resolves to one order identity.
One seat ownership rule admits at most one buyer.
One provider charge identity is applied at most once to the order.
Repeated messages converge on the same legal order state.

This wording does not weaken the product promise. It makes the mechanisms that defend it inspectable. If an interviewer asks for exactly-once checkout, locate the duplicates instead of debating the slogan: client request, outbox delivery, provider request, callback, ticket command, and any later replay. Give each path a stable identity, an atomic deduplication point, and a recovery rule.

Compensation records a new fact; it does not erase the old one

Now let payment succeed after the seat hold has expired. The system cannot roll back the provider’s history or pretend the charge never happened. It needs an explicit policy.

If the seat is still available, the order owner may reacquire it under a new conditional decision and continue. If another buyer owns it, the order moves to refund_required or manual_repair. A refund worker submits a stable refund identity, follows unknown outcomes through reconciliation, and records the final result. The buyer sees what is pending and what has completed.

That is a compensating action: a new state transition that offsets a committed effect. Other examples include releasing an expired inventory allocation, posting a correcting credit, revoking an issued entitlement, or creating a return after duplicate fulfillment. Compensation works only if the original facts remain available. Mutable history and vague “rollback” language make it impossible to know which action is safe.

Some effects cannot truly be compensated. An email cannot be unsent, disclosed data cannot be made unseen, and a scarce opportunity may have passed. The right design may prevent those effects until uncertainty is resolved, require manual review, or accept and name the business loss. A saga is not atomicity spread across services; it is a workflow whose partial states and compensations are part of the product.

A ledger preserves correctness over time

Order status answers what should happen next. A ledger answers what value moved and why. Treating the ledger as a mutable balance row loses the evidence needed to resolve duplicate callbacks, refunds, backfills, and disputes.

When a verified payment is recognized, the ledger transaction should append the domain’s balanced entries, enforce a unique source identity, and update any materialized balance together. The exact account names belong to the accounting model—for example, provider receivable and ticket-sales clearing—but the engineering properties are stable:

• entries are append-only;
• every posting has a unique business or source-event identity;
• all sides of the posting commit together;
• balances can be recomputed from entries;
• corrections and refunds use new reversing entries rather than rewriting history;
• actor, reason, timestamps, and source references survive for audit.

If the same provider callback is applied twice, its source identity makes the second posting a no-op or conflict, not another movement of money. If a balance is wrong after a backfill, the entry history lets reconciliation identify the missing, duplicated, or misclassified posting. Derived balances and statements may lag, but any freshness promise used for spending or customer trust must be explicit.

Inventory counts, credits, rewards, usage, and quotas often need the same historical discipline even when they are not financial ledgers. The test is not whether a field is called balance; it is whether later correction and audit must explain how the current value came to be.

Make the interview answer follow causality

A clear answer can stay with the checkout trace:

The invariant is one owner for the seat and one logical result for a checkout
attempt. I would create the order, idempotency result, and outbox event in one
local transaction. The buyer's status read gets read-after-write using the
commit version, and later reads cannot go backward. Payment happens outside
that transaction under the same stable attempt identity. Timeouts create an
unknown state; callbacks and reconciliation resolve it. Every consumer dedupes
at its own state boundary. If payment succeeds after inventory is lost, the
order enters an explicit refund or repair path. Money movements are appended
to a balanced ledger under unique source IDs, never repaired by rewriting
history.

This gives the interviewer places to apply pressure. If orders and seats have different shard owners, ask whether the product can require a pre-existing hold or needs coordination. If status reads must be global, compare a home-region owner with version-aware replicas. If idempotency records expire, ask how old provider callbacks remain safe. If reconciliation is delayed, decide what the buyer sees and which actions are blocked meanwhile.

Distributed transactions may be appropriate inside a controlled platform when participants and failure semantics justify their cost. Do not reject them by reflex. Compare them with the actual invariant, availability requirement, latency, operational support, and recovery model. For an outside card provider, however, a local transaction plus durable workflow is the honest boundary.

Practice the unresolved outcome

Design the same sequence for a wallet transfer. A client times out after asking to move 40 credits from account A to account B, then retries from another device. Before choosing a mechanism, write down:

1. the semantic identity of the transfer;
2. the owner of the atomic debit-and-credit decision;
3. the response to two concurrent requests with the same identity;
4. the read guarantee for both account views;
5. the entry or entries that allow balances to be rebuilt;
6. the state shown while the outcome is unknown.

Now make A and B belong to different shard owners. Do not copy the first design unchanged. Decide whether transfers deserve coordination, whether a dedicated transfer owner can reserve and settle in stages, or whether product constraints can keep the movement local. Name the partial states and what reconciliation proves before money becomes spendable.

For a final variation, deliver the credit event twice and then run a backfill that delivers it once more. If the answer depends on remembering which worker ran it, the logical identity is in the wrong place.

Field reference

PROMISE
  State the invariant without the word “consistent.”

OWNER
  Put each atomic decision under one store, aggregate, shard, or coordinator.

REPEAT
  Give the logical operation a scoped identity, fingerprint, uniqueness rule,
  stable response, and expiry policy. Expect concurrent retries and replays.

READ
  Name who needs read-after-write or monotonic reads, the required version,
  the wait or routing rule, and the truthful fallback when data is behind.

HANDOFF
  Commit owned state with an outbox or durable log. Make every receiving
  boundary deduplicate and apply a legal state transition.

UNKNOWN
  Preserve uncertain outcomes. Reconcile before declaring success or failure.

REPAIR
  Record compensating actions and reversing entries as new facts. Keep audit
  history and a manual path for effects automation cannot resolve.

The design is credible when every repeated signal leads back to one logical result and every uncertain outcome has somewhere honest to wait. Once that is true, the next pressure is operational: how long to wait, how much to retry, what to shed, and how far a failed dependency may spread. Reliability and Failure Engineering takes up those limits. Use Scaling and Partitioning for the ownership boundaries and Transactions and Isolation for database anomaly and isolation vocabulary.