Skip to content

Senior Engineering Interview Handbook / Chapter 93

Commerce and Financial Integrity

A sustained commerce-system design case that develops idempotency, inventory reservation, provider ambiguity, order state, double-entry posting, reconciliation, operational repair, and adaptations for carts, payments, ledgers, and fulfillment.

The checkout the provider may have authorized

The store has one camera left. Lena checks out, the inventory service holds the unit, and the payment provider does not answer before the request times out. The provider may have authorized the payment. Lena’s browser retries. While the system waits for a webhook, the inventory hold approaches its expiry.

What should the confirmation page say?

That question reaches further into the architecture than a diagram of services does. If the retry starts a second checkout, it may create a second authorization and later a second capture. If the timeout is treated as a decline, the first authorization may be forgotten. If the order is confirmed after the reservation expires, the store has promised an item it cannot ship. If operators later repair the incident by editing payment or ledger rows, the system may look correct while losing the evidence needed to explain the money.

A commerce system is a collection of promises with different strength. The cart records intent. A reservation makes a temporary claim on stock. Authorization records a provider’s willingness to fund a payment. The order is the business’s commitment to the customer. Capture moves money. A ledger records the financial effect under the business’s accounting rules. Reconciliation finds the cases in which those facts disagree.

For this case, assume scarce physical inventory, authorization during checkout, capture when fulfillment begins, and partial refunds after shipment. The store must not confirm more units than it can allocate, create two orders from one checkout attempt, or turn one intended payment into two provider operations. Those assumptions are choices, not universal commerce law. A digital download, a back-order business, a subscription, and a marketplace would move the binding points.

Separate the facts before joining the services

An overloaded order.status cannot honestly represent the incident. At the moment of the timeout, several facts coexist:

  • the cart still describes what Lena intended to buy;
  • the reservation may be held, allocated, released, or expired;
  • the provider request has an outcome that is currently unknown to this system;
  • the checkout attempt is incomplete but must retain its identity across retries;
  • no customer-facing order should be called confirmed until its stated inventory and payment conditions have been met.

Write those owners and transitions before drawing the architecture. A compact state sketch is enough:

checkout_attempt: started -> waiting_for_payment -> ready_to_commit
                  started -> rejected
                  waiting_for_payment -> needs_review

reservation:      requested -> held -> allocated
                  held -> released | expired

payment_intent:   created -> authorization_pending -> authorized -> captured
                  authorization_pending -> failed
                  authorized -> voided
                  captured -> partially_refunded | refunded | disputed

order:            pending -> confirmed -> fulfilling -> fulfilled
                  pending -> rejected
                  confirmed -> cancel_requested -> canceled

The transitions need guards, not merely names. Allocation and expiry compete on the same held reservation; a conditional transition lets only one win. One checkout attempt can acquire at most one committed order. A capture cannot exceed the authorized amount or the rules for partial capture. A refund cannot quietly rewrite a capture. It creates its own payment transition and financial posting.

This is also the right moment to clarify the prompt. Ask what is scarce, whether overselling or back-ordering is allowed, when price and tax become binding, when authorization and capture occur, what “confirmed” promises to the customer, and which external evidence arrives later from a provider, bank, marketplace, or warehouse. These answers alter the state machine. Raw QPS usually does not.

Give the attempt one durable identity

Lena’s browser sends checkout_key=ck_72 with the cart version it reviewed. The checkout service creates one durable attempt for that key and records a fingerprint of the request. A retry with the same key and same request returns the recorded attempt or its completed result. The same key with a different cart, amount, or currency is rejected instead of being mistaken for a duplicate.

The identity must survive every side effect. The inventory request can be keyed by the checkout id and line id. The payment service creates one internal payment intent and uses a stable provider idempotency key for the authorization. Order creation has a unique guard on the checkout id. Ledger journals are unique by source event and posting rule. Provider webhooks are deduplicated by provider event id while still updating the known state of the payment intent.

Idempotency therefore means more than retrying an endpoint:

  1. define the scope and lifetime of a key;
  2. bind it to the intended operation;
  3. persist the outcome or current attempt before acknowledging success;
  4. return a compatible result to duplicates;
  5. reject key reuse that describes a different operation.

A process-local cache cannot provide this guarantee after a crash. Nor can a new key safely resolve an ambiguous provider call: it may create a second charge rather than discover the first one’s result.

Reserve the last unit without freezing the store

The cart does not decrement inventory. At checkout, the inventory owner creates a time-bounded reservation for the camera. For a scarce SKU, it must serialize or conditionally guard the change that moves a unit from sellable to reserved. That can be a versioned database update, a transaction on the SKU-and-location row, or a single writer for that inventory partition. The choice follows the contention and availability needs; the invariant is the same.

The reservation has an owner, quantity, location, state, expiry, and checkout reference. An expiry worker does not simply add one to an inventory counter. It conditionally changes held to expired; if checkout already changed held to allocated, expiry loses the race. Cancellation and payment failure use the same guarded release path. Reconciliation later compares these internal claims with warehouse adjustments and physical stock.

A launch with a million shoppers and five hot SKUs is not primarily a million-key storage problem. It is contention on five ownership boundaries. Queueing or single-writer partitions can keep those counters honest, but they introduce wait time and regional availability questions. The business may instead allow back-orders, allocate safety stock, or reject aggressively. A senior design says which policy it chose and what the customer sees when capacity is exhausted.

Lena’s hold now exists until 10:05. The checkout can proceed, but “held” is still not “sold.”

Cross the payment boundary without guessing

The payment service sends the authorization with the stable intent reference. At 10:02, the HTTP request times out. This is an ambiguous result, not a decline. The service records authorization_pending, retains the provider request reference, and returns that state to checkout. Lena sees “We are confirming your order,” not an invented success or failure.

The browser retry with ck_72 attaches to the same checkout attempt. The payment service may query the provider by its stable reference, accept a signed webhook, or discover the authorization in later provider records. It does not issue a fresh authorization under a new identity while the first is unresolved. Provider failover has the same danger: sending the payment to a second provider is a business recovery decision, not a harmless network retry.

Suppose the webhook arrives at 10:04 and confirms authorization. Checkout now rechecks the reservation and the commercial snapshot. If the hold is still valid, the guarded transition allocates it and order creation can commit the business promise. If the hold expired at 10:03, the system attempts a new reservation under an explicit policy. If the camera is gone, it does not confirm the order; it voids the authorization when possible and exposes any unresolved reversal as work. The user gets an honest stockout message rather than a fictional order.

This sequence is a saga in the plain sense: several owners commit local state, and later steps compensate when the whole business action cannot finish. Calling it a saga does not decide orchestration versus choreography. A checkout orchestrator is useful here because it can make the current step and user-visible state easy to inspect. Events remain valuable after committed transitions for fulfillment, notifications, analytics, and repair. Neither pattern removes the need for local transactions, durable intent, or idempotent consumers.

Commit the order, then publish what happened

When the reservation and authorization rules pass, the order service creates one order for ck_72. It snapshots the commercial terms that must not drift: item, quantity, price, discount, tax basis, shipping promise, currency, and the references to reservation and payment. Future catalog or promotion changes do not recompute this historical agreement.

The transaction also records an outbox event. A worker can publish OrderConfirmed repeatedly until the broker accepts it; consumers deduplicate by event identity. The order remains real even if publishing is delayed, and the outbox provides a repairable record of what still needs to leave the service. Treating an event as the only copy of a transition would make a broker failure capable of erasing the business fact.

Downstream work has different consequences. A delayed email can be retried. A warehouse allocation must retain the order’s identity. Capture must reuse the payment intent and obey its amount rules. If the store captures on shipment, an order can be confirmed while its payment remains authorized. The UI and support tools should show those states separately instead of compressing them into “payment successful.”

The ledger records movements, not guesses

Authorization is provider state; it may not yet be a financial posting under the store’s accounting policy. When capture or another money-moving event occurs, the payment service records durable evidence and creates a ledger-posting obligation. The ledger applies a versioned posting rule and writes a balanced journal atomically.

An illustrative capture journal might look like this:

journal: capture/payment_intent_481/posting_rule_v3
currency: USD
debit:   processor_receivable       120.00
credit:  customer_payment_clearing  120.00

The exact accounts belong to the business’s accounting policy. The system property is that debits and credits balance within the journal and currency, the source reference makes the posting idempotent, and posted entries are immutable. A correction, chargeback, or refund creates a linked journal under the appropriate rule; it does not edit the capture out of history. Balance and statement views can be rebuilt from journals and cached for reads.

“The ledger is the source of truth” is too broad. It is authoritative for the financial facts it records. It does not own whether the warehouse still has a camera, whether the provider has settled funds, or whether the customer was shown a confirmation page. Those systems retain their own evidence. Integrity comes from explicit ownership plus reconciliation, not from declaring one database sovereign over every domain.

Ledger partitioning deserves care. Ordering may be required for one account or posting stream, while reports span many accounts. Partitioning by account can scale writes but complicates a journal that touches accounts on different partitions; partitioning by journal simplifies atomic posting but moves some ordering or balance work to derived views. State the required ordering and atomicity before choosing the key. Financial retention, access, and reporting requirements also belong in the prompt; do not invent universal periods or compliance rules.

Reconciliation begins where the request ends

By 10:05, Lena has a confirmed order and a valid authorization. That resolves the interactive request, not the integrity problem. The provider may later report a different status. Capture may succeed while a webhook is lost. A settlement file may contain an amount the internal ledger lacks. The warehouse may reject the allocation. Distributed commerce systems require a second route to truth.

Reconciliation ingests independent evidence: provider queries and settlement files, internal payment events, order transitions, ledger journals, warehouse allocations, refunds, disputes, and approved support actions. It first matches stable references, then uses constrained secondary evidence such as amount, currency, time, and customer or merchant reference. A fuzzy match can propose a case; it should not silently move money.

Each mismatch becomes owned work with type, age, financial or inventory exposure, evidence, and allowed actions. An automated repair may replay a missing ledger posting or outbox event. A risky amount mismatch may require review. The eventual action still goes through the domain API: post a reversal, issue a refund, release a hold, or reopen fulfillment. Support staff should not repair one dashboard by directly mutating the rows behind another.

This exception queue is part of the product. Measure how many captures lack journals, authorizations lack orders, refunds exceed eligible amounts, reservations remain held past expiry, and exceptions exceed their resolution target. Track the oldest age and total exposure, not just job success. A reconciliation batch that runs every night and produces an unread log has not repaired anything.

Operate around integrity, not only conversion

The main trace suggests the operational signals. Give every checkout a correlation identity without putting unrestricted payment or personal data into general logs. Measure reservation contention and expiry, checkout attempts by terminal state, provider ambiguity age, duplicate suppression, authorization and capture latency, outbox backlog, unposted financial events, unbalanced-journal rejection, reconciliation exceptions, and repair outcomes.

Alerts should point toward a threatened promise. A rising authorization timeout rate matters differently when reconciliation is current than when unresolved payment intents have been growing for hours. A low checkout error rate can hide a stuck ledger outbox. A healthy inventory API can still oversell if warehouse adjustments are not reaching the sellable count.

Security follows the same boundaries:

  • tokenize payment methods through an appropriate provider boundary and minimize the sensitive data the commerce system stores;
  • authorize checkout, capture, refund, ledger, reconciliation, and support actions separately, with stronger controls for money-moving and administrative paths;
  • sign or otherwise authenticate provider callbacks, deduplicate them, and retain enough evidence to investigate disputes;
  • protect against account takeover, card testing, promotion abuse, automated stock capture, and refund abuse with rate, identity, and risk controls;
  • make administrative actions attributable and use dual approval where the stated risk requires it.

Cost is not an afterthought, but it attaches to the design rather than replacing integrity. Validate cart and eligibility before expensive provider or fraud calls. Expire abandoned holds. Keep hot inventory ownership narrow. Batch appropriate reconciliation reads while preserving urgent repair paths. Separate audit-grade financial evidence from high-volume behavioral analytics with different access and retention. Make provider fees, risk checks, support workload, storage, and cross-region coordination visible when comparing architectures.

Transfer the reasoning to the named prompt

The failed checkout is not a canonical diagram to memorize. It supplies a set of questions: what promise is being made, who owns it, how an operation is identified, which transition needs a guard, what an ambiguous response means, and which independent evidence can later expose a mismatch. Different commerce prompts put pressure on different answers.

For a shopping cart, the cart preserves user intent across sessions and devices. Price, promotion, tax, availability, and shipping estimates may change, so checkout must refresh them and the order must snapshot the accepted terms. The difficult parts are merge rules, stale versions, shared-device privacy, and the boundary between display and commitment—not storing a list of SKU ids.

For inventory reservation, concentrate on hot ownership, conditional hold and allocation transitions, expiry races, regional or warehouse placement, back-order policy, and reconciliation with physical truth. A reservation is temporary by design; the answer must show how every hold leaves held.

For order management, make the business state machine and support surface central. Orders survive delayed notifications, partial fulfillment, cancellations, returns, and split shipments. Orchestration can make a long-running transition visible; events let downstream systems react. Neither justifies an unguarded status column.

For payment processing, follow authorization, capture, settlement, refund, and dispute as distinct facts. Provider timeouts and duplicate callbacks are normal inputs. Multi-provider routing must preserve intent identity and avoid turning uncertainty into two charges. Sensitive-data scope and operational review matter as much as the adapter interface.

For a double-entry ledger, begin with journals, posting rules, accounts, currencies, source identity, atomic balance, immutability, and reversal. Then ask about required ordering, derived balances, backfills, rule evolution, access, reporting, and the external evidence used for reconciliation. A mutable balance field is a read model, not an adequate account of how money moved.

For reconciliation, make exceptions and repair the main path. Define evidence sources, exact matching, conservative secondary matching, classification, ownership, allowed actions, exposure, aging, and proof that the repair completed. The hard scaling question may be the exception workload rather than the number of input records.

Rehearse the unresolved minute

Give yourself twenty minutes with the last-camera case. Begin at 10:02, when the authorization call has timed out. Draw only the state owners and durable identities needed to answer these questions:

  • What does a browser retry read, and why can it not create another payment?
  • What exact transition races the 10:05 reservation expiry?
  • What evidence permits “order confirmed” to appear?
  • What happens if authorization is discovered after the camera was allocated elsewhere?
  • What durable obligation exists if capture succeeds but ledger posting is down?
  • Which later record would reveal that a refund exists at the provider but not in the order or ledger?

Then change one constraint: allow back-orders, split the order across warehouses, capture only on shipment, permit partial refunds, introduce a second provider, or require a marketplace sub-ledger for each seller. Carry the change through state, identity, user-visible language, reconciliation, and repair. If the answer merely adds a service, the constraint has not traveled far enough.

At 10:04, the provider webhook answered one question: the authorization exists. It did not answer whether the camera was still held, whether the order could be confirmed, whether capture would later settle, or whether every financial record would agree. A strong commerce design does not collapse those questions into one green status. It gives each promise an owner, keeps enough evidence to recover, and tells the customer only what the system can defend.