Case Study: Payments, Ledger, and Financial Correctness

The Capture That Succeeded During a Timeout

A marketplace moves payment capture behind a queue to shorten checkout latency. Soon afterward, a shopper taps Pay twice when the mobile client times out. The API accepts two messages. Worker A calls the payment provider and also times out, but the provider has captured the money. A success webhook arrives before Worker A records an outcome. Worker B then claims the duplicate message. Tomorrow’s settlement file will contain the provider’s final account of the capture, its fee, and the net transfer.

The system has done little work, yet it already faces the hardest payment question: what is true now? “The worker failed” is false. “The capture succeeded” is not yet established internally. Retrying may charge the customer twice; declaring success may credit money the provider never collected. A queue can move the ambiguity around, but it cannot resolve it.

The architecture therefore begins with three kinds of truth. A command records one business intent. A provider attempt records an interaction whose outcome may be unknown. A ledger transaction records a financial fact the organization has accepted. Keeping these records distinct lets the system move quickly without pretending an external side effect participates in its database transaction.

Name What Must Never Be Ambiguous

Payment traffic may be modest beside search or telemetry traffic, but each write carries unusual consequence. One duplicate capture can create a complaint, a refund, support work, a finance adjustment, and an incident review. One missing refund can remain invisible until a customer, provider, bank, or settlement report contradicts the application.

The marketplace in this case authorizes a card when an order is placed, captures after inventory is reserved, recognizes a platform fee, and later pays the merchant. Its system also handles voids, partial refunds, chargebacks, provider fees, reversals, and manual adjustments. Checkout status, receipts, entitlements, support tools, finance reports, dispute review, and reconciliation all read different views of this activity.

Some delay is acceptable. A receipt may lag, a support projection may rebuild, and a capture may remain pending while its provider outcome is investigated. Other states must be impossible. Every posted transaction must balance. A refund must stay within the amount allowed by written policy. One intended capture must not become two provider captures merely because messages were redelivered. Corrections must remain visible rather than erasing the fact they correct.

Those promises form the invariant register:

• One intended operation has one durable command identity. Reuse with the same request fingerprint returns the existing result; reuse for a different account, amount, currency, operation, or business object fails loudly.
• Every ledger transaction has a unique identity, typed purpose, and equal debit and credit totals. Capture, fee, refund, chargeback, payout, and adjustment remain distinguishable.
• Concurrent refunds cannot reserve or post more than policy permits against the captured amount.
• Provider attempts, webhooks, settlement rows, and operator actions remain immutable evidence even when they do not change ledger truth.
• Every reconciliation difference has a class, owner, age, and disposition.

The last item is as important as the first. In a system joined to a provider and a bank, correctness is not a one-time transaction property. It is a control process that keeps comparing accounts of reality.

Give the Intent a Durable Identity

Before publishing work, the API creates or finds a command record. For the capture in this case, that record includes the merchant account, order, operation, amount, currency, actor, request fingerprint, idempotency key, state, creation time, and links to any provider attempts.

The key identifies the effect the business intends, not the delivery that happened to carry it. A queue message ID is too weak: the two mobile requests can create different messages while asking for the same capture. A useful scope includes the account, order or invoice, operation, amount, currency, and the retry window defined by product policy. The database enforces uniqueness at that scope.

This makes the shopper’s second tap unsurprising. With the same key and fingerprint, it receives the existing command state: accepted, pending, succeeded, failed, or unknown. If the second request changes $120 to $1,200 while reusing the key, the API rejects it as conflicting intent. It does not silently return the result of a different operation.

The worker carries the same identity across the boundary. It claims the command with a lease or guarded state transition, writes a numbered provider-attempt record, and sends a stable provider request key when the provider supports one. A cache may reduce duplicate lookups, but it cannot own this guarantee. Payment retries are most likely when processes restart, leases expire, queues redeliver, and networks fail—the moments when an ephemeral deduplication entry is least trustworthy.

Follow the Command Through Uncertainty

The failed capture now has an inspectable history:

mobile retry
    |
    v
command C-417: capture $120, fingerprint F9, state=pending
    |
    +--> duplicate API request finds C-417; no second intent is created
    |
    v
provider attempt A-1: request key C-417, state=unknown after timeout
    |
    +--> webhook W-88: provider says capture P-602 succeeded
    |
    v
validate account + amount + currency + prior postings
    |
    v
ledger transaction L-944 and payment state commit together
    |
    +--> receipts, entitlement, support, and analytics consume L-944

An unknown attempt is not a euphemism for failure. While A-1 is unknown, Worker B must not create A-2 merely because its lease allows work. It first queries by the stable provider key or waits for provider evidence according to a bounded policy. The authenticated webhook supplies such evidence, but it is not allowed to mutate balances blindly. The webhook processor deduplicates W-88, matches it to C-417 and A-1, then verifies the provider account, amount, currency, operation, reference, and absence of an earlier posting.

Only then does the internal invariant boundary close. In one database transaction, the system inserts the balanced entries for L-944, associates provider reference P-602, advances the payment state, and records an audit event. Uniqueness and a serializable transaction—or an equivalent guarded mechanism—ensure that Worker A, Worker B, and the webhook processor cannot post the same capture twice.

The provider call does not belong inside that transaction. The provider cannot roll back because the local database aborts. The design instead accepts a gap between external effect and internal knowledge, gives that gap the honest state unknown, and provides more than one route to resolve it: provider lookup, authenticated webhook, and settlement reconciliation.

What the Ledger Accepts

A ledger is not a history of hopeful payment events. It is the organization’s account of posted financial movement. Commands and attempts explain what the system tried; the ledger explains which financial facts it accepted.

Suppose the case uses a modeled $120 capture, a $12 platform fee, and a $108 merchant obligation. Its posting could debit $120 to processor clearing receivable, credit $108 to merchant payable, and credit $12 to platform fee revenue. The debits and credits both total $120. If settlement later deposits $116 and withholds a $4 processing fee, a separate settlement transaction can debit $116 to cash and $4 to processing expense while crediting $120 from processor clearing receivable. Real account names and recognition rules belong to the organization’s accounting policy; the engineering obligation is that each accepted transaction balances and remains linked to its source evidence.

Downstream systems consume the committed posting, not capture_attempt_succeeded or webhook_received. A duplicate webhook can then add evidence without sending a second receipt or activating an entitlement twice. If projections are lost, they can be rebuilt from the ledger and its typed transactions rather than from an event stream that has become the sole copy of the fact.

Financial life continues after capture. A partial refund gets its own command identity, provider attempt, posting, and reconciliation path. Concurrent refund commands reserve against the refundable balance before either calls the provider. A chargeback arriving months later becomes a new typed transaction. An operator adjustment names its reason, approver when required, and linked incident. None overwrites the earlier posting to make the current balance look tidy.

This append-and-correct discipline protects auditability and reasoning. The system can answer not only “what is the balance?” but “which sequence of accepted facts produced it, which external evidence supports them, and who repaired any disagreement?”

Let Settlement Challenge the System

The next morning, settlement reports P-602 with the expected gross capture but a provider fee different from the team’s estimate. This is not a reason to rewrite L-944. It is a new observation to reconcile.

The reconciliation job compares commands, attempts, ledger postings, webhooks, settlement rows, bank movement, payouts, invoices, and customer-visible state. For each disagreement it records the amount and currency, affected account, provider reference, expected and observed facts, age, owner, and permitted repair. Common outcomes include:

• a provider success matched to an internally pending command, which may be posted after the invariant checks pass;
• an internal success absent after the provider’s settlement window, which requires provider inquiry or escalation;
• duplicate provider evidence, which is retained without reposting;
• an amount or currency mismatch, which can freeze the related payout while evidence is investigated;
• an internal refund missing at the provider, which may require a guarded retry or immediate customer-impact escalation;
• an unexpected fee, which finance may accept through a linked adjustment or dispute with the provider;
• a balance violation, which stops the affected posting path and is repaired by an explicit correcting transaction.

Reconciliation has its own freshness objective. A fast API can coexist with a settlement import that has silently fallen two days behind. HTTP availability would remain green while financial uncertainty ages. The dashboard therefore tracks the oldest unresolved difference, unmatched settlement value and count, unknown provider attempts, refund exceptions, and time to disposition by class and owner.

SLOs That Can See Wrong Money

The service specification separates four promises. Acceptance measures whether a valid command receives a durable identity within its latency target. Posting correctness measures duplicate posting, balance violations, and commands whose provider evidence cannot be joined to a ledger result. Outcome freshness bounds how long an attempt may remain unknown. Reconciliation freshness bounds settlement ingestion and exception age.

The exact thresholds come from business risk and provider behavior, not from a generic payment template. A team might page immediately on any unbalanced posting, use a short operational target for unknown captures, and give ordinary fee differences a longer finance-owned window. The specification must still state a denominator, time window, owner, escalation action, and source query for each promise. “99.9% payment availability” cannot carry all four meanings.

Auditability follows the same concrete standard. A reviewer should be able to traverse C-417, A-1, W-88, P-602, L-944, the settlement row, and every operator action. That evidence supports accounting, support, security, and compliance work, but it does not by itself prove compliance with every jurisdiction, processor agreement, or accounting standard.

Repair Without Rewriting History

Suppose an alert finds that one worker version generated unstable provider keys during retry. Before replaying anything, responders preserve the affected records and pause the exact capture worker version. They scope the incident by deployment window, account, operation, provider reference, idempotency-key pattern, amount, and currency. Their evidence bundle joins commands, fingerprints, attempts, webhooks, settlement rows, ledger transactions, operator actions, and customer-visible state.

Only then do they classify each case: one provider capture with one posting; one capture still unknown; duplicate provider captures; a posting with no supported provider effect; or an unrelated retry. Each class has a written repair authority. An unknown capture may require provider lookup. A duplicate capture may require a refund and customer communication. A missing internal posting may be added only after provider evidence and invariant checks agree. An incorrect posting is offset by a correcting transaction; it is never silently edited.

The wrong-data runbook must therefore name:

1. the symptom and the invariant believed to be at risk;
2. the narrowest safe freeze control—command type, worker version, webhook processor, payout path, or downstream side effect;
3. the scope query and evidence-preservation location;
4. customer, merchant, finance, and reporting impact owners;
5. permitted repairs for each evidence class, including approval requirements;
6. verification queries across provider, ledger, settlement, and customer state;
7. the constraint, key scope, attempt-state rule, replay guard, alert, or ownership change that prevents recurrence.

Mass replay appears nowhere near the beginning. Replaying an ambiguous payment path can repeat the side effect that caused the incident. The first safe action is to stop new damage without destroying evidence; the last is to prove that every affected intent has one explainable disposition.

The Architecture Decision

The marketplace keeps a double-entry ledger as the source of posted financial truth. Durable command records own business intent and idempotency. Provider-attempt records own external uncertainty. Authenticated webhooks and settlement rows are immutable evidence. Balanced posting, internal payment state, provider linkage, and audit record commit within one invariant boundary; receipts, fulfillment, entitlements, support projections, and analytics follow that committed state.

The team rejects three easier-looking designs. It does not treat a queue message as the payment record, because delivery identity is not business identity. It does not interpret a timeout as failure, because the provider may have acted. It does not let webhooks become an unguarded balance API, because repeated or unmatched evidence cannot define financial truth.

The price of the chosen design is explicit: more durable records, unknown-outcome handling, provider lookup, immutable corrections, reconciliation ownership, and operational drills. Those costs are not bookkeeping around the architecture. They are the architecture that makes asynchronous payment work defensible.

To review the design, take the original seven-step failure and fill in the exact record after each step: command identity, request fingerprint, attempt state, provider reference, customer-visible state, ledger state, and next safe action. Then test three variations: the webhook arrives twice, settlement disagrees on amount or currency, and two partial refunds race for the remaining refundable balance.

If the design can answer those variations from durable state, duplicate delivery is routine and provider uncertainty is bounded. If the answer depends on which worker remembers what happened, adding more asynchronous throughput will only make the uncertainty arrive faster.