Time, Ordering, Idempotency, and Exactly-Once Myths

The Lie Hidden in a Clean Sequence Diagram

A checkout diagram usually shows one path: create the order, reserve inventory, capture payment, send confirmation, count the sale. Each arrow appears once. Time moves from left to right.

Now let the response to CreateOrder disappear. The client cannot tell whether the command failed or the response did, so it retries. The worker that handles the order event commits its database update, then crashes before acknowledging the message. The broker delivers it again. Payment succeeds, but its response times out. A mobile analytics event arrives tomorrow with yesterday’s event time. Next month, an operator replays the log to rebuild a projection.

None of these events is exceptional. They are the normal consequences of doing work across processes that can fail independently. A useful guarantee must therefore name one business effect and one boundary: this checkout attempt creates one order, or this payment intent produces no more than one captured charge. “Exactly once” without those nouns conceals the design.

One Order Has Several Times

Suppose order 12345 was placed on a phone at 12:00:00. The API received it at 12:00:03, a worker began at 12:00:07, and the order committed at 12:00:08. Those timestamps answer different questions.

Event time says when the business fact occurred. It may come from a device or an authoritative provider, so its trustworthiness is part of the contract. Ingestion time says when this system first received the fact. Processing time says when an attempt ran. Commit time says when durable state changed. Wall-clock time is what a particular machine believes now; clock skew and pauses make it unsafe as an accidental global sequencer.

Finance may assign the sale to a reporting period by an authoritative event time. Queue visibility and retry leases use processing or wall-clock time. An audit trail needs commit time. A lag alert compares event or ingestion time with processing time. Putting one field named timestamp on the message forces these decisions to share a meaning they do not have.

The distinction becomes visible when the analytics event arrives a day late. Nothing about its late ingestion changes when the customer placed the order. It may still change yesterday’s report.

The Retry Must Carry the Same Meaning

The client retry needs a stable identity for the checkout attempt. A fresh transport request ID is useful for tracing the second HTTP call, but it describes an attempt, not the business operation. If every retry gets a new business key, the server has no evidence that the caller is asking for the same effect.

Let the client choose a checkout key, say customer-84/cart-19/attempt-1, and keep it across retries. In the transaction that creates the order, the service also records that key under a uniqueness constraint and stores the result associated with it. The first call creates order 12345. A concurrent or later retry finds the key and returns the same outcome. It does not create order 12346.

That pattern is one form of idempotency: repeating an operation preserves its intended effect. Other state transitions get there differently.

• Setting email_verified = true is naturally repeatable, assuming authorization and audit requirements are still satisfied.
• Appending PaymentCaptured(provider_transaction_id) is safe when the provider transaction has a unique identity.
• Incrementing orders_completed is not safe when the increment has no event identity. Record unique completion events and derive or condition the count.
• Some operations commute: applying two independent additions to a set can produce the same final set in either order. Commutativity relaxes ordering, but it does not identify duplicates.
• Sending an email or calling a payment provider leaves the local transaction. It needs a stable intent, whatever idempotency the provider actually offers, recorded attempt state, and reconciliation.

Natural business keys can supply identity when their meaning is genuinely unique. An order number, provider transaction ID, or ledger-entry ID is stronger than a random message ID assigned anew at each hop. The producer that owns the key, its collision behavior, and whether a new user intention requires a new key must all be explicit.

Identity also has a lifetime. If a partner can redeliver for thirty days, a seven-day deduplication record leaves a twenty-three-day interval in which an old fact can become new work. Retaining every key forever may create unacceptable storage, deletion, and privacy costs. The deduplication window must cover retry queues, dead-letter repair, restore, partner behavior, and authorized replay—or the remaining risk needs an explicit reconciliation path.

Order Only What the Invariant Requires

Order 12345 should not move from delivered back to packed because a stale event arrived. Payment capture must not precede authorization. Those are ordering requirements with a scope.

A version per order can reject or quarantine an older state transition. A causal reference can say that capture depends on a particular authorization. Events for one order can share a partition key so a consumer sees their log order, while the handler still checks versions because retries, restores, and multiple producers can violate assumptions outside that log.

The system does not need to sequence order 12345 against an unrelated click from another customer. A total order across all events would add coordination and restrict parallelism without protecting either invariant. Some facts need per-aggregate order, some need causal dependency, and some have no meaningful order at all.

“The broker preserves order” is incomplete. The review needs to name the ordering key, the producers covered by the guarantee, what happens during repartitioning or retry, and how the consumer treats an older, missing, or concurrent version. Transport order can support a business rule; it cannot invent one.

Follow the Checkout Across the Commit Boundary

The order service can make its local claim precise. In one database transaction it inserts order 12345, records the checkout key, reserves locally owned inventory if that invariant shares the database, and inserts an OrderCreated row into an outbox. Either all of those changes commit or none do.

The rest of the workflow still happens as attempts:

1. An outbox relay reads the committed row and publishes OrderCreated. If it crashes before marking the row sent, it may publish the event again. The event therefore keeps the same identity across relay attempts.
2. A payment consumer receives the event. At-least-once delivery favors eventual work over duplicate-free delivery: an unacknowledged message can return. At-most-once delivery avoids redelivery by accepting that a crash can lose work. Neither mode alone produces one business effect.
3. The consumer opens a transaction, inserts the event ID into an inbox table, and records the payment intent. A unique inbox key turns a redelivery into a known duplicate. The state change and duplicate record must commit together; otherwise a crash can separate the proof from the effect.
4. Payment capture crosses into another system. No transaction around the consumer’s own database can atomically include an arbitrary external API. The consumer sends the same payment-intent key on every uncertain retry, records what it knows, and reconciles its state with the provider. If the provider cannot prevent duplicate capture, the local design must detect, void, or refund duplicates according to a named policy.
5. The consumer acknowledges the message only after the durable local work required by its delivery contract. A crash before acknowledgment causes redelivery, which the inbox recognizes. Acknowledging first would trade duplicates for possible loss.

This is often called effectively-once processing: duplicate attempts are possible, but identity, atomic local state, and repair make the chosen observable effect occur once within a stated horizon. Some stream processors can atomically commit their own state and input positions; some producer protocols can suppress duplicate records within their scope. Such guarantees are valuable precisely because their boundary is narrow. They do not automatically include a payment processor, email provider, search index, webhook target, or another service’s database.

Late Facts Need a Way Back Into the Truth

At 12:05, the reporting pipeline publishes the noon sales total. At 12:08, an offline phone delivers an order event whose trusted event time is 11:59. Should the report change?

The answer is a business policy, not a property of the clock. A streaming watermark is a progress declaration: the system is willing to close or publish a time range under its lateness assumptions. It is not proof that no older fact exists. Allowed lateness says how long the normal path will revise a result. A correction path says what happens afterward.

For analytics, the late order may recompute the affected window and publish a revised total. For finance, it may post a dated adjustment with an audit trail. For an order state machine, an older version may be rejected or sent to repair rather than applied. When an event arrives before its dependency, the consumer may buffer it briefly, retry, or quarantine it; indefinite buffering merely hides a broken contract.

Backfills amplify every weak assumption. They run old facts through current code, at unusual volume, outside the timing of live user intent. A replay that rebuilds an internal order projection should not resend confirmations or recapture payments. Side-effect gates, deterministic output identities, versioned transformations, checkpoints, and reconciliation turn replay into controlled repair rather than a second production history.

Make the Claim Small Enough to Test

Before accepting an “exactly once” claim, finish the sentence:

For one checkout key, the order database contains one committed order and one outbox intent, for as long as the key is retained. Duplicate publication and consumption are expected. Payment capture is a separate effect protected by the provider intent key and reconciliation.

That sentence admits where the guarantee ends. It also gives an operator evidence to inspect: command record, order, outbox row, broker event, inbox entry, payment intent, provider transaction, acknowledgment, and any corrective action. If those records cannot be joined by stable identity, the workflow may be retryable in theory and unrepairable in practice.

Write an idempotency and ordering plan for every retrying command, event family, webhook, job, or replay that can change important state. It should settle these decisions in ordinary language:

• Effect and identity: What business fact should occur once? Which key distinguishes a retry or replay from new intent, and who creates it?
• Time and order: Which clock governs each decision? Is ordering global, per aggregate, causal, or irrelevant? Which version or key enforces it?
• Transition and retention: Is the handler conditional, versioned, commutative, append-only, or compensating? Where is duplicate evidence stored, and for how long?
• Commit boundary: Which state, input position, acknowledgment, inbox, and outbox records commit together? Where does the workflow first leave that boundary?
• Late data and replay: Which late facts revise history, create corrections, or go to repair? Which side effects are disabled or given deterministic identities during replay?
• Observation and repair: Which metrics expose retries, duplicates, lag, lateness, dedupe expiry, and reconciliation drift? What may an operator safely retry, skip, quarantine, void, refund, or rebuild?

The plan has done its job when it identifies a crash point the design cannot yet survive.

Break the Workflow on Purpose

Take payment capture, confirmation email, and analytics ingestion. For each, name the business effect, stable key, ordering rule, deduplication horizon, relevant clock, external boundary, replay behavior, and repair action. The answers should differ: a delayed analytics correction is not the same failure as a second charge.

Then trace one consumer that writes to an external database or API. Interrupt it immediately before the call, after the external effect but before the response, after the response but before local commit, and after local commit but before acknowledgment. Retry each case. Restore an old snapshot and replay the input. Deliver an older version after a newer one. Advance beyond the deduplication window and deliver the original event again.

For every interruption, predict the user-visible result and name the record that proves what happened. If the team cannot distinguish a missing effect from an uncertain effect or a duplicate effect, it has found the next design task.

Retries, duplicates, disorder, and late data do not disappear behind a messaging guarantee. They become ordinary inputs when business identity survives transport, order is scoped to an invariant, local changes commit atomically, external effects can be reconciled, and replay has rules. The next challenge is keeping the copies produced by this workflow honest: caches, projections, search indexes, materialized views, and analytical tables all inherit the same imperfect history.