Event Processing, Idempotent Effects, and ‘Exactly Once’

Mercury’s order service once implemented acceptance with two ordinary calls:

commit order.status = ACCEPTED
publish OrderAccepted

The code looked honest. The database client returned success, then the broker client returned success. During one network partition, 312 accepted orders never reached fulfillment. The proposed fix was to publish first. That version created 47 fulfillment events for database transactions that later failed.

Neither order can make two independent systems atomic:

Retry does not repair the ambiguity. If the database committed but the response was lost, retry may create another event. If the broker accepted but its acknowledgment was lost, retry may publish another record. If compensation runs after a consumer has acted, “undo” is a new business operation, not erasure of history.

The repair starts by naming one authoritative transaction. Store the business mutation and an outbox record together in that transaction. A relay later publishes the outbox at least once. Consumers assume duplicates and enforce one intended effect at their own authority boundary.

The governing rule follows: specify exactly which effect must be unique, where uniqueness is enforced, and how replay behaves; avoid unqualified “exactly once” language. A system can provide exactly-once state transitions inside one transactional boundary while sending a duplicate email or making an ambiguous carrier booking outside it. That is not hypocrisy; it is a boundary that must be stated.

Name what the record means before processing it

Event-driven designs often call every payload an event. Meaning determines retry, authority, versioning, and correction.

• A command asks a named authority to attempt an action: ReserveInventory. It may be rejected, expire, conflict, or produce several facts. Its name is imperative, and it needs an outcome contract.
• An event or fact states that something occurred: OrderAccepted. It is immutable. A later correction is another fact linked to the original, not an in-place rewrite.
• A change record reports a mutation in a source representation: row 18 changed status from PENDING to ACCEPTED. It may be sufficient for replication but lack the domain meaning, causation, or stable schema required by business consumers.
• A snapshot states the current view at a version: order 18 is accepted at version 7. It can repair missed notifications but may collapse intermediate transitions.

SendOrderAcceptedEmail is a command, even if it travels on a topic called events. OrderAccepted is a fact, not permission for any consumer to repeat an unbounded side effect. orders.status changed is a storage fact whose column semantics may be too coupled for independent consumers.

A useful event envelope carries stable event_id, source, domain type, subject or aggregate identity, aggregate version, occurrence time, schema version, correlation identity, causation identity, and payload. The identity must survive producer retry. A trace identifier is not a substitute: one trace may contain many effects, and a replay may create a new trace for the same semantic event.

{
  "event_id": "evt_01K1M8Q2Y6",
  "source": "mercury.order-service",
  "type": "commerce.order.accepted.v3",
  "subject": "order/o-1842",
  "aggregate_version": 7,
  "occurred_at": "2026-07-13T12:04:18.381Z",
  "correlation_id": "checkout/c-8821",
  "causation_id": "cmd/accept-o-1842-v7",
  "data": { "order_id": "o-1842", "reservation_version": 3 }
}

Portability specifications can standardize envelope fields; they do not decide the domain invariant. Mercury defines uniqueness for source + event_id and ordering/version semantics for the order aggregate.

Put source state and publication intent in one transaction

The outbox changes the source write from two independent commits to one local commit:

BEGIN;

UPDATE orders
SET status = 'ACCEPTED', version = 7
WHERE order_id = 'o-1842' AND version = 6;

INSERT INTO outbox (
  event_id, aggregate_id, aggregate_version, event_type,
  occurred_at, payload, publication_state
) VALUES (
  'evt_01K1M8Q2Y6', 'o-1842', 7, 'commerce.order.accepted.v3',
  CURRENT_TIMESTAMP, :immutable_payload, 'pending'
);

COMMIT;

If the transaction commits, both order state and publication intent exist. If it rolls back, neither exists. A relay polls committed rows or consumes the database’s change log, publishes them, then records progress. The relay may crash after the broker accepts an event but before it records publication. It must publish the same event_id again. This is intentionally at least once.

The sequence is:

request ──> [order mutation + outbox row] one database commit
                                      │
                                      └─> relay/change capture ──> broker
                                                    │                 │
                                               retry same ID     accepted record

An outbox is not free atomicity across the broker. It relocates the ambiguity into a duplicate-safe relay while eliminating lost or phantom source facts. Its table or log needs partitioning, indexes, cleanup, monitoring, and backpressure. Publishing order must preserve the aggregate’s version rule. A relay that scans one global created_at index can become a write hotspot; a relay that publishes versions concurrently can reorder one aggregate.

Outbox evidence should include oldest unpublished age, pending rows and bytes, relay attempts, unknown outcomes, publish latency, rows skipped by lock contention, cleanup lag, and source-to-broker age. Retention must exceed incident diagnosis and replay needs, while cleanup must not delete the only evidence of an unknown publish.

Change-data-capture systems can route outbox table changes without polling. That removes polling load but adds database-log retention, connector offsets, schema mapping, snapshot behavior, failover, and connector recovery to the contract. “CDC” names a mechanism; it does not remove the relay’s ownership.

Put consumed identity and the durable effect in one transaction

On the consumer side, Mercury inventory reserves once for (order_id, reservation_version). The inbox identity and reservation write share the inventory database transaction:

BEGIN;

INSERT INTO inbox (consumer_name, event_id, first_seen_at)
VALUES ('inventory-reservation-v3', 'evt_01K1M8Q2Y6', CURRENT_TIMESTAMP)
ON CONFLICT DO NOTHING;

-- Continue only if the inbox insert created a row.
INSERT INTO reservations (order_id, reservation_version, state)
VALUES ('o-1842', 3, 'HELD')
ON CONFLICT (order_id, reservation_version) DO NOTHING;

COMMIT;
-- Advance transport progress only after commit.

The inbox constraint suppresses the same event. The reservation constraint enforces the semantic outcome even if two distinct events request the same version. These identities answer different questions. An event ID is transport history; an effect key is business meaning.

If the consumer crashes after commit but before acknowledging transport, redelivery encounters both uniqueness constraints and returns the recorded outcome. If it acknowledges before commit, the effect can be lost. If inbox and reservation live in different databases, the dual-write problem returns.

An idempotent operation must define response replay. A duplicate request should not merely say “already exists” if the caller needs the original result. Store outcome state, relevant response identity, and conflict evidence. Distinguish:

• same key, same canonical intent: return current or original outcome;
• same key, different intent: reject as conflict;
• attempt still pending: return pending and a status handle;
• previous outcome unknown: reconcile rather than blindly repeat.

Deduplication has a scope, horizon, and storage bill

“We cache IDs” is not a durability design. Specify:

1. Identity: event ID, command ID, source position, aggregate version, or business effect key.
2. Namespace: producer/source, consumer, tenant, operation, and environment.
3. Canonical intent: which fields must match when a key repeats.
4. Retention horizon: maximum producer retry, broker redelivery, replay, offline consumer, and backfill age.
5. Eviction behavior: what happens if an old duplicate arrives after evidence expires.
6. Authority: local memory, durable database, sink uniqueness constraint, or external idempotency service.

A process-local cache improves speed but loses history on restart and cannot coordinate replicas. A Bloom filter can reject likely duplicates cheaply only if false positives are acceptable or confirmed in authoritative state. A time-to-live index bounds storage but turns a late replay into a new attempt.

For Mercury’s stream, 18,000 events/s with a 12-hour deduplication window produces:

entries = 18,000 × 12 × 3,600 = 777,600,000
storage = 777,600,000 × 96 bytes = 74.6496 GB

The 96-byte modeled entry includes key, state, time, and index/storage overhead for the teaching calculation; real overhead must be measured. Replication, write amplification, compaction, cache, and backups increase physical cost. If approved replays reach seven days, a 12-hour horizon is invalid regardless of its attractive size. Options include a durable effect constraint with longer natural lifetime, hierarchical identities, per-partition source frontiers plus exception state, or a separate replay namespace whose writes are compared before promotion.

Deduplication itself can be a hot path. Partition it by a stable scope, test skew, bound cleanup work, and monitor lookup/insert latency and false conflicts. Deletion storms at the TTL boundary can compete with live writes.

Stateful processors recover from positions plus consistent state

A stateless map can replay input and produce the same deterministic output bytes if code, schema, reference data, and ordering are stable. A stateful operator—window, join, aggregate, session, pattern detector, or deduplicator—depends on accumulated state.

A useful checkpoint binds:

• source positions included in the computation;
• operator state at a consistent cut;
• timers and event-time progress;
• code and state schema compatibility;
• destination commit or transaction state when the sink participates;
• checkpoint identity, duration, bytes, and completion evidence.

On failure, the processor restores a completed checkpoint and replays source records after its recorded positions. Records processed after the checkpoint may execute again. Exactly-once processor state means the restored state reflects each source record once according to the engine’s protocol. It does not automatically retract an HTTP call, email, file write, or database mutation performed outside the checkpoint transaction.

Checkpoint and replay arithmetic

Mercury processes 18,000 events/s and completes a checkpoint every 30 seconds. Assuming failures are uniformly distributed within the interval, the average uncheckpointed input is:

18,000 events/s × 15 s = 270,000 events

The worst full interval is 540,000 events. With recovery processing at 24,000 events/s while 18,000 events/s continue arriving, net catch-up is 6,000 events/s. Ideal catch-up is 45 seconds for the average position and 90 seconds for the full interval. Restore time, source fetch, cache warming, skew, sink demand, and checkpoint coordination add to this model.

Shorter intervals reduce replay but increase snapshot, coordination, storage, and sink-commit overhead. Longer intervals amortize checkpoint work but enlarge recovery point, transaction visibility delay, and duplicate exposure for external sinks. Measure completed rather than merely started checkpoints; repeated timeout or alignment delay can silently extend the actual interval.

For large state, incremental checkpoints can copy only changed blocks, but recovery still depends on the base chain, object-store throughput, metadata, and compatibility. Reserve recovery bandwidth separately from live checkpoints and backfills.

Event time makes result meaning explicit

Processing time is when a worker observes the record. It is easy and low latency, but a retry or backlog moves the same fact into a later window. Event time is when the domain event occurred, derived from the record. It preserves historical meaning across delay and replay, but the processor must decide when a time range is complete enough to emit.

A watermark is a progress claim: the processor estimates that events at or before time (T) are unlikely or not expected to arrive on time according to a defined source policy. It is not proof that no earlier event can ever appear. With multiple inputs, progress is commonly constrained by the slowest non-idle input; an idle partition can stall all windows unless idleness is detected safely.

Mercury sets an operational watermark from the maximum observed event time minus two minutes for a bounded source class. It emits a provisional 10-minute demand aggregate when the watermark passes the window end, accepts corrections for 15 additional minutes, and then routes later facts to a correction workflow. Those numbers must come from observed arrival-lateness distributions, source outages, clock behavior, and the consequence of delayed versus corrected results.

Late-data policies include:

• wait longer, increasing result latency and retained state;
• update or retract a prior result, requiring sinks and readers to understand versions;
• emit an explicit correction fact linked to the prior result;
• route late data to review or batch reconciliation;
• drop it only when the lost contribution is declared acceptable and measured.

“Final” is a product policy. A financial ledger may never silently revise a closed period; it posts an adjusting entry. A dashboard may update a displayed bucket. A notification system may ignore a late presence event. State the policy in the result contract.

Event timestamps also need source and uncertainty semantics. Device clocks can be wrong. Database commit time, domain occurrence time, ingestion time, and processing time answer different questions. Chapter 33 develops clock, sequence, epoch, and causality choices in detail.

External side effects expose the edge of the transaction

Suppose fulfillment must book a carrier pickup through an HTTP API. The processor can atomically mark its local state CALL_REQUIRED; it cannot atomically commit the carrier’s database in the same local transaction.

The strongest practical path is:

1. derive a stable effect key such as (order_id, pickup_version);
2. store local effect intent durably;
3. call an API that accepts that idempotency key and binds repeats to canonical intent;
4. persist the returned carrier operation ID and outcome;
5. on timeout, query status by idempotency key or operation ID before retrying;
6. reconcile pending/unknown states until terminal.

If the carrier accepts the request, commits a pickup, and loses the response, the caller sees an ambiguous timeout. When the API supports neither an idempotency key nor status lookup, Mercury cannot prove whether retry creates a second pickup. No broker setting or processor checkpoint closes that gap. The honest contract is “at least one attempt; unique pickup not guaranteed under ambiguous completion,” followed by a redesign, manual reconciliation, or compensating cancellation if the business permits it.

Email has a similar boundary. The local outbox can ensure one send intent, but a provider may accept twice after an unknown response, and recipients may observe provider retries. Define whether the unique effect is intent creation, provider acceptance, message ID, or human-visible delivery. Avoid claiming more.

Processing guarantees and outcome guarantees are different

Use this decision table to keep claims scoped:

An “exactly once” review should require a sentence of this form:

For <effect>, identified by <key and namespace>, <authority> enforces at most one
committed outcome for <retention/lifetime>. Transport and processing may retry.
On replay, <same intent> returns <recorded result>; <different intent> conflicts;
unknown external completion is resolved by <status/reconciliation mechanism>.

Mercury can say: “For inventory reservation (order_id, reservation_version), the inventory database enforces one committed reservation for the lifetime of the order; repeated deliveries return the stored state.” It cannot extend that sentence to the carrier without the carrier’s key/status contract.

Reprocessing is a new versioned production run

Reprocessing old records with today’s code is not automatically reproduction. The reader must handle historical field presence, defaults, unknown enum values, renamed meanings, deleted reference data, changed joins, and new side-effect policy. Record the event schema version, processor version, state migration version, reference-data snapshot, time policy, and destination namespace.

Prefer immutable event facts plus explicit correction events. If a schema evolution changes meaning, write an adapter per historical version or produce a versioned normalized fact. Never reinterpret an absent old field using a current default without checking semantics.

A safe reprocessing run uses a new consumer identity and isolated output, validates counts and invariants by partition/time/key, compares old and new results, and promotes through an explicit cutover. If it must update the live destination, use a run namespace and effect version so replay does not collide accidentally or repeat external side effects.

Historical compatibility includes the code that restores checkpointed state. Removing a serializer or changing key structure can make the last good checkpoint unreadable. Test restore and migration before deployment, not during an incident.

Backfill capacity must be subordinate to live correctness

Mercury needs to recompute 1.2 billion historical demand facts. The live stream arrives at 18,000 events/s. The processing estate has a tested safe envelope of 25,000 events/s for the combined mix. The plan reserves 20,000 events/s for live work, 1,000 events/s for variance and control traffic, and caps backfill at 4,000 events/s:

1,200,000,000 / 4,000 events/s = 300,000 s = 83.33 hours

The reservation intentionally exceeds current live arrival by 2,000 events/s. That absorbs bursts and prevents the controller from treating average spare capacity as guaranteed. Backfill uses a separate group, worker pool, state namespace, sink quota, checkpoint path, and retry budget. It pauses when live age, checkpoint duration, sink saturation, compaction debt, or correctness errors cross their bounds.

Resource isolation must reach shared bottlenecks. Separate worker deployments still contend if they share broker fetch quota, network link, schema service, object store, state database, compaction I/O, or external API quota. Track service demand and queue age at every stage.

Backfill completion is not merely offset end. Validate record counts and bytes, key coverage, rejected/late/schema-error classes, state invariants, output checksums or aggregates, and a sampled semantic comparison. Keep checkpoints so a pause resumes rather than restarts 1.2 billion events.

Observe one causal chain without confusing its identities

Asynchronous processing breaks one request trace across time, retries, and services. Preserve distinct identities:

• correlation ID: groups a user or business journey;
• causation ID: points to the command or event that caused this event;
• event ID: identifies one immutable fact publication intent;
• aggregate ID and version: order facts within a domain authority;
• transport position/delivery ID: locates a broker record or attempt;
• processor run and checkpoint ID: identifies code/state execution context;
• effect key and provider operation ID: identify the intended outcome.

Propagate trace context when useful, but create linked spans for asynchronous processing rather than pretending a three-day replay is one continuously executing request. Store correlation and causation in durable event metadata so evidence survives trace sampling and retention.

An evidence query for o-1842 should answer:

1. Which source transaction and outbox row created the fact?
2. When did the broker accept it, at what partition and position?
3. Which consumer attempts ran, with what code/schema/checkpoint versions?
4. Did inbox insertion win or detect a duplicate?
5. Which reservation or carrier operation became durable?
6. When was transport progress advanced?
7. Was the result provisional, corrected, replayed, compensated, or quarantined?

Measure age between stages, not only duration inside them: source-to-outbox, outbox-to-broker, broker queue age, fetch-to-effect, effect-to-ack, watermark delay, checkpoint age, and end-to-end effect age. A healthy processor latency histogram can coexist with a six-hour-old outbox row.

Applied repair: Mercury order acceptance to inventory effect

The repaired workflow has four explicit authorities:

1. The order database atomically commits ACCEPTED(v7) and outbox evt_01K1M8Q2Y6.
2. The relay publishes that immutable event at least once, preserving order_id key and event ID.
3. The inventory database atomically records inbox identity and reservation (o-1842, v3).
4. The inventory consumer advances its transport position only after that commit.

The outcome claim is not that the event “is delivered exactly once.” It is:

effect: inventory reservation
semantic key: (order_id, reservation_version)
authority: inventory database unique constraint and state machine
lifetime: order retention plus audit horizon
same intent: return stored reservation state
different intent: conflict; do not mutate the existing reservation
transport replay: allowed; inbox/event ID and effect key suppress repeats
external carrier pickup: separate pending/unknown/succeeded protocol

The relay may publish twice. The consumer may execute twice. A checkpoint may restore and replay. The inventory effect remains one because its authority recognizes the semantic key. Carrier pickup remains a separate claim because its provider boundary differs.

Failure-window drills

Prove the outbox repair. Crash before source commit, after source commit, after relay send, after broker acceptance, after relay progress, after consumer inbox insert, after effect commit, and before transport progress. For each point, show the durable evidence, owner of retry, possible duplicate, and repair query.

Price deduplication honestly. Reproduce 777.6 million entries and 74.6496 GB for 12 hours. Add replication and a measured storage-engine amplification. Compare the horizon with maximum broker retention, producer retry, offline consumer, dead-letter redrive, and approved replay. If any exceeds 12 hours, change the contract rather than hiding the gap.

Restore processor state. With 18,000 events/s and 30-second checkpoints, reproduce 270,000 average and 540,000 full-interval replay. Add measured restore time and a 25% hot-key slowdown. Verify timers, watermark, sink transactions, and state schema after restore.

Challenge an external API. Ask whether it accepts a stable idempotency key, rejects different intent under the same key, retains the result through the retry horizon, and supports status lookup after timeout. If one answer is no, write the weaker outcome claim and the reconciliation or compensation path.

Reprocess without rewriting history. Select one old event schema and one changed rule. Pin schema adapters, code, reference data, event-time policy, and output namespace. Run a bounded range, compare invariants, then design cutover. Ensure no email, payment, carrier, or irreversible effect is repeated accidentally.

Durable event-processing rules

1. Distinguish commands, immutable facts, storage change records, and snapshots before assigning delivery behavior.
2. Repair source database plus publication dual writes with one authoritative transaction and durable publication intent.
3. Commit consumed identity and local effect together; transport progress follows the effect.
4. Give deduplication a semantic key, namespace, canonical intent, durable authority, horizon, and eviction policy.
5. Treat processor checkpoints as consistent state-and-position recovery, not as transactions over arbitrary external effects.
6. State event-time, watermark, lateness, correction, and finality policy explicitly.
7. Name the unique effect and enforcement boundary; processing exactly once is not automatically an exactly-once business outcome.
8. Treat reprocessing as a versioned run over historical schemas, code, state, reference data, and time semantics.
9. Isolate and govern backfill through every shared bottleneck, reserving live and recovery headroom.
10. Preserve causation, event, position, checkpoint, and effect identities so one outcome can be reconstructed.

Event identity and event time have now become correctness inputs, not logging conveniences. Chapter 33 examines the harder cases: clocks disagree, messages cross ordering domains, leaders change epochs, identities are generated in several places, and causality cannot be recovered from timestamps alone.

Evidence and transfer limits

• The CNCF CloudEvents core specification defines portable event context and the uniqueness scope of source plus id. It does not define Mercury’s aggregate version or business-effect key.
• Debezium’s stable outbox event router documentation documents one change-data-capture implementation of the outbox pattern. Its routing behavior does not remove source schema, relay, cleanup, or consumer idempotency obligations.
• Apache Kafka’s design documentation scopes idempotent and transactional log processing. Apache Flink’s stable documentation on stateful processing and event time explains checkpoints, replay, watermarks, and late data in that engine. Product-level guarantees do not include arbitrary external effects.
• W3C Trace Context standardizes propagation fields useful across messaging intermediaries. Trace identity remains observability context, not an idempotency or business-effect key.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/event-processing/ reproduces 270,000 average and 540,000 full-interval checkpoint replay, 45/90-second ideal catch-up, 777.6 million dedup entries and 74.6496 GB, and an 83.33-hour governed backfill at 4,000 events/s. These are deterministic teaching calculations, not measured engine or storage performance.