Event Logs, Queues, and Streams

One Purchase, Four Promises

A billing service asks a payment provider to capture an invoice. The provider completes the charge, but its response never reaches the service. The customer retries. Meanwhile, account access must change, a receipt must be sent, support needs an explanation, and finance expects the payment in its next report.

Putting a message bus in the middle does not resolve any of those obligations. It only gives them somewhere to wait. The design still has to answer four different questions:

• What state is the payment process in while the provider’s result is uncertain?
• Which business facts must remain available to independent consumers?
• Which work should one worker attempt, and what makes another attempt safe?
• Which derived views can be rebuilt from retained input?

Those questions lead to different mechanisms. A workflow owns durable process state. An event log retains facts. A queue coordinates attempts. A stream processor computes derived state over time. Pub/sub is useful for fan-out when its delivery and retention contract is sufficient, but it does not acquire stronger semantics merely because many subscribers listen.

The distinction is practical: it tells the operator where to look when the customer says, “I was charged twice,” or, “I paid, but my account is still locked.” A system that calls every asynchronous path a queue has erased its own recovery map.

Facts Need a Log

Suppose the payment workflow eventually establishes that invoice I-2048 was captured under provider operation P-771. PaymentCaptured is now a fact. Account access, customer support, finance analytics, settlement reconciliation, and an audit archive may all need it for different lengths of time and at different speeds.

A durable event log gives those consumers independent positions in retained history. Producers append records to partitions. Each partition has an order, and consumers track offsets. A consumer group can divide one logical subscription among several instances while another group reads the same records on its own schedule.

That independence is the log’s real value. If the support projection is corrupted, its owner can reset to an earlier offset and rebuild without asking finance to stop consuming. If analytics is down for an hour, account access need not wait for it. The log decouples progress; it does not make every consumer correct.

Replay works only within retained history. Retention must cover ordinary lag, failed deployments, long outages, investigation, backfills, and the longest credible rebuild. If a consumer falls behind the retention boundary, an offset cannot recover records that no longer exist. A serious rebuild plan therefore combines log retention with snapshots or archived history where the recovery window demands it, and alerts on remaining retention margin rather than raw lag alone.

Ordering is bounded too. The useful guarantee is normally order within one partition, determined by a key. Keying payment events by invoice_id keeps one invoice’s transitions together. Keying by account_id may be necessary when account state depends on the order of several invoices, but one very large account can then become a hot partition. A random key spreads load by surrendering business order. The team must choose which order protects an invariant and accept the throughput boundary that follows.

The log answers, “What facts were retained, and how far has each consumer read?” It does not answer, “Should this external effect happen again?” Replaying PaymentCaptured into a support projection can be safe. Replaying it into a handler that captures payment or sends a receipt may repeat the very effect the operator is trying to repair.

Work Needs a Queue

Once PaymentCaptured exists, sending receipt R-2048 is work. One worker should own an attempt. If it crashes before acknowledging completion, the work needs another attempt. That is the queue’s job.

A queue presents a message to a competing consumer and waits for acknowledgment. Many queue designs make an unacknowledged item visible again after a lease or visibility timeout. If that interval is shorter than real processing time, two workers may send the same receipt. If it is much longer than the desired recovery time, a crashed worker leaves the customer waiting. Timeout, acknowledgment, and retry policy are therefore correctness settings, not tuning details.

The handler must survive redelivery. A stable semantic key such as receipt:invoice:I-2048 lets it record that this particular customer-visible effect has already succeeded. A transient process-local flag does not. For a local database update, an inbox or deduplication record can often be committed in the same transaction as the state change. For an external API, the consumer may need the provider’s idempotency facility, its own durable outcome record, and later reconciliation because the network can still fail after the provider acts.

Failed work eventually needs judgment. A dead-letter queue should be an owned repair lane, not a destination that converts a page into silence. Its policy should state why an item was rejected, who is alerted, how long it may wait, whether replay is safe, how customer impact is found, and when a person must decide between retry, alternate completion, compensation, and cancellation. A growing DLQ with no age limit is lost work under a tidy name.

The queue answers, “What work still needs an attempt?” That is different from the log’s question, “What happened?” Keeping both records lets an operator prove that a payment was captured even while the receipt job remains unresolved.

The Process Needs Durable State

Payment capture itself is neither a retained fact nor an ordinary background job while the result is uncertain. It is a long-running process with external calls, timers, branches, retries, possible compensation, and a customer-visible state.

The workflow record might move from capture_requested to awaiting_provider_result, then to captured, failed, or manual_review. After a timeout, it must not simply enqueue the same command and hope. It should query the provider using the original operation or idempotency key, reconcile any recorded outcome, and retry only when the evidence permits it. Operators and customer support need to see that state directly; reconstructing it from a pile of pending messages during an incident is too late.

A workflow engine can make durable state, timers, retries, and activity history explicit. Application-managed state machines can provide the same essential ownership when built carefully. The important choice is not the product label. It is whether the business process has one inspectable state and one place where timeout, compensation, and human intervention are decided.

Sagas describe coordination when one transaction cannot cover every participant. Choreography lets local services react to events; orchestration gives an explicit coordinator the next-step decision. Choreography can keep participants independent, but the process becomes difficult to see when many reactions and compensations interact. Orchestration concentrates knowledge and responsibility. Neither approach supplies idempotency, compensation, or observability by itself.

The workflow answers, “Where is this business process, and what transition is safe next?” A queue can schedule one of its activities. A log can record a completed transition. Neither should be mistaken for the process state itself.

Streams Turn History Into Derived State

Finance wants payment totals by minute, support wants the latest account status, and risk wants a rolling signal from recent attempts. These are computations over event flows. A stream processor consumes retained records, keeps state, and emits new records or materialized views.

Time now becomes part of correctness. The design must say whether a window follows event time or processing time, how late records revise an emitted result, where processor state is checkpointed, and how that state is restored. When code or schemas change, the team must know whether outputs can be recomputed from old input and whether the new interpretation is compatible with reports already used.

A stream output remains derived data. It needs lineage, a freshness target, a replay or backfill procedure, and a way to compare input progress with output completeness. A stream processor can offer strong internal state guarantees and still duplicate an email or external call at its output boundary. The guarantee must be stated where the observable effect lands.

Pub/sub occupies a narrower place. It fans a message to subscribers, which is useful for cache invalidation, transient notifications, or integrations whose loss and delay contract is explicitly acceptable. Some pub/sub systems also retain messages and support replay; some do not. The category name does not decide the guarantee. If rebuilding an account view depends on the history, require a durable retained source rather than assuming the bus will remember it.

Close the Publication Gap

The first lost event can occur before any consumer sees the log. The billing service commits a captured payment in its database and then publishes PaymentCaptured. If the commit succeeds and publication fails, downstream consumers miss a true fact. If publication succeeds before the transaction later fails, they receive a lie.

A transactional outbox closes most of that gap. The business change and an outbox record commit in the same local transaction. A relay publishes committed outbox records and records its progress. The relay may publish a duplicate after crashing at an awkward moment, so consumers still need deduplication, but publication intent can no longer disappear independently of the business change.

Change data capture can publish committed database changes without requiring every application write path to send an event. It is well suited to derived-data movement when table ownership, snapshots, catch-up, deletes, and schema changes are explicit. Raw row changes are not automatically good business events, however. A column update may reveal storage mechanics while concealing the business transition consumers need.

At the receiving side, an inbox or deduplication table closes the corresponding local gap. The consumer records event E and its local state change in one transaction. The record must live longer than actual retry, replay, restore, and manual-repair windows; expiring it sooner merely schedules a future duplicate.

These patterns do not create a single transaction across the billing database, log, email provider, warehouse, and payment provider. They make each boundary explicit enough to retry and reconcile.

Lag Reveals the Real Load

Asynchrony does not remove demand. It stores demand in queue depth, consumer lag, retry schedules, stream state, workflow timers, and operator attention.

In the billing path, a finance consumer that is 30,000 records behind might still meet an hourly reporting target. An account-status projection that is five minutes behind may already be locking out paying customers. A receipt queue with one million young messages may be healthier than a queue with ten messages that have been failing for two days. Connect telemetry to the promise: age of oldest required work, customer-visible freshness, distance from retention loss, and time spent in an uncertain workflow state.

Queue owners need depth, oldest age, attempts, processing latency, worker saturation, and DLQ growth. Log consumers need lag by partition, remaining retention margin, processing failures, rebalance behavior, and hot-key evidence. Stream processors need watermark delay, late-record rate, checkpoint health, state growth, and output freshness. Workflow owners need stuck-state age, retry exhaustion, timer backlog, compensation frequency, and manual-review age.

Retries can amplify an outage. If the email provider is down, immediate retries spend worker capacity repeating a known failure. A malformed message can monopolize a partition or poison every fresh worker. Classify failures before scheduling another attempt: transient failures receive bounded backoff; permanent rejections move to owned repair; ambiguous external outcomes trigger reconciliation; and new safe work should not be trapped indefinitely behind one impossible item.

Every Message Is a Contract

An asynchronous API is harder to inspect than a synchronous one because the producer may not know every consumer, failure may appear hours later, and replay can expose payloads to code written years afterward.

The message name should reveal its purpose. PaymentCaptured states a completed fact. CapturePayment is a command. SendReceipt is a work item. ProviderResultReceived may be a workflow signal. A vague PaymentUpdated event forces every consumer to reverse-engineer meaning from fields and timing.

Important messages need stable identity for deduplication, ordering, tracing, and repair. Record the event or message id, business object id, ordering key, event time, producer time when different, contract version, producer, and correlation id. Carry a semantic idempotency key for an external effect when one exists.

Schema compatibility is an operating obligation. Adding an optional field is commonly easier than changing the meaning of an existing one. Removing or renaming a field requires knowledge of consumers and a deprecation path. Sensitive fields also acquire copies in logs, archives, DLQs, traces, and replay environments; retention and deletion policies must cover those copies rather than stopping at the source database.

Delivery claims should end at a named boundary. “Exactly once” is incomplete unless it says exactly once where. A precise account might be: “The account projection consumes at least once, commits the event id with the projection update, and can be rebuilt from the retained log. Receipt delivery is not replay-safe and is protected by the semantic send key.” This language exposes the remaining failure instead of hiding it under platform vocabulary.

Write the Semantics Record

Before approving an asynchronous path, write one short record for each promise, not one generic platform diagram.

Begin with purpose: retained fact, command, work item, stream input, workflow signal, or transient notification. Name the mechanism and the guarantee boundary. Then record the business ordering key, the hot-key consequence, retention or expiry, replay need, and whether replay is safe for every effect.

Follow the path to its owner. State who monitors lag or oldest age, who approves schema changes, which alert signals customer harm, and how poison messages, failed side effects, and stuck workflows are repaired. Include the maximum unresolved age and the point at which human review is mandatory.

Finish at the inconvenient boundaries: external calls, payments, inventory, email, permissions, privacy deletion, archives, and DLQs. State the idempotency or reconciliation mechanism and the evidence that proves recovery worked. If the record says only “at least once” or “events retained for seven days,” it has described infrastructure without describing the product promise.

Follow the Uncertain Payment

Return to invoice I-2048. Let the provider capture the payment and drop the response. Decide what the workflow displays, which evidence it queries, and what prevents a second charge. Once capture is established, publish the fact through an outbox and deliver it twice. Prove that the account projection changes once, finance retains the event, and the receipt queue creates one semantic send.

Now stop the account-status consumer until its lag threatens retention. Recover it from a snapshot and the remaining log, and show which alert fires before history is lost. Make the receipt payload invalid and follow it into repair: owner, customer impact, expiry, correction, and safe replay. Change the event schema while the finance consumer is one version behind. Finally, erase sensitive payment metadata from every retained copy whose lifecycle requires deletion.

The exercise succeeds when each failure has an obvious home. Facts remain available for independent consumers. Work remains visible until completed or deliberately resolved. Derived state can be reconstructed. The payment process exposes its durable state while an external result is uncertain. That is the useful difference between a log, a queue, a stream, and a workflow: each preserves a different kind of promise.