Queues, Logs, and Stream Processing

“Saved” is the beginning of the async story

A merchant changes a product’s price from 100 to 90. The product API commits the new version and returns 200 OK. Within seconds, three reports arrive:

• the product page shows 90, but search still shows 100;
• one merchant receives the price-change notification twice;
• the catalog-change dashboard has stopped advancing.

The database write succeeded. Everything that follows it is work the system moved across time: updating a search index, refreshing recommendation inputs, sending a notification, and incorporating the change into analytics. The API is no longer waiting for those effects, but somebody still owns them.

That is the useful way to approach queues, logs, and stream processors. They do not make work disappear. They transfer responsibility while the producer and consumer run at different times and can fail independently.

For this price change, the design must answer four questions before it names a product or protocol:

1. What fact became durable when the API said “saved”?
2. Which later effects may lag, and what will the user see while they do?
3. What happens when a record is delayed, duplicated, or processed out of order?
4. How will operators repair or replay the work without repeating an irreversible effect?

Put the fact and the publication intent in one commit

Suppose the product service first updates its database and then publishes a message:

update product price to 90
commit

publish product_updated

A crash between the commit and the publish leaves the source of truth at 90 with no record for downstream consumers. Reversing the operations creates the opposite lie: consumers can observe an event for a database change that later fails.

The transactional outbox closes this gap by committing the business change and the intent to publish in the same database transaction:

begin transaction
  update product
    set price = 90, version = 184
    where product_id = 42

  insert outbox_event(
    event_id,
    event_type,
    aggregate_id,
    aggregate_version,
    occurred_at,
    payload
  ) values (
    'evt-7f3',
    'product_price_changed',
    42,
    184,
    '2026-07-15T08:10:00Z',
    '{"old_price":100,"new_price":90,"currency":"USD"}'
  )
commit

An outbox publisher can now retry until the record reaches the broker. The publisher itself may send evt-7f3 twice if it publishes successfully and crashes before marking the outbox row complete. That is acceptable only if the event carries stable identity and consumers expect duplicates.

The API’s success still has a precise limit. It means version 184 and its publication intent are durable. It does not mean search, notifications, and analytics are already current. If immediate search freshness is part of the product promise, the request must wait for that stronger condition or the UI must read the source of truth. A broker cannot quietly redefine “saved.”

A log keeps facts; a queue assigns attempts

The price change is a fact that several systems need to observe. A durable append log fits that shape: records remain available for a retention period, each record has a position, and independent subscribers can advance at their own pace. Search can fail for an hour without preventing analytics from continuing. A new consumer can begin later and read retained history.

The downstream notification is different. “Send this merchant one email” is a command with one operational owner. The notification service may consume the price-change event, decide that a message is warranted, and enqueue a provider-specific job:

product_price_changed     fact: many groups may observe it
send_merchant_email       command: one worker should perform it

This distinction is more useful than arguing that queues and logs are rival technologies. A real path often uses both:

product transaction
  -> outbox
  -> product-events log
     -> search consumer group
     -> recommendation consumer group
     -> analytics consumer group
     -> notification consumer
        -> send-email work queue

The log answers “what happened, in what order for this key, and who may need to replay it?” The work queue answers “who owns this attempt, how long may it wait, and when should it be tried again?”

Queue workers usually receive a job under a lease, reservation, or visibility timeout. A worker that finishes acknowledges it. If the worker crashes or the lease expires, another worker can receive the same job. The timeout must be long enough for ordinary work yet recover when a worker disappears; long jobs may need lease renewal. Either way, redelivery is part of normal recovery.

Partition by the order the product actually needs

The log is divided into partitions so several workers can process it in parallel. Order exists within a partition, not across the whole log. The producer therefore keys product events by product_id:

partition key: product_id

product 42, version 182
product 42, version 183
product 42, version 184

All versions for product 42 reach the same partition, so a consumer can apply them in sequence. Product 99 may be processed on another partition at the same time. This preserves the order that matters without forcing every merchant’s changes through one global line.

The key is an invariant decision, not a load-balancing afterthought. Random partitioning would spread traffic evenly but could let version 184 reach a search worker before version 183. Keying only by tenant would preserve far more order than needed and could place a large tenant on one hot partition. Keying by product keeps the ordering scope narrow, although one exceptionally hot product can still limit its own parallelism.

A consumer group is one logical subscription. Several search workers in the same group divide its partitions; they do not each receive a copy of every event. Search and analytics belong to different groups because they are independent subscriptions. Adding a tenth search worker cannot increase parallelism if the search group owns only six busy partitions, and adding partitions later may change key placement and operational balancing. Capacity begins with the partition model.

The offset makes the crash visible

For each partition, a log consumer tracks an offset: the position from which it will resume. Consider the search consumer handling evt-7f3 at offset 812. There are two relevant durable facts:

search document for product 42 is at version 184
search group has committed its position beyond offset 812

Their order determines the failure mode. If the consumer commits its offset first and crashes before updating search, version 184 is skipped. If it updates search first and crashes before committing the offset, the event is read again.

The safer default is therefore:

read record
perform durable work
commit offset or acknowledge

This chooses possible duplicate processing over silent loss. The search write then needs a guard such as “replace the document only when the incoming source version is newer.” Reprocessing version 184 becomes a no-op, and a late version 183 cannot overwrite it.

For a relational consumer, the same idea can be made explicit in one local transaction:

begin transaction
  insert processed_event(event_id = 'evt-7f3') if absent
  apply state change only when that insert succeeds
commit
commit offset 812

If the process dies after the database commit but before the offset commit, the unique event ID absorbs the replay. The deduplication record must live at least as long as the event can be redelivered or replayed. A seven-day deduplication window cannot protect a financial command that may be replayed from six months of retained history.

The phrase “exactly once” does not remove these boundaries. A stream platform may atomically commit an input position and records written to its own transactional sink. That guarantee stops where an email provider, payment API, search service, or webhook receiver begins unless the external operation also has stable idempotency.

The email job can use a key derived from the event and template:

notification key = evt-7f3 + price-change-v2 + merchant-91

The notification service records that key and passes it to the provider when the provider supports idempotent requests. If the network fails after the provider accepted the email, retrying with the same key recovers the stored result instead of sending a second message. If the external system offers no such contract, the design must admit that duplicate effects remain possible and choose reconciliation or a different boundary.

This explains the merchant’s two emails. “At least once” described transport recovery, while the implementation treated delivery as a once-only business effect. Delivery semantics apply at a named boundary; end-to-end behavior is built from the boundaries that follow.

Backlog age is part of the product contract

Now suppose a bulk importer changes two million products. The log accepts the records faster than the search group can index them. Durable buffering protects the producer from a brief consumer slowdown, but it does not create capacity. The product is accumulating old work.

Queue depth or raw offset lag gives only a rough count. The consequential measure is usually age: how old is the earliest price change not yet visible in search? Ten thousand tiny records may drain quickly; one hundred expensive records may violate the same freshness promise. Useful evidence includes oldest unprocessed age, records and bytes arriving per second, drain rate, retry rate, worker saturation, and downstream latency.

Backpressure is the path by which that evidence changes behavior. For the bulk importer, the service might reduce import concurrency, reject new bulk jobs, or admit them as scheduled work with an honest completion estimate. Search workers may scale until the search cluster, not the worker count, becomes the limit. Low-priority recommendation refreshes may pause so customer-visible search can recover. A very large tenant may need its own queue or rate budget so it cannot age everyone else’s work.

The response must meet the caller contract. An interactive edit may still be safe because the product database owns the current price, while the UI shows “search update pending.” A fraud decision that must complete before checkout cannot be allowed to sit invisibly behind the same backlog. “The broker is healthy” is irrelevant when the oldest required decision is twenty minutes late.

Retries also consume capacity. Permanent schema errors retried immediately can crowd out healthy records and amplify a downstream outage. Retry policy should separate transient failures from permanent ones, use delay and jitter, and stop after a bounded number of attempts or a bounded age.

A dead letter is a repair obligation

Assume the search consumer deploys code that expects currency, but an old producer emits a retained event without it. Retrying the same bytes will not make the field appear. After the bounded retry policy is exhausted, the record moves to a dead-letter lane so the main partition can make progress.

That lane needs enough evidence to support a decision: the original record, partition and offset, event identity, schema version, consumer version, failure reason, attempt history, and timestamps. It also needs an owner. The operator must be able to repair and replay the record, deliberately discard it, or compensate for the missed effect.

A dead-letter queue with no alert, age target, inspection tool, or replay path is deferred data loss. Its depth can remain small while one poisoned price change causes a costly inconsistency.

Compatibility prevents many dead letters before they exist. Event records outlive a deploy, so producers and consumers cannot assume they upgrade in lockstep. Additive fields need defaults or optional handling; breaking changes need a versioned transition; consumers should tolerate retained records for as long as the log promises to keep them. A log can rebuild state only if old facts remain decodable.

Replay separates rebuildable state from irreversible effects

Search is a derived view. If a bad index mapping corrupts it, the team can create a new index and replay product events from a chosen offset or snapshot. The version guard makes repeated writes safe. The rebuild can run beside live consumption, compare counts and sampled documents, then switch the read alias when the new index is ready.

Notifications have a different posture. Replaying six months of product events should not send six months of email again. The notification consumer must distinguish live delivery from a rebuild, retain durable notification identities, or stay detached from replay altogether. Payment capture and customer webhooks deserve the same caution.

A credible replay plan names:

• the retained source and starting position;
• the consumer version and schema compatibility rules;
• whether output is rebuilt in place or into a new version;
• which external effects are disabled or idempotently guarded;
• how replay is throttled so live work is not starved;
• how the result is compared, cut over, and rolled back.

Replay is not merely moving an offset backward. It is a production write path through old data. The fact that the source is historical does not make its effects harmless.

Stream time asks when the event happened

The product pipeline mostly cares about entity order, but many stream processors compute over time. Imagine a delivery marketplace whose courier phones send pickup_completed events. A phone may be offline for ten minutes and upload the event after reconnecting.

Processing time records when the analytics worker saw the event. Event time records when the pickup happened on the device or authoritative source. A five-minute operations graph based on processing time answers “what reached the platform during this interval?” A service-level report based on event time tries to answer “what happened during this interval?” Those are different questions when records arrive late.

A fixed window might count pickups in [09:00, 09:05). A sliding window might calculate the last fifteen minutes every minute. A session window might group one courier’s activity until a long gap ends the session. Each window still needs a policy for incomplete knowledge.

A watermark expresses how far the processor believes event time has advanced enough to emit or close results. It is a progress rule, not proof that no older event will ever arrive. Waiting longer improves completeness but delays results and retains more state. Advancing quickly produces fresher output but creates more late records.

For records that arrive after the watermark, the system may update the old aggregate and emit a correction, route them to a side output for repair, or drop them after a documented threshold. A dashboard can show provisional figures and later corrections; a finalized settlement report may need a longer close period and reconciliation against the source of truth.

Deduplication also has a time horizon. Telemetry may remember event IDs for a bounded window because a stray duplicate changes an approximate aggregate slightly. Money movement needs a durable command record because the consequence of one duplicate does not expire when an in-memory window closes.

Follow one record before naming the architecture

In a design interview, “put a queue here” is only the start of the answer. Take one consequential record and follow it across time:

business commit
  -> publication boundary
  -> partition and ordering key
  -> consumer group and position
  -> durable effect and idempotency guard
  -> acknowledgment
  -> retry, dead-letter, or replay path

At each step, say what the caller can observe and what a crash can leave behind. Then apply load: which age or lag breaches the product promise, and how does that signal slow admission or degrade the feature? If the stream aggregates over time, state which clock answers the product’s question and what happens to late events.

The price-change incident is now explainable without blaming “eventual consistency.” Search stayed at 100 because its group had not safely advanced through version 184. The duplicate email crossed an external boundary without a durable idempotency key. The dashboard froze because its backlog grew with no age-based backpressure or repair owner.

Asynchronous systems are dependable when delayed work remains visible work. The next chapter widens the view: once several machines can disagree about time, ownership, and success, the same questions become the foundations of distributed-system correctness.