Messaging, Logs, and Delivery Semantics

The broker dashboard was green again at 09:17. Publish latency had returned to normal, all twelve partitions had leaders, and the fulfillment consumer group had a full assignment. Yet the oldest unreserved order grew older every minute.

The recovery command was doing exactly what its operator requested: consume the backlog as fast as possible. It raised broker fetch throughput from 2,400 to 9,000 events per second. The inventory service could safely commit about 3,300 reservations per second under this workload. Its write pool saturated, p99 effect latency crossed the consumer lease limit, deliveries expired, and the broker redelivered them. Input throughput looked healthy because the same work crossed the transport boundary repeatedly. Useful completion fell below the live arrival rate.

This is the central trap in durable messaging: transport progress is not effect progress. A message can be accepted, replicated, fetched, acknowledged, retained, replayed, or compacted without the intended business result becoming true. Conversely, the result can become true just before a consumer crashes and therefore be followed by a legitimate redelivery.

A durable asynchronous edge is a small distributed system. It has storage, ownership transfer, an ordering domain, admission and flow control, consumer state, failure detection, and a recovery workload. Treating it as an arrow hides the very states that determine correctness and performance.

The controlling rule is simple: define the durable effect and deduplication boundary before claiming a delivery guarantee; transport acknowledgment alone is not business completion.

Four transport forms answer different questions

Queue, publish/subscribe, append-only log, and notification are not interchangeable names for “send later.” They define who owns a record, how many independent readers may observe it, and whether history is a first-class resource.

A queue transfers work ownership. A log preserves a sequence from which many reader groups derive their own progress. Pub/sub describes fan-out but does not by itself say whether the implementation stores a queue per subscription, retains a shared log, or drops delivery for disconnected subscribers. A notification deliberately carries less authority: it prompts a read from an authoritative store.

These semantics can be layered. A retained log may expose consumer groups that behave like competing queues. A pub/sub service may materialize one queue per subscriber. The architecture review should therefore record observable contracts—retention, progress identity, redelivery, ordering, and overload—not infer them from a product label.

A message moves through several owners

Name the participants before choosing settings:

• The producer creates a record and decides its identity, key, schema, priority, and retry behavior.
• A broker accepts, stores, indexes, replicates, and serves records according to a declared durability policy.
• A partition is an ordering and storage unit with finite throughput, bytes, and leadership.
• A consumer fetches or receives records and attempts application work.
• A consumer group coordinates a set of consumers so each partition or delivery has an active owner under that group’s rules.
• The effect store or external system makes the intended result durable.

For Mercury fulfillment, an accepted order produces FulfillmentRequested. One consumer group reserves inventory; another prepares customer communications; a third builds an operational projection. Their progress and failure policies are independent. The email projection cannot advance or block the inventory group’s offset. The inventory group may use twelve consumers for twelve partitions, but a thirteenth consumer adds no partition parallelism under a one-owner-per-partition model.

Broker replication protects broker state. It does not make the producer’s source mutation atomic with publication, nor the consumer’s effect atomic with offset advancement. Those are application boundaries. Chapter 32 repairs them with outbox, inbox, state, and sink protocols.

Ordering is scoped by the invariant

“Messages are ordered” is incomplete. Ask four questions:

1. Ordered where: one producer session, one queue, one partition, one key, or a whole topic?
2. Ordered for whom: one consumer, one group, or all subscribers?
3. Ordered through what failures: retry, leader movement, rebalance, timeout, and replay?
4. Ordered by which identity: creation time, producer sequence, partition position, aggregate version, or business causality?

The partition key converts an invariant into a serialization boundary. If inventory reservations for one order must observe Requested(v3) before Cancelled(v4), order_id is a defensible key. A global topic order would serialize unrelated orders and limit throughput for no correctness gain. A tenant_id key is too broad if only per-order transitions require order; one large tenant can monopolize a partition. A random key spreads load but destroys the required per-order sequence.

Ordering does not resolve duplicates or stale facts. A redelivered Requested(v3) may arrive after the effect for Cancelled(v4) has committed. The consumer needs an aggregate version or state transition rule, not merely the fact that each partition has a position. When an entity may change keys, define the handoff: a new key can place related records in different ordering domains.

Partition count is also contractual. More partitions can increase parallelism and reduce average ownership per consumer, but they increase leaders, files, indexes, replication traffic, recovery assignments, open requests, and coordination state. Changing the count can remap new keys while old records remain where they were. Size partitions from required keyed parallelism, per-partition byte and request limits, recovery time, and operational overhead—not from a generic “more scales better” rule.

Delivery names are outcomes over failure windows

Delivery semantics describe what may happen across the producer, broker, consumer, and effect boundaries.

At-most-once permits loss but prevents transport redelivery by advancing progress before processing or by not retrying ambiguous publication. It is reasonable for expendable telemetry or invalidation hints when the authoritative state can repair loss. It is unsafe for an irreplaceable reservation request.

At-least-once retains or redelivers work until an acknowledgment or progress update. It trades possible duplicates for reduced loss across recognized failure windows. It requires the consumer effect to tolerate repeat attempts.

Effectively-once is an application outcome: repeated delivery attempts converge to one intended durable effect within a named identity and retention boundary. It usually combines at-least-once transport with an inbox uniqueness constraint, versioned state transition, idempotent sink operation, or transactional write of effect plus consumed identity.

The word exactly invites a much broader claim: one delivery, one process execution, one database change, one charge, one email, or one externally visible result? The transport cannot answer that without the effect boundary. Keep the claim narrow.

The delivery-state timeline

Consider one reservation event evt-7, keyed by order o-1842:

The critical window is between steps 4 and 5. It is not an anomaly; it is the normal reason at-least-once consumers must be idempotent. Reverse the steps and the window changes from duplicate attempt to lost effect.

Producer acknowledgments deserve the same precision. “Acked by the broker” might mean received in leader memory, appended locally, persisted, or replicated to a quorum according to one product’s configuration. Record the exact durability point, failure assumptions, minimum replica condition, timeout behavior, and response to an unknown outcome. A socket write is not acceptance. An acceptance acknowledgment is not consumer processing. A consumer acknowledgment is not automatically a business effect unless the application makes that its contract.

Consumer progress is a correctness boundary

Queue-style systems often expose a lease or visibility timeout. A delivery becomes temporarily unavailable to other workers. If the worker acknowledges it, the broker can delete or retire it; if the lease expires or the connection fails, the message becomes eligible for redelivery.

Log-style systems expose a position or offset. Fetching position 9,100 does not mean positions through 9,100 are durably applied. The group should advance its committed position only through the highest contiguous effect known to be safe. Parallel processing within a partition needs a completion frontier; committing past one slow record can skip it after failure.

The lease must exceed normal effect time with margin for tail behavior, pauses, and acknowledgment delivery, but an enormous lease turns a crashed worker into a long outage. Heartbeat or lease extension can help long work, provided extension stops when ownership is uncertain. Cancellation should stop local work that can no longer commit safely.

Flow control bounds the number and bytes of unacknowledged deliveries. Prefetch of 1 minimizes duplicate in-flight work but can waste capacity when round trips dominate. Unlimited prefetch can move the broker’s queue into consumer heap, erase fairness, extend redelivery after failure, and let a slow consumer retain work needed elsewhere. Tune it from concurrent effect capacity and byte size:

in_flight_bytes ≈ consumers × prefetch_per_consumer × high_percentile_event_bytes

Use the high-percentile expanded size, not only encoded bytes. A 50 KiB compressed record may allocate a 4 MiB object graph after decoding.

Batching and log layout move the bottleneck

Messaging has fixed per-request, per-record, per-batch, and per-byte costs. Batching amortizes system calls, network frames, checksums, compression dictionaries, replication requests, and storage index work. It also holds the oldest event until a count, byte, or time trigger fires.

Mercury’s teaching model batches up to 400 events or 8 ms. At an average raw event size of 1,500 bytes, a full batch is 600,000 bytes. A modeled 2.4:1 compression ratio makes it 250,000 bytes. These are modeled values, not a codec benchmark. Under low traffic the 8 ms timer bounds batching delay; under high traffic the count or byte cap bounds allocation and frame size.

Choose all three triggers:

• count limits per-record metadata and processing loops;
• bytes limits memory, network frame, decompression, and broker request exposure;
• time bounds the oldest record’s batching latency.

Compression trades CPU and buffering for fewer network and storage bytes. Measure representative entropy, producer and consumer CPU, decompressed size, tail latency, and failure behavior. Large batches improve sequential writes but can dominate a partition, extend retries, and create head-of-line blocking for small urgent records.

Append-only logs commonly group records into segments with sparse indexes. Larger segments and sequential I/O can improve throughput; they also change retention granularity, recovery scanning, compaction work, page-cache behavior, and deletion timing. The application contract should not depend on a segment disappearing at an exact event timestamp unless the broker explicitly provides that guarantee.

Retention, compaction, replay, and backfill serve different jobs

Retention keeps records by time, bytes, or both so consumers can recover or replay. It is a storage horizon, not a promise that every group has processed the record and not a replacement for application deduplication.

Compaction retains selected records, often the latest value per key, rather than every historical transition. It is useful for rebuilding current keyed state. It is not an immutable audit log: intermediate facts may disappear, tombstones have lifecycle rules, and a bad key collapses unrelated entities.

Replay asks an existing consumer to apply retained history again, perhaps after code repair or state loss. Backfill introduces a historical population into a computation, often with a different schema, distribution, event-time range, and resource demand. Both are production workloads. Name their identity, range, code version, schema reader, destination state, rate limit, cancellation, checkpoints, and validation.

Never reset a live group’s offset as an improvisational replay plan. A separate replay group and isolated destination allow comparison before cutover. If replay must update the live effect, its deduplication window and semantic version must cover the historical horizon; otherwise old events may look new.

Lag is work divided by spare service, not a record counter

Lag has at least three useful units:

• records describe count but hide size and service-demand skew;
• bytes expose transfer, storage, and decode work;
• age measures how stale the oldest required effect is relative to its objective.

For an incoming rate (\lambda) and sustainable effect completion rate (\mu), backlog changes at (\lambda-\mu). When (\mu>\lambda), ideal drain time for backlog (B) is:

drain_time = B / (μ - λ)

The denominator is spare useful effect capacity, not broker fetch throughput. Measure it under the recovery record mix, cache state, schema versions, sink limits, and error rate.

Applied replay plan: recover without restarting the incident

Mercury receives 2,400 fulfillment events per second at a modeled average of 1,500 bytes. A 45-minute consumer outage creates:

backlog = 2,400 events/s × 2,700 s = 6,480,000 events
raw bytes = 6,480,000 × 1,500 bytes = 9.72 GB

The downstream inventory effect has a tested safe envelope of 3,300 events/s for this mix. Mercury reserves 2,400 events/s for live arrivals, 300 events/s of headroom for variance and control work, and only 600 events/s for replay. The modeled replay time is:

6,480,000 events / 600 events/s = 10,800 s = 3 hours

The plan uses a separate replay group, explicit partition ranges, a 600 events/s global token bucket plus per-partition limits, and an abort condition when live queue age, inventory p99, write-pool saturation, or error rate crosses its envelope. It records applied event identities and compares reservation counts and invariant violations before advancing a range. It can pause without losing its checkpoint.

The fastest safe recovery is not maximum broker throughput. It is the highest replay rate that preserves live objectives, correctness, and recovery headroom end to end.

Poison records need diagnosis, not a dark queue

A transient failure may succeed later: a bounded sink overload, leader movement, or temporary dependency loss. A permanent record failure will not improve with time: invalid schema, impossible state transition, missing required identity, or data outside the consumer’s supported version. A consumer defect may affect a class of otherwise valid records. Treating all three as “retry” creates a poison loop.

Use bounded attempts with backoff and jitter inside the event’s useful deadline. Track attempt count and first-failure time independently of broker delivery metadata if redrive creates a new message. After the bound, write a quarantine record containing the original immutable bytes or a protected reference, topic/partition/position or delivery identity, schema and code version, error class and hash, attempt history, correlation and causation identity, and replay authorization state.

A dead-letter queue is a transport destination, not resolution. It needs an owner, access control, retention, alarms, inspection tooling, correction policy, and a route back through normal validation. Blindly replaying it after a deployment can reapply effects whose original attempt succeeded ambiguously.

Quarantine also needs capacity isolation. If poison traffic floods the same broker, database, or on-call path as live processing, the safety mechanism becomes another amplifier. Alert on rate and age, not merely queue depth.

Hot partitions and uneven consumers defeat aggregate capacity

Aggregate arrival below aggregate service does not prove stability. With twelve partitions, a tenant_id key sends a tenant producing 35% of Mercury’s 2,400 events/s to one partition: 840 events/s. If one partition consumer can apply 300 events/s for that tenant’s mix, lag grows by 540 events/s while other consumers are idle.

The correction begins with the invariant. Fulfillment order transitions require per-order order, not per-tenant order. Keying by order_id spreads the tenant while preserving the real ordering boundary. If one order can itself be hot, use a different domain model—sub-reservations with a versioned aggregate, a directory, or explicitly commutative updates—rather than salting a key and hoping order survives.

Uneven service demand can create the same signature even with equal record counts. Large orders, cold product lookups, one schema version, or one geography may cost more. Track per-partition records, bytes, effect time, error class, oldest age, and key concentration. Reassignment moves ownership; it does not remove a hot key. Adding consumers beyond partitions does nothing. Adding partitions may help only future key placement and can complicate ordering.

Broker and group recovery are visible pauses

Broker failure and leader movement interrupt reads or writes while a new owner is chosen, state catches up, and clients refresh metadata. A durability configuration may keep committed records safe yet still violate a latency or availability objective during election and recovery. Test the actual cluster, replica placement, minimum acknowledgment policy, storage saturation, and client timeout/retry interaction.

Consumer-group rebalance revokes and reassigns partitions. A stop-the-world protocol can pause every member; a cooperative protocol may move fewer assignments but still requires consumers to stop committing revoked work safely. Long processing, heartbeat starvation, deployment churn, and unstable members can create rebalance loops. Measure revoke-to-resume duration, uncommitted work, duplicate attempts, cache warm-up, and lag added per partition.

Recovery has cold costs: segment/page-cache reads, decompression, schema fetches, connection creation, state restoration, and downstream cache misses. Reserve capacity for them. A normal-load benchmark with warm state does not establish recovery throughput.

The Mercury fulfillment messaging contract

interaction: FulfillmentRequested v3
business effect: one inventory reservation for (order_id, reservation_version)
source authority: order service accepted-order transaction

transport form: retained partitioned log
producer identity: stable event_id; retries preserve bytes, key, and identity
key / ordering: order_id; increasing reservation_version within one key
partition count/change rule: 12; expansion requires ordering and remap review

publish acceptance: declared broker committed-record rule; unknown ack is retried
delivery: at least once to inventory-reservation-v3 group
consumer progress: offset advances after inbox + reservation commit
deduplication: unique (consumer_name, event_id) plus reservation version invariant
lease/session: bound to tested effect tail; revoked partitions stop commits

batch bounds: max 400 events, 600 kB raw modeled cap, or 8 ms
in-flight bounds: per-consumer count and expanded-byte budget
retention: live recovery and approved replay horizon; separately monitored
poison: 5 bounded attempts by class, then protected quarantine with owner

objectives: live oldest age, effect completion rate, publish acceptance latency
recovery: 600 events/s replay cap; 300 events/s downstream headroom
abort signals: live age, sink p99, saturation, error rate, invariant violation
fault tests: lost publish ack, consumer crash after effect, leader move,
             member rebalance, poison burst, hot key, slow sink, replay pause

The contract does not say “exactly once.” It says what may repeat, what must be unique, where uniqueness is enforced, how long the evidence survives, and what recovery may consume.

Design and operating drills

Choose the form. For a cache invalidation, image-rendering job, audit fact, and product-price projection, decide whether the consumer needs exclusive work ownership, independent subscriptions, retained replayable positions, or only a repairable notification. State what happens when a reader is offline longer than retention.

Mark the crash windows. Draw producer source commit, send, broker acceptance, publish acknowledgment, consumer delivery, effect commit, and consumer acknowledgment. Crash before and after every edge. For each window, classify loss, duplicate attempt, ambiguous outcome, redelivery delay, and repair authority.

Budget recovery. Reproduce the 6.48-million-event, 9.72-GB backlog and three-hour governed replay. Replace average record size with p95 bytes and average effect time with the recovery mix. Reduce one downstream limit by 30%. Find the replay rate that keeps at least the declared headroom.

Break the group. Kill a broker leader, pause one consumer beyond its session limit, deploy consumers one by one, inject one partition at 840 events/s, and slow the effect store. Verify useful effect rate, oldest age, rebalance duration, duplicate attempts, and memory—not only fetch rate.

Durable messaging rules

1. Choose queue, pub/sub, log, or notification semantics from ownership and history needs, not fashion.
2. Scope ordering to the smallest invariant-preserving key and treat partitions as finite ordering/storage units.
3. Distinguish producer acceptance, consumer delivery, consumer progress, and durable application effect.
4. Use at-least-once transport only with a named duplicate strategy; call an outcome effectively once only within its enforcement and retention boundary.
5. Bound batches by count, bytes, and time; bound in-flight work by expanded bytes and effect concurrency.
6. Measure lag in records, bytes, age, and service demand; calculate recovery from spare effect capacity.
7. Isolate and govern replay/backfill as production workloads with checkpoints and abort rules.
8. Quarantine poison records with evidence and ownership; a dead-letter destination is not resolution.
9. Diagnose hot keys and costly record classes per partition; aggregate throughput can hide an unstable shard.
10. Include leader movement, rebalance, cold state, redelivery, and downstream recovery in the objective.

The next problem begins after transport succeeds. A record may be durably accepted and redelivered exactly as specified while the source database, processor state, and external side effect disagree. Chapter 32 places atomicity and identity around those effects and gives “exactly once” a boundary narrow enough to test.

Evidence and transfer limits

• Apache Kafka’s current design documentation describes partitioned logs, consumer groups, committed records, idempotent publication, transactions, and the scope of read-process-write guarantees. Those product mechanisms do not prove Mercury’s inventory outcome or transfer unchanged to another broker.
• RabbitMQ’s reliability guide, publisher guide, and consumer acknowledgment and publisher confirm guide distinguish publisher acceptance from consumer acknowledgments, redelivery, and in-flight flow control. Queue-specific behavior should be verified against the deployed version and queue type.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/messaging-delivery/ reproduces the 6.48-million-event and 9.72-GB outage backlog, three-hour 600-events/s replay, 3,300-events/s capacity allocation, 840-events/s hot-partition arrival, 540-events/s lag growth, and 600-kB-to-250-kB modeled batch compression. These are deterministic teaching calculations, not observed broker or sink performance.