Asynchronous Execution, Queues, and Backpressure

At 09:17:04.120, Pulsepipe accepts event p-8841 from a producer. At 09:17:04.126, the ingestion service returns 202 Accepted after a durable append. At 09:17:04.131, a notification is emitted. The transform worker does not start the event until 09:48:12.504. It completes at 09:48:12.517. The producer’s business deadline expired at 09:22, but neither the durable record nor its eventual effect disappeared.

All five timestamps can be correct. They answer different questions:

The incident began because the dashboard retained only “ingest latency” and “consumer throughput.” Ingest p99 remained below 18 ms. Consumer throughput remained near 420 records/s. Offered load had risen to 650 records/s, so the queue accumulated 230 records/s. Fast acceptance made the producer experience look healthy while completion receded by minutes.

The design decision is: which component owns accepted work, what completion means, and how the system prevents accepted demand from exceeding bounded downstream capacity?

Asynchrony changes the critical path; it does not erase it

A synchronous call places the caller’s wait, deadline, and outcome on one visible path:

request ── validate ── transform ── durable effect ── response
          <-------------- caller deadline ----------->

An asynchronous call splits that path into at least two contracts:

submission path: request ── validate ── persist/transfer ownership ── accepted
completion path:                     queue ── start ── effect ── report outcome
                                       <---- completion objective ---->

The submission path may become shorter, but the user journey is not faster unless the user no longer needs to wait for completion. An email notification, search-index refresh, thumbnail, or audit export may tolerate delayed completion. Inventory reservation, payment authorization, or a read-after-write response usually cannot be detached without changing the product and correctness contract.

Ask two counterfactuals. If the caller disconnects immediately after acceptance, must the work still complete? If the queue is unavailable, can the caller proceed truthfully? A “yes” to the first requires transferred ownership. A “yes” to the second may mean the work is optional, reconstructible, or not actually accepted. Returning success while retaining ownership only in volatile caller memory is not decoupling.

An async boundary earns its cost when it isolates independent timing, absorbs a bounded burst, permits a different concurrency model, or allows replay and recovery. It is suspect when it hides a capacity deficit, fragments one atomic invariant, or forces users to poll indefinitely for a result they need now.

Queue families encode different failure boundaries

The data structure name matters less than the owner and loss model.

An in-process queue transfers work among tasks in one process. It can be extremely cheap and can enforce memory and concurrency bounds. Process termination normally loses queued work unless another persistence mechanism owns it. Use it for reconstructible work, request-scoped pipelines, or handoffs whose failure is coupled to the process.

A channel combines a send/receive interface with synchronization and, often, a capacity. A rendezvous channel with zero buffering transfers only when sender and receiver meet; a buffered channel permits limited timing independence. Closing, sender loss, receiver loss, and cancellation semantics are library-specific. A channel is not durable merely because its API is asynchronous.

A mailbox attaches a queue to a logical owner such as an actor or shard. It can make mutation serial and ownership clear, extending Chapter 18’s owned-state design. The mailbox still needs a bound, priority rule, supervision policy, and restart semantics. If the owner restarts, “mailbox survives” and “effects are replay-safe” are separate claims.

A durable queue or append-only log can transfer ownership across process and host failure, subject to its acknowledgement and replication boundary. Durability adds serialization, writes, acknowledgements, retention, replay, consumer state, and repair. It does not automatically provide unique effects, global order, or infinite absorption.

Choose the least durable boundary that satisfies the consequence of loss. Reconstructible cache warming can use a bounded volatile queue. An accepted financial command needs a durable identity and recovery path. Treating every background task as durable creates storage and replay obligations; treating every task as volatile turns restarts into silent data loss.

Consumption style controls where demand becomes visible

Push and pull are not opposites so much as locations for scheduling authority.

With push, the producer or dispatcher initiates delivery. Push can reduce detection delay but can overrun a slow receiver unless delivery is gated by credits, concurrency limits, or explicit rejection. A callback per item is still push even when implemented by a thread pool.

With pull, a consumer requests work when it has capacity. Pull exposes receiver readiness and naturally supports competing consumers. Aggressive prefetch can turn a bounded broker queue into large hidden client buffers, however. Those prefetched records are waiting even if the broker no longer counts them as available.

With polling, consumers inspect for work periodically. The interval trades empty work and backend load against detection latency. Synchronized pollers can create bursts. Add jitter, long polling, or notifications when empty-poll cost is material.

With notification plus pull, a notification says work may exist; the consumer then retrieves within its capacity. Notifications may be duplicated, delayed, or coalesced, so the queue remains authoritative. This hybrid is often a useful separation of low-latency wake-up from bounded transfer.

Whatever the API, trace the actual buffer chain: producer memory, client library, kernel socket, broker ingress, partition log, consumer prefetch, worker executor, sink pool. A bound at one layer does not bound the sum. A send() that returns after copying into a client buffer has not proved broker acceptance.

Capacity has four dimensions: count, bytes, cost, and age

A queue length of 10,000 could mean 10 MiB of cheap records or 20 GiB of expensive records. Count alone cannot bound memory or recovery time. For each waiting point, specify:

• maximum records or tasks;
• maximum serialized and resident bytes;
• maximum estimated downstream service demand; and
• maximum oldest age for each service class.

Age is often the first user-relevant saturation signal. Depth can remain stable while larger or slower records increase waiting. Depth can grow while age remains acceptable during a brief burst. Track enqueue-to-start wait as a distribution, oldest eligible age, and age by partition, tenant, and priority. Do not average away a stalled partition.

Bounds require outcomes. When full or too old, a producer can block within a deadline, receive an explicit retryable rejection, spill to a separately bounded durable tier, degrade optional work, replace an older coalescible item, or drop according to a declared policy. “Keep accepting” is not an overload policy.

Drop policy follows semantics:

• drop newest preserves admitted backlog but rejects current demand;
• drop oldest favors freshness only when old work is safely obsolete;
• coalesce by key replaces superseded state updates but not independent events;
• sample is valid for telemetry whose statistical contract permits it;
• dead-letter or quarantine isolates poison records, with its own bound and owner;
• reject/defer at source preserves downstream capacity and makes loss visible.

Priority adds another queue. Strict priority can starve background work forever. Weighted service, reserved capacity, aging, or deadline scheduling can bound that risk. Measure the oldest age and completion share of every class rather than claiming fairness from a scheduler name.

Ordering belongs to a scope, not to the whole architecture

Ordering is expensive when stated too broadly. Define the smallest key whose operations must be observed in order: one account, document, session, partition, or workflow instance. Route that key to a stable partition or owner. Parallel consumers can then process independent keys without inventing a global order.

Even a FIFO queue may not deliver FIFO effects. A consumer can fail after starting the first record, a later record can finish first on another worker, a retry can re-enter behind new work, or the sink can commit in a different order. Distinguish enqueue order, delivery order, start order, completion order, and externally visible effect order.

Partition affinity preserves a local sequence only while routing and ownership epochs are correct. Rebalancing introduces pauses and potentially overlapping owners. Fence an old owner before a new one applies effects. Carry a sequence, version, or predecessor when the sink must reject stale operations. If independent tasks can complete out of order, make that freedom explicit instead of paying for ordering no invariant needs.

Competing consumers improve capacity when work is interchangeable. Work stealing can reduce idle time and long tails by allowing an idle worker to take tasks from another worker’s deque. It can also damage cache or NUMA locality, violate tenant reservations, and pull a long task ahead of older short tasks. Evaluate completions, age, migrations, stolen-work cost, and fairness under the actual duration distribution.

Backpressure is a protocol, not a rising metric

Backpressure is a downstream constraint communicated upstream soon enough to change admission or production. A queue-depth alert after the producer has already deposited hours of work is observation, not control.

Credit-based flow control makes permission finite. A receiver advertises or grants (C) units—records, bytes, or another bounded resource. Sending consumes credits; processing or releasing buffers returns them. The unit must match the resource being protected. Record credits fail when record sizes vary by three orders of magnitude. Byte credits fail when CPU cost varies independently. Some pipelines need both byte and work-cost budgets.

The protocol must define lost and duplicated credit messages, reconnect, epoch changes, and what consumes capacity. Otherwise a retry can mint credits twice or a restart can forget in-flight work. Credits are usually scoped to one hop. End-to-end overload control requires every stage to translate its downstream capacity into an upstream admission decision.

The HTTP/2 specification provides a concrete, limited example: stream and connection windows are receiver-controlled octet credits on one hop. It explicitly does not define the receiver’s window algorithm, and its window is not an application queue or business-work budget. The lesson is the explicit grant/consume/replenish lifecycle, not that HTTP/2 solves service backpressure.

Reactive Streams similarly standardizes an asynchronous demand signal and requires demand not to be exceeded under its scope. It does not decide durable ownership, business deadlines, partition order, or replay for an application. A library’s backpressure contract is one layer of the end-to-end design.

Across three stages, credit propagation should follow the constrained sink:

Stage C releases 200 effect slots
        ↓ translated to bytes and work estimates
Stage B may pull at most the work fitting those slots
        ↓ remaining Queue AB credits
Stage A accepts, defers, or rejects before local bounds are exceeded

If Stage B continues pulling after Stage C stops granting credit, B merely moves the backlog into its own memory. If Stage A cannot slow an external producer, it must reject or persist within a finite envelope. Backpressure cannot force an uncooperative source to stop; admission is the terminal control.

Cancellation is a lifecycle request, not proof of reversal

A caller deadline bounds how long the caller is willing to wait. A timeout is the caller’s observed outcome. Cancellation asks downstream work to stop. None proves that an accepted effect did not occur.

Model each item with a durable operation identity and states appropriate to the effect:

offered -> accepted -> queued -> started -> committed -> reported
              |          |         |
              +-> rejected          +-> cancel requested
                         +-> expired     -> stopped before commit
                                          or committed despite request

Before start, a consumer can discard expired work if the contract permits. After start, cooperative cancellation needs safe checkpoints and cleanup. After commit, cancellation may require compensation rather than rollback. Return a status keyed by operation identity so the caller can resolve ambiguous completion without blind duplication.

Stale work should not silently consume peak capacity. Carry the original deadline or freshness deadline with the item. Check it before expensive service and before external effects. A tombstone can record that a key or operation was cancelled or superseded so delayed or replayed work cannot resurrect it. Tombstones need retention long enough to cover maximum replay and duplication windows; premature deletion reopens the race.

Coalescing is valid for state-setting operations such as “desired thumbnail version is 18,” where version 17 is superseded. It is invalid for additive events such as “debit 10” and “debit 20.” The queue must know the algebra of its work, not merely share a key.

Persistence selects a latency and recovery spectrum

Queue acknowledgement can mean copied to process memory, appended to an operating-system buffer, flushed to local durable media, replicated to another failure domain, or committed by a quorum. Name the boundary and failure class. “Durable” without both is incomplete.

Stronger acknowledgement usually adds latency, write amplification, and coordination. Weaker acknowledgement shifts loss risk to reconstruction or the user. Batch append and group commit can amortize durability cost, but then queueing and partial-batch deadlines enter the contract; Chapter 20 develops that trade.

Persistence also creates a recovery workload. Retention must cover the stated replay horizon. Consumer progress must be recoverable and reconciled with effects. Poison records need bounded quarantine. Replaying a two-hour backlog at maximum speed can overload the sink, invalidate caches, or starve foreground traffic. Reserve or rate-limit recovery capacity and expose estimated drain time.

The normal, saturated, draining, and failed states need explicit transitions:

The async interface contract is a product and operations artifact

Use a contract that another team can test without knowing the implementation:

operation: enrich-event
acceptance:
  means: validated and durably appended in one zone
  response: operation_id plus accepted_at
completion:
  means: versioned enrichment committed to the authoritative sink
  objective: 99% of accepted priority-A events within 5 minutes
status:
  outcomes: pending | running | succeeded | rejected | expired | failed
ownership:
  authoritative_record: pulsepipe ingress log
ordering:
  scope: source_id
  rule: monotonically increasing source_sequence
bounds:
  records: 1800000
  logical_bytes: 2520000000
  oldest_age_seconds_before_guarded: 2700
backpressure:
  unit: records plus estimated transform-milliseconds
  terminal_action: explicit retryable rejection with retry_after
cancellation:
  before_start: mark expired and tombstone operation_id
  after_start: cooperative stop before sink commit; otherwise report committed
recovery:
  rule: fence consumer epoch, reconcile sink version, replay by operation_id
observability:
  timestamps: offered, accepted, enqueued, started, committed, reported
  context: trace_id, operation_id, source_id, source_sequence, consumer_epoch

Security and privacy are part of the contract. Queues often retain payloads longer and replicate them more widely than request memory. Apply authorization at submission and effect boundaries when authority can change during delay. Encrypt and classify persisted payloads, constrain operators who can inspect them, propagate deletion, and avoid putting sensitive data into diagnostic attributes.

Worked overload model: three hours behind

Pulsepipe receives a modeled 650 records/s while a degraded transform stage completes 420 records/s for three hours. The deficit is

[ D = \lambda - \mu = 650 - 420 = 230\ \text{records/s}, ]

where (\lambda) is offered arrival rate and (\mu) is successful service rate during the incident. With no bound, the modeled backlog after 10,800 s is

[ Q = D t = 230 \times 10{,}800 = 2{,}484{,}000\ \text{records}. ]

The queue contract caps admitted backlog at 1,800,000 records. At a mean serialized size of 1,400 decimal bytes, that is 2.52 GB of logical payload before indexes, replication, allocator overhead, or compression. The bound is reached after about 7,826 s, or 130.4 minutes. The remaining 684,000 arrivals must be rejected, deferred at an upstream owner, or admitted only by displacing work under an explicitly safe policy.

At the incident service rate, 1,800,000 records represent about 4,286 s—71.4 minutes—of FIFO work. Pulsepipe enters guarded mode when eligible oldest age reaches 45 minutes, before the count bound fills. That leaves time for producers to react rather than treating “full” as the first signal.

During recovery, arrivals fall to 420 records/s and service reaches 850 records/s. The drain surplus is 430 records/s. Starting from the hard bound, ideal drain time is

[ T_d = \frac{1{,}800{,}000\ \text{records}} {850 - 420\ \text{records/s}} \approx 4{,}186\ \text{s} = 69.8\ \text{minutes}. ]

This is a lower bound, not a promise. Partition skew, poison records, sink throttling, retries, and recovery verification can lengthen it. Running recovery faster is useful only while the sink, network, and consumer state remain within their own bounds.

The reproducible fixture is examples/performance-engineering-system-design-handbook/part-03/async-backpressure/. It labels every result modeled. It assumes constant rates, fixed mean size, FIFO service, no retry amplification, and no skew. Those transfer limits are the point: production policy needs distributions by partition and class.

Async observability must retain causal time

Record at least offered, accepted, enqueued, started, committed, and reported timestamps with one monotonic duration source within a process. Wall-clock timestamps help correlate systems but may be skewed. Derive queue wait as start minus enqueue, service time as commit minus start, and completion latency as commit minus accepted or offered according to the user contract.

Carry an operation identity separately from a trace identity. One logical operation may cross retries and traces; one trace may contain multiple messages. Preserve producer context in message metadata and link consumer processing to the creation context. OpenTelemetry’s messaging conventions distinguish creation, receive, and process work and caution against pretending prefetched messages are being processed before an application receives them.

Useful views include:

• offered, accepted, rejected, started, committed, expired, cancelled, and failed rates;
• depth, resident bytes, estimated service demand, oldest age, and wait distribution;
• lag and age by partition, tenant, key, priority, and consumer epoch;
• in-flight work and credits at every hop;
• retry attempts, duplicate detections, tombstone hits, and ambiguous completions;
• worker service distributions, steal rate, idle time, and sink throttling; and
• modeled drain time using current eligible backlog and sustainable surplus.

A single end-to-end histogram can hide two populations: immediate rejections and accepted work that finishes hours later. Report both. Do not count acceptance as goodput when the objective names completed effects.

Choose the boundary by the promise it can keep

Keep the call synchronous when the caller needs the effect now and one deadline can bound the whole invariant. The waiting remains visible and no durable backlog forms, though caller failure can still interrupt unfinished work. Critical-path budgets and dependency tails decide whether that direct contract fits. If optional work dominates the path or a bounded burst must be isolated, the synchronous shape may be charging every caller for work the journey does not require.

Use a bounded in-process queue or channel when the work is reconstructible or deliberately coupled to one process. The handoff can be cheap, and memory and worker limits stay local. It cannot honestly accept work that must survive process loss. Resident bytes, oldest age, and a process-recovery test reveal that boundary more clearly than low send latency does.

Choose an owner mailbox when mutation belongs to one actor or shard. Serialization then follows the invariant instead of a global lock, but skew, supervision, fencing, and mailbox recovery become part of the design. Per-owner age and sequence checks, followed by restart and rebalance tests, show whether the ownership boundary holds. A global invariant or an unsplittable hot key defeats the premise.

A durable queue or log earns its write, acknowledgement, and replay costs when accepted work must outlive its producer. It also creates retention, duplicate handling, sink reconciliation, and bounded recovery demand. Test the acceptance claim across the stated failure, prove replay correctness, and model drain time. If loss is harmless and replay operations would dominate the work, durability is ceremony rather than protection.

Use credits where receiver capacity can be expressed and propagated. Observe granted and consumed credits, blocked time, and bytes and age at every hop; reconnect and epoch tests must show that retries cannot mint capacity twice. Credits can bound in-flight work but cannot control a source that neither slows nor accepts rejection or deferral. In that case, the terminal problem is admission.

Competing consumers and work stealing fit interchangeable tasks whose duration variance leaves workers idle. They can reduce stragglers while spending locality and weakening per-key fairness. Age and completion share by key, the cost of steals, and cache or NUMA evidence decide whether the exchange is worthwhile. Affinity, tenant reservations, or ordering may matter more than aggregate worker use.

Across all six choices, introduce asynchrony only with explicit ownership, durability, ordering, bounds, backpressure, and user-visible completion semantics.

Field checklist

• What does acceptance prove, and which failure class can accepted work survive?
• Does the user journey need acceptance or completed effect, and what is its deadline or freshness objective?
• Where are all explicit and hidden buffers, including client and consumer prefetch?
• What are the count, byte, service-demand, and oldest-age bounds by class and tenant?
• Which key, if any, requires effect order, and how are owner epochs fenced?
• How do downstream capacity and credits reach the terminal admission point?
• What happens when the caller expires before start, during work, or after commit?
• Which operations may be dropped, sampled, coalesced, tombstoned, retried, or compensated?
• How are poison work, replay, failover, and backlog drain isolated from foreground goodput?
• Can traces reconstruct offered, accepted, enqueued, started, committed, and reported time?
• Do authorization, privacy, retention, and deletion rules survive delayed and replayed work?
• What measured condition returns saturated or draining state to normal?

Applied decision: should enrichment become asynchronous?

Pulsepipe’s ingest request validates a schema, appends the event, performs a 40–900 ms third-party enrichment, writes the enriched projection, and returns success. Producers retry after 250 ms. Moving enrichment after the durable append is justified only if raw-event acceptance has standalone value, consumers can tolerate the declared projection freshness, and the product exposes completion or degradation separately.

Compare three options. First, retain synchronous enrichment but bound it by the caller deadline and fail submission when it is required. Second, accept the raw event durably and perform optional enrichment asynchronously, marking the projection pending and expiring work that can no longer affect a reader. Third, split required validation from optional augmentation and maintain a synchronous minimal projection plus asynchronous richer fields.

The evidence that chooses among them is not handler duration alone. Measure how many journeys need the enriched result immediately, retry duplication after ambiguous timeouts, raw-event usefulness, sink order invariants, third-party outage duration, acceptable staleness, and cost to rebuild enrichment. If users immediately poll until enrichment completes, asynchrony may move the same wait into a less efficient loop.

Principal drill: a consumer can fall behind for hours

Design Pulsepipe’s queue policy using the worked rates, then add these constraints: 8% of events are priority-A security signals, one tenant supplies 38% of bytes, 1% of records cost twelve times the median service, source order matters per device, consumers can commit an effect and fail before recording progress, and deletion requests must affect queued payloads within 30 minutes.

Deliver:

1. acceptance, completion, durability, ordering, and ambiguous-outcome contracts;
2. count, byte, cost, age, per-tenant, and priority bounds;
3. push/pull/poll/notification choices and consumer prefetch limits;
4. credit propagation and the source-facing rejection or defer response;
5. deadline, cancellation, coalescing, tombstone, poison, and deletion behavior;
6. partition ownership, fencing, duplicate detection, sink reconciliation, and replay;
7. normal, saturated, draining, and failed state transitions;
8. a capacity model for three-hour deficit and N-minus-one recovery; and
9. dashboards, traces, alerts, rollout thresholds, and rollback conditions.

A strong answer may reserve capacity for priority A while applying weighted service and an age ceiling so other traffic eventually progresses. It does not put every security signal into one globally ordered lane, claim that durable delivery makes effects unique, or let the largest tenant consume the byte budget because record counts look fair.

Review questions and durable conclusions

1. A queue receives 900 records/s and completes 750 records/s for 40 minutes. Ignoring bounds and retries, how many records accumulate? Which additional input converts that count into logical bytes, and which input converts it into an approximate FIFO age?
2. A submission endpoint returns in 12 ms after copying a command into a client library buffer. What evidence is missing before the service can claim durable acceptance?
3. Why can consumer prefetch defeat a broker-side queue bound? State the telemetry needed to measure the complete buffer chain.
4. Give one operation that may be safely coalesced by key and one that may not. Identify the algebraic or business property that changes the answer.
5. A caller times out after a sink commit but before acknowledgement. Explain why cancellation is insufficient and name the identity/status contract needed to resolve the outcome.
6. When would strict FIFO harm the product objective? Propose an alternative fairness definition and the population-level metric that verifies it.
7. Why is an HTTP/2 flow-control window useful as a model yet insufficient as end-to-end application backpressure?
8. A recovering consumer has spare capacity, but its sink is already near saturation. Which rate should determine drain admission, and what observation would falsify the chosen rate?

The durable conclusions are conditional. Asynchrony is timing independence only after ownership transfers at a named boundary. Queue capacity is meaningful only when count, bytes, service demand, and age have explicit outcomes at their limits. Ordering must be scoped to the smallest invariant-bearing key, because enqueue order alone does not guarantee effect order. Backpressure is effective only when downstream permission changes upstream admission before buffers overflow. Cancellation requests a transition; an operation identity and status resolve whether the transition occurred. Finally, recovery is a workload with its own capacity and correctness path, not a return to normal that happens for free.

Evidence and transfer limits

• Reactive Streams 1.0.4 specification defines a scoped asynchronous demand protocol with nonblocking backpressure. It does not define durable application ownership or effect semantics.
• RFC 9113, HTTP/2 specifies receiver-controlled stream and connection flow-control windows. These are hop-by-hop octet credits, not an end-to-end work queue.
• OpenTelemetry messaging span conventions define creation, receive/process relationships and message context links. Instrumentation still needs application operation IDs and business completion states.
• The Pulsepipe numbers are deterministic modeled teaching evidence from examples/performance-engineering-system-design-handbook/part-03/async-backpressure/. They are not observations from a broker, runtime, protocol, storage device, or production service.

Explicit queues make waiting inspectable, but many systems deliberately wait a little longer to do less fixed work per item. The next decision is how much batching and pipeline overlap improve goodput before the added queueing, memory, fairness, and deadline cost becomes the new constraint.