Queues, Utilization, and Backpressure

A system can remain below its apparent capacity and already be late. Pulsepipe averaged 72% CPU, and its ten-minute completion rate still matched its arrival rate. Yet message age had risen from 18 ms to 1.4 seconds, heap residency was climbing, and producers were timing out and retrying. Aggregate CPU was not the constraint: work was accumulating behind a saturated twelve-worker enrichment stage while a five-minute host average concealed short intervals at full worker occupancy.

The useful result came from separating three clocks. Service time at the stage remained about 8 ms. Queue wait accounted for almost all of the new latency. End-to-end response time included both, plus earlier and later stages. Adding workers might move the constraint, but it would not define what happened to work during the next burst.

The design decision is therefore larger than “how many workers?” For every waiting point, decide how much work may wait, how old it may become, who is allowed to add more, and what externally visible outcome occurs when the limit is reached. A queue with no bound, age metric, or overload policy is deferred failure, not resilience.

Waiting exists wherever admission and service are decoupled

A named broker topic is an obvious queue. Many consequential queues do not have “queue” in their API:

• a network interface ring and kernel socket backlog;
• an accept backlog before application admission;
• runnable threads waiting for CPU;
• tasks ready in an async executor;
• callers blocked on a semaphore, lock, connection pool, or rate limiter;
• requests resident in a proxy, load balancer, or client retry loop;
• batches waiting for a size or time trigger;
• storage commands waiting in a device or controller;
• durable messages waiting for a consumer group;
• stale work held in an unbounded in-process channel.

These waiting points compose. A shallow application channel does not bound a request path if the client library silently buffers thousands of requests or if every rejected caller immediately retries. A “queue depth of zero” at one stage can coexist with a large upstream socket backlog and many blocked threads. Draw the path from offered attempt to terminal outcome and mark every place where ownership can outlive immediate service.

For each mark, record the unit of work, capacity unit, owner, scheduling discipline, persistence, bound, admission rule, cancellation behavior, and telemetry. That inventory is the first applied exercise: take the reference path producer → edge → intake → parser → enricher → durable writer → acknowledgment and include the NIC, socket backlog, runtime task queue, worker semaphore, outbound connection pool, broker partition, and producer retry scheduler. If any layer can retain work, it participates in the latency and memory budget.

A queue is five interacting distributions, not one depth gauge

A useful queue model declares:

• the arrival process: rate, burstiness, correlation, classes, and open or closed pacing;
• the service-time distribution for each class, including slow and failure paths;
• the number and heterogeneity of servers;
• the capacity available in the stated interval and operating state;
• the scheduling discipline: FIFO, priority, fair sharing, deadline order, shortest-job variants, or a domain rule.

Let λ be mean arrival rate in jobs/s, S mean service time in seconds/job, and one server provide one second of service per second. For one server, offered utilization is ρ = λS. For m equivalent independent servers, a rough offered utilization is ρ = λS/m. This ratio is dimensionless. It is not a complete capacity result: setup, coordination, shared resources, affinity, class mix, and service-time variance can invalidate the simple server model.

Stability requires long-run offered work to remain below effective capacity. That condition is necessary, not sufficient for an objective. A system can be mathematically stable and still violate a 250 ms deadline on every burst. It can also appear stable only because an upstream closed-loop generator lowers offered load when responses slow, the coordinated-omission problem from Chapter 5.

Depth is a stock; arrival and completion rates are flows. Little’s Law relates their long-run averages under its conditions, but it does not say how delay changes as utilization approaches one. For that, the shape of arrivals and service matters.

The knee is created by lost slack

When a server is lightly loaded, an arrival often finds idle capacity. Near saturation, the same arrival is likely to find earlier work in progress. Small bursts and long service events consume the shrinking idle intervals that previously absorbed variation. Waiting grows faster than utilization.

For a single-server G/G/1 queue in steady state, a practical Kingman-style approximation for mean queue wait is:

E[Wq] ≈ (ρ / (1 - ρ)) × ((ca² + cs²) / 2) × E[S]

Here Wq is queue wait, ca² is the squared coefficient of variation of inter-arrival time, cs² is the squared coefficient of variation of service time, and E[S] is mean service time. A squared coefficient of variation is variance divided by mean squared, so it is dimensionless. The first factor describes the utilization knee; the second describes variability; the last restores time units.

The fixture compares an 8 ms service with two illustrative variability cases. Low variability uses ca² = 0.5 and cs² = 0.25; high variability uses ca² = 2.5 and cs² = 2.0.

At 95% utilization, the high-variability approximation yields 342 ms of mean waiting before 8 ms of mean service. This is a modeled illustration, not a p99 prediction. Kingman’s heavy-traffic work concerns a single-server queue; this compact approximation is most useful for intuition and option screening. It does not model finite buffers, multiple heterogeneous servers, time-varying arrivals, priorities, abandonment, correlated stages, retries, or a network of queues. Simulate or measure the actual policy when those features decide the outcome.

No single “safe utilization” applies to all systems. A deterministic offline worker with a deep completion horizon may operate efficiently near saturation. An interactive stage with bursty arrivals, correlated slow work, and a hard deadline needs more slack. Host-wide CPU can be 50% while one core, lock, partition, connection pool, tenant quota, or device queue is saturated.

Head-of-line blocking changes which job pays

FIFO is simple and often fair by arrival order, but one expensive job can delay many cheap jobs behind it. Consider ten jobs arriving together: one 200 ms export followed by nine 5 ms lookups on one worker. FIFO makes the last lookup wait at least 240 ms. Separating interactive and export classes or using a policy that accounts for size can protect lookups, but only if classification is trustworthy and the export class retains a service guarantee.

This is head-of-line blocking: earlier work prevents later work from using a service opportunity it could otherwise complete quickly. It appears in application queues, multiplexed protocols, storage scheduling, locks, and partitioned streams. The remedy depends on the ordering invariant. Reordering ledger mutations may be illegal even when it lowers latency; moving independent tenants to separate lanes may be both correct and useful.

Priority inversion is different. High-priority work depends on a resource held by lower-priority work, while medium-priority activity prevents the holder from running and releasing it. Priority queues alone do not fix that dependency. Possible controls include priority inheritance in a scheduler, shorter critical sections, eliminating the shared lock, or making dependency priority explicit.

Strict priority can starve low classes. Weighted fair sharing, deficit scheduling, per-tenant concurrency, or reserved capacity can provide bounded progress, but each consumes some peak efficiency and requires a policy for unused reservations. “Premium first” is not a scheduling specification. State weights, minimum service, maximum concurrency, aging behavior, and what happens during prolonged overload.

Bounds turn hidden latency into explicit outcomes

Pulsepipe’s enrichment stage has twelve equivalent workers and an illustrative mean service time of 8 ms:

capacity = 12 workers / 0.008 s per event = 1,500 events/s

An externally paced launch burst offers 2,200 events/s for 400 ms. Ignoring other constraints, excess arrivals are:

(2,200 - 1,500) events/s × 0.4 s = 280 events

The end-to-end deadline is 250 ms. The design allocates at most 120 ms to waiting at enrichment. At 1,500 completions/s, an age-derived FIFO bound is:

1,500 events/s × 0.120 s = 180 events

The full burst would therefore exceed that bound by at least 100 events. A 280-item buffer would retain the burst but spend more than the allocated wait budget at its tail. That is not successful absorption; it is delayed deadline failure. The values are assumed and modeled for the exercise. Production selection needs the service-time distribution, class mix, scheduler, downstream capacity, recovery state, and repeated-burst spacing.

Pulsepipe chooses a 180-event hard bound, begins propagating pressure at depth 120 (about 80 ms of nominal work), and stops accepting new work before the age ceiling is breached. It acknowledges an event only after durable admission. A rejected pre-ack event remains the producer’s responsibility and receives a retryable overload outcome with a server-directed delay range. Already acknowledged work is not silently dropped.

The choice exposes rather than erases scarcity. During this burst the system must reject or divert at least 100 events unless spare capacity, a correct alternate path, or a class policy changes the arithmetic. Product and correctness requirements decide whether the outcome is retry-later, durable spill with a longer objective, partial degradation, or loss. Memory availability does not decide it by itself.

Deadlines and cancellation complete the bound:

• carry an absolute deadline or remaining budget across stages, not a fresh timeout at every hop;
• reject work that cannot plausibly finish within its remaining budget;
• check cancellation before expensive service and at safe interruption points;
• make cleanup release permits, connections, memory, and downstream work;
• distinguish client abandonment from successful completion;
• avoid continuing stale work merely because it has already waited a long time.

Cancellation is a resource protocol. If a timed-out caller disappears while its database query and three retries continue, the queue looks bounded at the edge while abandoned work consumes the constraint. Conversely, interrupting a non-idempotent mutation after its commit point can violate correctness. Define cancellation points around state transitions, not only around await syntax.

Backpressure must travel far enough to change offered work

Backpressure is a signal from constrained consumer toward producer that regulates further production. It is not synonymous with a buffer, a retry, or a slower consumer. A useful path has three properties:

1. the signal originates at the actual constraint or a reliable leading indicator;
2. upstream components can reduce, defer, reshape, or reject work;
3. the control loop has bounded delay and does not create more work than it suppresses.

Demand-signaled streaming can limit how many elements are outstanding across an asynchronous boundary. The Reactive Streams specification is one formal instance, but even its scope does not choose application-level admission, persistence, fairness, or failure semantics. A database pool cannot usefully backpressure an edge if every waiting request already holds megabytes and the client retries on a shorter timer.

Propagate pressure by reducing read credit, consumer concurrency, pull rate, or advertised capacity; by returning a prompt overload result; or by shedding optional work. Preserve feedback across protocol boundaries. If an upstream system cannot slow—telemetry devices may continue emitting—place a deliberate durable buffer sized for a declared recovery horizon, then define what happens when that buffer fills.

Collapse is a positive-feedback loop

An overloaded stage responds slowly. Callers time out and retry. Retries increase offered load and queue age. Longer age causes more timeouts, more connection holding, larger resident sets, and additional retries. Garbage collection, cache eviction, or context switching reduces service capacity, tightening the loop further.

Breaking the loop may require admission control before expensive allocation, concurrency limits at the constrained resource, retry budgets, exponential backoff with jitter, deadline propagation, and a fast failure mode. These controls must be evaluated together. A larger client timeout may reduce retries but retain stale work longer. Aggressive shedding may protect admitted latency while reducing acceptance. Report offered attempts, admitted work, correct goodput, rejections, timeouts, retries, queue age, and resource demand per correct result.

A familiar counterexample is “add a buffer to handle bursts.” It fails when the burst duration exceeds the buffer’s recovery horizon or when queued work expires before service. The changed variable is burst spacing relative to drain time. If a 180-item queue ends full and steady arrivals continue at full 1,500/s capacity, it never drains. Buffering can smooth a finite burst only when subsequent offered work leaves recovery capacity.

Queue telemetry must reconcile stocks, flows, and age

Depth alone is ambiguous. A depth of 100 may be harmless at 100,000 completions/s and disastrous at 10/s. Measure:

• current and maximum depth by class and partition;
• oldest-item age and the distribution of queue wait;
• scheduled, attempted, admitted, started, completed, rejected, dropped, expired, and cancelled counts;
• service-time distribution apart from wait and end-to-end response;
• active server count, concurrency permits, and effective capacity;
• retry lineage and downstream work after caller cancellation;
• bytes and other resident resources, not only item count;
• scheduler outcomes, including service share and starvation by class.

Use a conservation check over a declared interval:

opening depth + admitted - terminal removals = closing depth

Terminal removals must classify completed, rejected-after-admission, expired, cancelled, dead-lettered, and lost work according to the queue’s semantics. A mismatch signals instrumentation gaps, duplicates, or hidden transfers. Queue age is often the earliest user-relevant saturation signal because it measures accumulated delay directly. Linux Pressure Stall Information similarly measures time lost to CPU, memory, and I/O contention; it can reveal resource pressure that a coarse utilization average conceals, but it does not identify an application queue or choose an overload policy.

OpenTelemetry’s messaging conventions provide common operation and destination attributes and attempted-send metrics. As of semantic conventions 1.43.0, parts of messaging metrics remain in development, so pin the convention version and supplement it with application-specific admission, age, outcome, and correctness measures. A standard name does not supply the missing boundary.

Queue design review card

For each waiting point, answer all of these before approving the path:

Run the burst drill without changing its facts: 2,200 events/s for 400 ms, twelve workers, 8 ms mean service, 250 ms end-to-end deadline, and 120 ms queue-age allocation. Reproduce the 1,500 events/s nominal capacity, 280-event excess, 180-event age-derived bound, and minimum 100-event overload outcome. Then change service-time variance while holding the mean at 8 ms. Explain why the same bound may no longer protect a tail objective and identify the measurement needed to choose a new one.

The fixture makes the arithmetic reproducible:

$ node examples/performance-engineering-system-design-handbook/part-01/queues-and-causality/verify.mjs
queues: verified low=3/7/12/27/57 ms high=18/42/72/162/342 ms, burst excess=280, bound=180
causality: ranked=connection held across enrichment, weighted CPU=5.56 ms, connection=26.64 ms

It verifies declared calculations, not a production fit. Before changing a live queue, replay or generate the scoped workload, preserve open-arrival semantics, validate correctness, observe every terminal outcome, and test nominal, burst, degraded, and recovery envelopes.

Decision rules

• Inventory every place that can retain work; a bound at one layer does not bound the path.
• Separate service time, queue wait, and end-to-end response before choosing capacity or timeout changes.
• Treat utilization and variability jointly; do not adopt a universal utilization target.
• Derive bounds from latency, memory, persistence, correctness, and recovery constraints, not available RAM alone.
• Carry deadlines and cancellation through the work graph, respecting commit points and cleanup obligations.
• Make overload an explicit terminal or deferred outcome before expensive work is admitted.
• Propagate pressure to an actuator that can actually reduce offered work, and bound the control-loop delay.
• Reconcile queue stocks and flows while measuring age, wait, service, bytes, outcomes, and class fairness.
• Test collapse and recovery, including retries and abandoned work, rather than only a steady nominal state.

Queues explain where delay accumulates. They do not by themselves prove why one resource, dependency, lock, or coordination step is limiting useful output. The next job is causal: distinguish a busy component from the system constraint and spend the next measurement on the hypothesis most likely to change the decision.

Sources and evidence scope

• J. F. C. Kingman, “The single server queue in heavy traffic”, Mathematical Proceedings of the Cambridge Philosophical Society 57(4), 1961, is the seminal heavy-traffic source. The chapter uses a Kingman-style mean-wait approximation for intuition; it does not transfer that approximation to arbitrary multiserver or networked queues.
• Reactive Streams 1.0.4 specifies interfaces and rules for asynchronous streams with non-blocking backpressure. Its scope explicitly does not select transport, application persistence, admission, or overload semantics.
• Linux kernel Pressure Stall Information documentation defines CPU, memory, and I/O stall signals and pressure triggers. PSI is resource-pressure evidence for Linux environments, not a universal application queue metric or root-cause proof.
• OpenTelemetry messaging semantic conventions 1.43.0 define common messaging spans, metrics, and attributes; portions are still marked development. Pin versions and add domain-specific queue age, admission, and correctness semantics.
• All Pulsepipe values, variability curves, queue bounds, and overload results are assumed or derived teaching fixtures. The Node verifier establishes arithmetic and internal consistency only.