Resilience Under Load: Admission Control, Backpressure, and Degradation

The first capacity review of relay-service produces one number: 3,145,728 bytes.

That is the modeled payload memory for 512 queued 4 KiB items plus sixteen 64 KiB active working sets. It excludes allocator overhead, task state, socket buffers, decoded expansion, retries, telemetry, caches, native libraries, and the process baseline. Yet the existing design calls the queue “just a channel” and leaves it unbounded.

Overload begins wherever accepted work retains a resource faster than the system releases it. The resilience contract is: admit only work that fits a named capacity and deadline; propagate pressure to an ownership boundary that can wait or reject; isolate scarce resources; degrade by explicit priority; and restore capacity slowly enough that recovery does not recreate overload.

Draw the capacity envelope before choosing a mechanism

Capacity is multidimensional. A service may have spare CPU but no database connections, memory but no file descriptors, worker permits but an exhausted downstream quota, or average throughput but an unacceptable latency tail. Define the envelope as constraints:

arrival rate × retained time < retained-work budget
in-flight work ≤ concurrency budget
queued bytes + active bytes + baseline + reserve ≤ memory limit
completion time ≤ caller deadline
downstream demand ≤ downstream safe capacity

Little’s Law relates average work in a stable system to arrival rate and time in system, but averages do not authorize an infinite tail or burst. Use measured distributions, burst assumptions, payload-size bounds, and a reserve for uncertainty. A queue capacity expressed only as item count is incomplete when payloads vary by orders of magnitude. Consider byte permits or reject oversized work before it occupies the normal lane.

The lab uses checked arithmetic:

capacity_bytes(512, 4096, 16, 65_536)

It returns None on integer overflow. The calculation is deliberately narrow; production budgets must include every retained representation and validate them against the deployment limits from Chapter 89. If the system cannot name the budget consumed by accepted work, it cannot make a defensible admission decision.

Read the left side as a set of simultaneous constraints, not nested reserves; then use the right side to identify exactly which amplification edge each control is intended to cut.

Bound every place that can retain work

A bounded queue limits one storage location. It does not limit work waiting before send, spawned tasks holding payloads, kernel accept queues, client retries, per-connection buffers, downstream pools, or alternate priority lanes. Trace ownership from ingress to completion and mark every retention point.

Tokio’s bounded mpsc channel waits when capacity is exhausted; try_send rejects immediately. That is a mechanism choice. Policy determines whether this caller may wait, for how long, with what owned bytes, and what happens on timeout. If an HTTP handler waits indefinitely while retaining a decoded request and socket, pressure has moved upstream but is not bounded. If it spawns a task before waiting, the task set may become the new unbounded queue.

Concurrency limits bound simultaneously active work. Use a semaphore or worker set around the scarce operation, not merely around a convenient function. A database permit should cover the interval that actually holds a connection; a CPU permit should not remain held while waiting on unrelated network I/O. Conversely, releasing a permit before the protected resource is free defeats the bound.

Coordinate queue and concurrency limits. A queue of 1,000 feeding two workers can preserve bursts but may make 998 requests miss their deadlines. A concurrency limit of 1,000 in front of a pool of 20 merely relocates waiting into the pool. Capacity is an end-to-end envelope, not a collection of locally reasonable integers.

Admit at the last boundary that still has choices

Admission control should run before expensive parsing, allocation, authentication work that can safely be deferred, or task creation, while still having enough information to apply tenant, priority, and cost policy. This is often a staged decision:

1. enforce transport and request-size limits;
2. authenticate enough to identify policy without doing full work;
3. reject expired or impossible deadlines;
4. acquire capacity for the expected cost class;
5. enqueue only with an owned permit or bounded reservation;
6. release the permit exactly when the accounted resource is released.

The fixture’s gate makes four outcomes explicit: queue full, concurrency full, deadline expired, and optional work shed. A real protocol should map them to stable responses that tell clients whether retry is permitted and, if useful, when. Do not report every rejection as an internal error. Controlled rejection is successful overload policy and must be counted separately from bugs and dependency failures.

Waiting and rejecting are both valid. Backpressure works well when the producer is cooperative, waiting consumes bounded resources, and the remaining deadline can still succeed. Immediate shedding works when waiting would only create stale work, the caller can select an alternative, or the service must preserve resources for admitted requests. Durable ingestion may append to a bounded log and acknowledge ownership transfer, but “durable” needs disk, replication, quota, and replay capacity contracts.

Deadlines prevent obsolete work from consuming recovery

A timeout limits how long one caller waits. A deadline identifies when the result loses value and should travel with the operation. Before queueing, starting a retry, or calling a dependency, calculate the remaining budget. Refuse work that cannot plausibly finish. At dequeue, drop expired items without spending the full service cost, while preserving any required audit or compensation behavior.

Cancellation does not guarantee rollback. If an operation crossed its commit boundary, the client needs idempotency, deduplication, or status lookup rather than a blind retry. The service should distinguish:

• not admitted: safe to retry according to policy;
• admitted but not committed: retry depends on cancellation evidence;
• committed but response lost: use the same idempotency key or query status;
• outcome unknown: never translate uncertainty into unlimited retries.

Deadline propagation should reserve time for response serialization and upstream handling. Giving every downstream the full original deadline causes the outer layer to time out first, discard the response, and retry while the original work continues.

Stop retry storms at their source

Retries multiply arrival rate during the very condition that reduced completion rate. If five layers each attempt three times, one user action can fan out far beyond three calls. Choose one retry-owning layer for each operation. Bound attempts by a shared retry budget, remaining deadline, idempotency, and error classification. Use exponential backoff with jitter to decorrelate clients, but remember that delayed retries still retain intent and can form a recovery wave.

Servers should return stable overload evidence; clients should honor it without synchronizing on one instant. A retry budget can be a fraction of normal successes or a token bucket replenished by healthy completions. When the budget is empty, fail rather than borrow from future capacity.

Hedged requests can reduce tail latency when independent replicas occasionally straggle, but they deliberately add load. Gate them by latency evidence, idempotency, a separate budget, and cancellation of losers. Hedging during systemic overload is amplification.

Circuit breakers and bulkheads solve different coupling

A circuit breaker stops calls to a dependency after evidence suggests they are unlikely to succeed. Its states usually include closed, open, and a limited trial state. It protects callers and dependencies from repeated doomed work, but it does not increase capacity, repair the dependency, or replace deadlines. Thresholds based on raw failure counts can open during tiny samples; global breakers can punish healthy tenants or regions; synchronized probes can create a thundering herd.

Define which failures count, the sampling window, minimum volume, open duration, trial concurrency, and fallback. Export state transitions and allow bounded operator intervention with auditability. Prefer automatic recovery based on cautious probes over a permanent manual switch, but provide a safe way to hold or disable behavior during an incident.

Bulkheads isolate resource pools so one workload cannot consume everything. Separate interactive from batch work, control-plane from data-plane calls, or one dependency from another when their failure domains justify it. Reserve capacity rather than creating many tiny pools that strand resources. A shared global limit can cap total memory while sub-limits preserve critical lanes.

Fairness is part of admission policy. FIFO is predictable but allows large or slow jobs to block small ones. Per-tenant queues prevent one tenant from monopolizing capacity but multiply buffers. Weighted fair scheduling, deficit schemes, cost-class permits, or reserved critical capacity can help, yet each needs resistance to identity splitting and misclassified cost. State the fairness objective—equal requests, bytes, CPU time, tenant share, or deadline class—because they conflict.

Degrade capabilities, not invariants

A degraded mode should be a smaller valid product, not ordinary behavior with weaker correctness. relay-service might stop enrichment, reduce optional fan-out, serve a bounded stale cache, lower batch detail, or accept only critical topics. It must not skip authentication, corrupt ordering guarantees, acknowledge work before ownership transfer, expose secrets, or silently discard required records.

Define entry and exit conditions, affected operations, client-visible response, data freshness, observability, and maximum duration. Precompute or test the degraded path; an emergency branch that has never received traffic is not resilience.

Memory pressure deserves an early signal. Queue depth alone misses payload size, allocator fragmentation, caches, and active working sets. Observe queued bytes, in-flight cost classes, resident memory, allocation failure where recoverable, and platform memory events. Shed before the supervisor kills the process. Avoid allocating a rich error body while memory is exhausted; reserve a minimal rejection path.

Operator controls can lower concurrency, disable optional work, open a breaker, freeze batch admission, or select a known safe mode. Bound their range, authenticate changes, record actor/reason/expiry, expose effective state, and make rollback obvious. A control that bypasses all capacity limits is an incident accelerator.

Recovery is a state, not the absence of errors

When a dependency returns or arrivals fall, queued work, retries, caches, connection establishment, and delayed maintenance all compete. Returning instantly to full concurrency can collapse the recovering dependency. Use hysteresis: require sustained healthy windows, raise limits in steps, cap probes, and prioritize already-admitted or oldest still-useful work.

The fixture enters Degraded on an overloaded window, moves through Recovering, and returns to Normal only after three healthy observations. Three is teaching data, not a universal threshold. Production thresholds should follow measurement intervals, dependency behavior, and recovery objectives. The key invariant is separate entry and exit evidence so the system does not flap at one boundary.

Drain obsolete queues during recovery. Expired requests should not run merely because capacity reappeared. Smooth client retry releases with jitter and budgets. Warm caches and pools at a rate that leaves room for current demand. Watch completion rate, queue age, rejection reasons, memory, downstream latency, breaker state, and degraded-mode duration together; a falling error rate caused by rejecting nearly everything is not recovery.

Prove predictable failure with controlled faults

Chaos is not random destruction. Start from a falsifiable steady-state hypothesis and a bounded blast radius. For the unbounded relay ingestion path, redesign it as follows:

• set byte and item budgets from measured payload distributions;
• reject oversized or expired requests before allocation-heavy parsing;
• acquire an ingress permit and bounded queue reservation;
• isolate critical and optional work under a shared memory ceiling;
• propagate one deadline and idempotency key;
• allow retries only at the owning client layer under a retry budget;
• shed optional enrichment when queue age or memory crosses the entry threshold;
• recover concurrency in measured steps after sustained health.

Then run four experiments.

Normal: hold arrival below measured service rate; prove low rejection, stable queue age, and no leaked permits. Saturated: exceed the envelope with bounded payloads; prove memory plateaus and rejections use the documented taxonomy. Coupled failure: slow one dependency and burst one tenant; prove its bulkhead and fairness policy preserve the critical lane. Recovery: restore the dependency while releasing delayed retries; prove hysteresis, expired-work removal, and no second overload peak.

Add faults one at a time: latency, refusal, partial response, lost acknowledgement, memory pressure, CPU quota, descriptor exhaustion, worker panic, clock jump where relevant, and operator-control error. Abort the experiment if safety bounds fail. Preserve the exact artifact, configuration, traffic seed, timeline, and telemetry queries so the result is reproducible.

Resilience review

• Is every retained-work location bounded in items, bytes, time, and ownership?
• Do queue and concurrency limits protect the actual scarce resources?
• Can admission reject before expensive work while still applying priority and fairness policy?
• Do deadlines propagate and prevent obsolete queued work from running?
• Is exactly one layer responsible for retries, with idempotency and a shared budget?
• Are breaker evidence, trial traffic, and reset behavior explicit?
• Do bulkheads isolate real failure domains without hiding a global memory limit?
• Does degradation preserve security, durability, ordering, and acknowledgment invariants?
• Are overload signals distinct from internal failures and visible when telemetry is stressed?
• Does recovery use hysteresis, staged concurrency, expired-work removal, and retry smoothing?
• Can operators change bounded controls with authentication, audit, expiry, and rollback?
• Have saturation, coupled failure, and recovery been exercised under deployment limits?

Resilience is not the ability to accept every request. It is the ability to keep promises about the work that was accepted, give precise outcomes to work that was not, and return to normal without another collapse. Those same admission and isolation boundaries become security boundaries when demand is malicious rather than accidental; capacity policy must therefore hand off to an explicit threat model rather than rely on memory safety alone.

Sources and version notes

• Tokio bounded mpsc documents bounded waiting, try_send, disconnection, and clean shutdown; queue policy and end-to-end memory bounds remain application responsibilities.
• Tokio Semaphore documents permit behavior and FIFO fairness, including head-of-line implications of multi-permit acquisition; pin and revalidate the crate version.
• Tokio timeouts document cancellation by dropping the future when the duration elapses; this does not establish transactional rollback of external effects.
• Kubernetes probes warns that incorrect liveness behavior can cause cascading failures and distinguishes readiness during overload.
• Reactive Streams specifies asynchronous stream processing with non-blocking backpressure; it is a protocol model, not proof of a whole-service capacity envelope.

The fixture targets Rust 2024 with declared MSRV 1.85 and no third-party dependencies. Its deterministic gate proves only its stated item/concurrency/deadline and hysteresis rules. Production conclusions require workload distributions, runtime and dependency versions, platform limits, network behavior, and fault experiments using the deployed artifact.