Production Async Service Case Study

In the first modeled load test, relay-service reported 9,800 accepted requests per second, a 34 ms p99, and no storage errors. The dashboard looked strong because it measured only requests that reached the handler. The listener had already accepted 42,000 sockets, spawned one task per connection, and allowed those tasks to wait behind a 512-permit storage semaphore. Resident memory climbed after the generator stopped. Shutdown took 78 seconds because no component owned the waiters as a drainable set.

The throughput number was real and still failed to describe the service. relay-service had bounded one expensive stage while leaving admission, connection tasks, parser buffers, and shutdown work effectively unbounded. A deployable async service needs one lifecycle model connecting every owned resource to a limit, deadline, cancellation rule, telemetry outcome, and shutdown owner.

This capstone integrates the contracts from the preceding async chapters. It does not prescribe one framework. Its release criterion is stricter than successful requests under load: every accepted unit of work must have a bounded, observable lifecycle from admission through effect resolution and shutdown. An engineer should be able to trace a request from accept to durable effect, state the maximum live work at every boundary, and explain what happens when the caller disappears, storage replies late, configuration changes, or the process receives a termination signal.

Read the load result as an ownership report

Before tuning code, rewrite the benchmark summary in terms of owned resources:

The table exposes the original error: “512 storage operations” was not a service capacity bound. An accepted connection may own a socket, parser state, read buffer, tracing span, deadline timer, and queued request before storage capacity is acquired. If those resources lack bounds, the storage semaphore merely moves the overload point.

Admission should occur before creating work that is intended to be bounded. For a task-per-connection design, acquire a connection permit before or immediately around accept, according to the listener API and desired refusal behavior. For multiplexed protocols, a connection bound is insufficient; requests or streams need their own limit. A permit must travel with the task that owns the admitted work and return on every exit path, including timeout, parser rejection, panic containment, and shutdown cancellation.

The fixture uses try_acquire_owned to model deliberate shedding:

let permit = Arc::clone(&self.admission)
    .try_acquire_owned()
    .map_err(|_| ServiceError::Overloaded)?;
self.lifecycle.active.fetch_add(1, Ordering::AcqRel);

Waiting for a permit is also a valid policy when a bounded upstream queue and remaining deadline make waiting useful. Immediate shedding is appropriate when the caller can retry elsewhere, the process is already at its safe concurrency ceiling, or retaining the request would consume a scarcer resource. The important choice is not semaphore versus queue; it is whether queued work has an explicit count, byte budget, time budget, and owner.

One request, one budget, several state machines

The request lifecycle is a product of several state machines rather than one handler future:

connection: accepted → reading → request-ready → writing → keep-alive | closed
request:    admitted → parsed → dispatched → effect-known → responded
effect:     absent | ambiguous | committed | replayed | rejected
process:    serving → quiescing → draining → terminating

Each transition must have a unique owner. The listener owns accepted sockets until it transfers them to supervised connection tasks. A connection task owns framing and response writes. An admission permit owns permission to consume request-level capacity. The storage adapter owns transport state, while the idempotency repository owns the durable mapping from logical operation identity to result.

Do not let a parser allocate according to untrusted lengths before admission or validation. A length-prefixed protocol should cap the declared length before allocating. A line protocol needs a maximum line size while reading, not a check after an unbounded buffer has grown. Compression adds a second limit because a small encoded body can expand into a large decoded body. The lab’s compact PUT key body parser validates operation shape, key alphabet, and body length before copying the body into owned storage input.

The request carries one absolute deadline. Queueing, parsing, readiness, storage, response serialization, and cleanup spend that same budget. The service reserves a final slice for cleanup and response handling:

let storage_deadline = request
    .deadline
    .checked_sub(config.storage_reserve)
    .ok_or(ServiceError::DeadlineExpired)?;

let replayed = timeout_at(
    storage_deadline,
    self.store.put_once(&request.operation_id, body),
).await.map_err(|_| ServiceError::DeadlineExpired)??;

This local timeout drops the storage future. It does not prove that a remote effect did not occur. The storage boundary must classify failure as known-before-effect, ambiguous, or known-after-effect. The fixture’s injected AfterCommit failure commits one record and returns EffectAmbiguous; replaying with the same operation ID and body returns the stored result without creating a second record. A reused key with a different body is a conflict, not a replay.

That distinction belongs in the response and telemetry model. Mapping overload, invalid input, deadline expiry, dependency unavailability, and ambiguous effect to one “internal error” destroys the evidence callers and operators need. It also invites middleware to retry a request whose first effect may already exist.

Put backpressure at every ownership transfer

A service can have several independently saturating resources:

• file descriptors and connection memory;
• CPU for parsing, validation, encryption, or compression;
• request/task slots;
• outbound connection-pool slots;
• storage transaction or readiness permits;
• queue bytes rather than queue item count;
• telemetry buffers and response writers.

Each transfer needs either readiness, bounded buffering, or rejection. Backpressure is not complete when only the deepest dependency exposes readiness. If the listener continues accepting and spawning while a storage pool is saturated, pressure becomes memory. If telemetry uses an unbounded channel, an incident can exhaust memory while the request path appears correctly bounded.

Fairness also changes capacity. A first-in-first-out semaphore prevents starvation under ordinary one-permit acquisition, but weighted acquisitions can cause head-of-line blocking. Per-tenant limits can prevent one customer from consuming all global slots, yet unused tenant reservations can lower utilization. A production design often combines a global concurrency ceiling, per-identity quotas, and a small bounded queue with deadline-aware eviction. Document which property is guaranteed and which is a policy approximation.

The worker boundary should not conceal spawning. A future returned to the request task naturally remains attached to that task’s cancellation and deadline. If an adapter must spawn background work, it should register the child with a supervisor and return a handle or operation identity. Fire-and-forget storage work makes request completion, panic propagation, and shutdown unknowable.

Derive capacity instead of copying a semaphore count

Start with a target and measured service time. If the sustained accepted rate is 4,000 requests per second and storage holds a permit for 20 ms on average, Little’s Law estimates about 80 storage operations in the system on average:

concurrency ≈ arrival rate × time in system
            ≈ 4,000/s × 0.020 s
            ≈ 80

That is a planning estimate, not a safe setting. Little’s Law relates long-run averages; substituting a p95 latency does not produce p95 concurrency. Tail latency, burstiness, retries, dependency limits, CPU, and connection-pool behavior need separate distribution and overload evidence. Suppose each admitted request can retain 96 KiB of body, decoded representation, response, and task state. A 512-request limit can then retain roughly 48 MiB before allocator overhead, connection buffers, runtime queues, caches, and the process baseline. If the body cap is 256 KiB and two copies exist during parsing, the same permit count may imply more than 256 MiB.

Build the budget from the tightest constraints:

1. measure per-request retained bytes at representative body sizes;
2. identify the dependency’s safe concurrency and queueing envelope;
3. reserve headroom for connections, telemetry, reload overlap, and shutdown;
4. choose request and byte limits together;
5. verify latency and rejection behavior under bursts, not only steady load;
6. verify recovery after the generator backs off.

Retries consume capacity too. If 10% of calls retry once during a dependency slowdown, the attempt rate is already 1.1 times the logical request rate. Unbounded client retries can create positive feedback: latency causes retries, retries consume more slots, and the extra load raises latency. The service should publish overload and retry-after signals, enforce a retry budget, and preserve the caller’s absolute deadline.

Make observability follow the lifecycle

Telemetry should let an operator reconstruct ownership transitions without logging payloads or every poll. Use one stable logical operation ID and separate attempt IDs. A request span can record:

• admission outcome and queue duration;
• request and response byte buckets;
• configured body limit and policy generation;
• remaining budget at dispatch;
• dependency readiness wait;
• attempt number and idempotency identity;
• effect outcome: known_absent, ambiguous, committed, or replayed;
• cancellation origin;
• response write outcome;
• total duration and terminal lifecycle state.

Metrics should answer capacity questions. Track active connections, active requests, available permits, bounded queue items and bytes, overload rejections, deadline expiry by phase, storage attempts per logical request, ambiguous effects, replay hits, late results, and shutdown drain time. Histograms need controlled cardinality; operation IDs and raw error text belong in traces or structured events, not metric labels.

Health and readiness have different meanings. Liveness should answer whether the process needs replacement. Readiness should answer whether new work should be routed here. During quiescing, the process can remain live while readiness becomes false. A failing optional telemetry exporter should not necessarily fail liveness; a storage dependency that makes every request impossible may remove readiness according to policy.

Telemetry itself needs a failure contract. A full exporter queue may drop low-priority events, apply sampling, or briefly block a bounded producer. It must not silently become an unbounded memory sink. Security review should verify redaction of request bodies, credentials, storage errors, and configuration secrets.

Reload policy, not half a running architecture

Configuration reload is a transaction. Parse and validate a complete candidate, derive dependent values, and publish one immutable snapshot for new work. Existing requests should normally retain the snapshot under which they were admitted. Mixing an old body limit with a new timeout or retry policy inside one request makes incidents difficult to reproduce.

Some settings can reload safely: parsing limits, logging levels, sampling ratios, per-route deadlines, and feature policy when their transition semantics are defined. Other settings require a supervised replacement: listener addresses, TLS identity, storage pools, or global concurrency limits. Replacing a pool should establish the new pool before retiring the old one, bound the overlap, and drain old users. Reducing a semaphore’s intended capacity while permits are live needs an explicit algorithm; changing a number in configuration does not revoke owned permits.

The lab reloads one validated Config behind an async read-write lock. Request handling clones the snapshot, so a request cannot observe half an update. A zero body limit is rejected before publication. This is a teaching model: a production service should version snapshots and emit the policy generation in request telemetry.

Never reload secrets or certificates by mutating shared byte buffers in place. Construct new owned material, validate it, atomically swap the handle, and define when old material is destroyed. Failed reloads should keep the last known-good configuration and emit a bounded, redacted diagnostic.

Shutdown is a protocol with deadlines

Graceful shutdown should move through named phases:

1. Quiesce: mark readiness false and stop admitting new requests.
2. Close producers: stop accepting sockets and prevent internal schedulers from creating work.
3. Drain: let owned requests finish within their existing deadlines; propagate cancellation to children.
4. Reconcile: record or query ambiguous effects and flush essential telemetry within a bounded budget.
5. Terminate: abort remaining local work, close transports, and exit according to process policy.

The order matters. Waiting for active work before closing admission allows the active count to grow forever. Dropping a listener without signalling connection supervisors can leave keep-alive tasks waiting. Cancelling every task immediately may discard responses for already committed operations and create retry ambiguity.

The fixture’s begin_shutdown publishes the shutdown flag with Release ordering and closes the admission semaphore. drain waits until active request guards reach zero. The guard owns both the active count and permit, so timeout and error returns release capacity through Drop. Production code should supervise actual task handles and distinguish successful completion, cancellation, panic, and forced abort. A drain timeout is necessary; “graceful” cannot mean infinite process lifetime.

Configuration reload and shutdown must serialize at a deliberate owner. Once quiescing begins, reject new reloads or prove that the replacement is needed for drain. A signal loop should coalesce repeated termination signals and usually escalate the second signal to a faster termination policy.

Inject failures at transitions, not random lines

Failure injection is valuable when it targets semantic boundaries:

• after admission but before parse;
• while waiting for dependency readiness;
• before any request bytes are written;
• after a partial write;
• after remote commit but before reply;
• during response write;
• while configuration is swapping;
• during each shutdown phase.

The test must assert owned resources and effect state, not only the returned error. The fixture verifies five properties: capacity is reserved before work proceeds; timeout returns the permit; validated reload changes only later requests; an after-commit failure is ambiguous and replayed idempotently; and shutdown rejects new work while draining an existing request. Controlled Tokio time makes the deadline and drain tests deterministic.

Random chaos is useful after deterministic transition tests exist. Record the seed, policy generation, injected transition, and operation trace so failures can become regression tests. A test that kills a process at arbitrary times without durable effect reconciliation may demonstrate fragility without locating the violated contract.

Operational runbook for overload and slow shutdown

When latency rises, operators need an ordered diagnostic path:

The runbook should include commands or dashboards for every evidence field, named owners for traffic shedding and forced termination, and a policy for preserving enough trace data without leaking payloads. “Restart the service” can be valid containment, but it does not resolve duplicate remote effects or explain why work outlived ownership.

Capstone review and load-test plan

Submit the architecture to a review board with the following evidence:

• an ownership map for listeners, connections, request tasks, permits, buffers, storage attempts, and telemetry;
• a table of count, byte, time, and retry bounds at every queue;
• one absolute-deadline trace through admission, parsing, readiness, storage, response, and cleanup;
• cancellation traces before send, after partial send, and after commit;
• configuration snapshot and rollback semantics;
• phased shutdown with a maximum drain time and forced-termination rule;
• an observability map from operator question to event, metric, or trace;
• deterministic failure tests and a load-test protocol.

The load test must contain more than a peak-throughput target. Define steady, burst, slow-dependency, retry-storm, large-body, slow-reader, reload, and shutdown workloads. Record offered load separately from accepted throughput. Measure latency distributions, rejection rate, queue items and bytes, active tasks, memory, CPU, attempts per request, ambiguous effects, replay rate, and recovery time. Run long enough to expose retained work and allocator behavior.

Set acceptance conditions before execution: memory remains within its budget; queue bytes are bounded; overload produces the documented response; accepted latency meets its objective until the capacity knee; retries remain within budget; no logical operation commits twice; the system recovers after load falls; and shutdown reaches termination within its deadline. Set stop conditions for memory growth, dependency harm, excessive ambiguous effects, and lost observability.

The review fails if any bound is merely “configured” but cannot be observed, if cancellation is equated with remote rollback, if a background task lacks a supervisor, or if the load generator reports only successful responses.

Service integration review

• Admission occurs before bounded work is spawned or retained.
• Connection, request, buffer, queue, storage, retry, and telemetry resources have count or byte limits.
• Parser limits apply during accumulation and decompression, not after unbounded allocation.
• One absolute deadline crosses every layer and reserves cleanup time.
• Effect ambiguity survives error translation and controls retry behavior.
• Storage readiness represents real capacity and idempotency is durable at the commit boundary.
• Internally spawned work is registered with a lifecycle owner.
• Telemetry distinguishes offered, admitted, rejected, cancelled, ambiguous, committed, replayed, and late work.
• Configuration candidates are validated and published atomically; transition semantics are explicit.
• Shutdown closes producers before waiting, drains within a deadline, and has an escalation rule.
• Failure tests assert permits, tasks, connections, and effects, not only error values.
• Load evidence includes saturation, recovery, reload, and shutdown behavior.

relay-service-lab is intentionally smaller than a network server. Its value is executable ownership: five tests pin the admission, deadline, reload, idempotency, and shutdown transitions on Tokio 1.52.3. The next part crosses a different boundary. Where this service fixture forbids unsafe code and relies on compiler-enforced contracts, systems work sometimes needs operations whose preconditions the compiler cannot establish. Those operations require a written safety case, not a suspension of rigor.

Sources and version note

Tokio 1.52.3 documents its semaphore as a fair counting semaphore, try_acquire_owned as an owned nonwaiting permit acquisition, close as preventing further permits, and cancellation of queued acquisition as losing queue position. Its timeout_at API supplies the local absolute-deadline wrapper used by the fixture. The fixture pins that crate version in Cargo.lock and declares Rust 1.85 as its MSRV. It passed Rust 1.97.0 checks, tests, and Clippy during drafting. The architecture is a teaching model, not a benchmark claim or framework recommendation. Before adopting it, verify the capacity assumptions, shutdown races, atomic-ordering choices, and operational behavior against the service that will actually run.