Appendix M — Concurrency and Async Review Checklist

A service has returned 202 Accepted. What, exactly, does it own now?

The answer cannot be “a task is running.” The request may be waiting for a permit, sitting in a queue, blocked on a lock, sleeping before a retry, executing a remote effect, or abandoned by its caller while work continues. Each position has a different owner, memory cost, cancellation boundary, and shutdown obligation.

Review the component as a finite lifecycle:

caller
  │ request + absolute deadline
  ▼
admission ──reject──> caller receives overload
  │ permit acquired
  ▼
owned work ──cancel request──> cleanup / outcome reconciliation
  │ queue slot
  ▼
active attempt ──retry budget──> delayed attempt
  │ terminal outcome
  ▼
permit released + result/event recorded
  │ shutdown waits for this proof
  ▼
quiescent

The controlling invariant is: every admitted unit has one accountable owner, consumes bounded resources, has a defined terminal outcome, and releases its capacity before the component claims quiescence. Send, locks, atomics, futures, timeouts, and runtime APIs are mechanisms inside that lifecycle; none substitutes for it.

Build the admission-to-quiescence ledger

Write one row for every place work can wait or outlive its caller.

“Bounded” must name the limiting dimension. A queue of 1,000 items is not a memory bound until payload retention and allocator overhead are estimated. A semaphore of 100 does not bound tasks spawned before permit acquisition. A timeout does not bound detached work that survives the timed-out future.

The checked-in relay-service-lab acquires an owned semaphore permit before the request becomes active. Its permit travels with the operation and is released by ownership when the active guard drops. That shape is reviewable because admission precedes task ownership; a design that spawns first and awaits a permit inside each task merely moves the unbounded queue into the executor.

Prove transfer and sharing separately

Send means a value can be transferred across thread boundaries. Sync means shared references to the value can be used across thread boundaries; precisely, T: Sync when &T: Send. They are unsafe auto traits because an incorrect manual implementation can make otherwise safe code unsound.

Neither trait promises:

• that two operations form a transaction;
• that an API is logically thread-safe for the caller’s protocol;
• fairness, bounded waiting, deadlock freedom, or prompt cancellation;
• that a future will be scheduled on multiple threads;
• that interior operations are cheap.

This rejected program is useful evidence that Rc<T> cannot cross a spawned-thread boundary:

use std::{rc::Rc, thread};

let value = Rc::new(String::from("tenant-7"));
thread::spawn(move || println!("{value}"));

The compiler reports that Rc<String> cannot be sent safely. Replacing Rc with Arc repairs transfer, but it does not make the inner value mutable, serialize a multi-field invariant, or eliminate atomic reference-count traffic. Credible alternatives are:

• keep Rc inside one thread or local task set and communicate by messages;
• transfer owned data instead of shared ownership;
• use Arc<T> for immutable sharing;
• use Arc<Mutex<T>> only when a lock-protected invariant and contention policy are explicit.

For every public concurrent type, record why its fields make automatic Send and Sync correct. A negative auto-trait result is often a design signal, not an obstacle to erase with unsafe impl.

Match lifetime strategy to the owner

An ordinary thread::spawn closure must own what may outlive the caller, hence its common 'static bound. 'static does not mean “allocated forever”; it means the closure contains no non-static borrowed data. std::thread::scope is the alternative when children should borrow stack data and the scope can guarantee their joins before borrowed data expires.

Async tasks have a similar ownership question, but spawning, cancellation, locality, and join behavior are runtime contracts. Review:

• which executor or thread owns polling;
• whether the spawned future must be Send and 'static;
• whether a local-task facility permits !Send state;
• who retains the join handle and observes panic or failure;
• whether dropping that handle detaches, requests cancellation, or does something runtime-specific;
• whether child tasks are structurally joined before their parent returns.

Do not add move, clone, or 'static until the intended owner is named. Moving an Arc into a detached task may compile while weakening shutdown: the task can retain a database pool or configuration generation after the service believes it has stopped.

Put a number on every waiting room

Capacity follows from a workload envelope, not from a round constant. A first review estimate is:

resident queue bytes
  ≈ queue_items × (retained_payload_bytes + per-item overhead)

maximum admitted work
  = queued + active + retry-delayed + cleanup-in-progress

Then test the assumptions: payload percentiles, burst duration, service rate, deadline distribution, and the behavior of producers when capacity is exhausted.

std::sync::mpsc::sync_channel(n) is bounded; send waits when full and try_send exposes fullness immediately. A bound of zero is a rendezvous. Async channel libraries add runtime-aware waiting and reservation APIs, but their fairness, cancellation safety, and closure rules must be read from the pinned library version.

Backpressure needs an outward policy:

An unbounded queue converts overload into memory growth and stale work. A bounded queue without an overload contract converts it into blocked producers. Review both the container and the caller-visible consequence.

Review locks as temporal dependencies

For each mutex or read-write lock, name the protected invariant, acquisition sites, maximum intended hold work, and ordering relative to other locks. Keep blocking I/O and .await outside ordinary synchronous guards unless the primitive and design explicitly support that interval.

Ask:

• Can a callback or destructor run while the guard is held?
• Can cancellation occur after mutation begins but before the invariant is restored?
• What does poisoning mean for this particular state?
• Are read locks actually read-mostly, or do writers starve under the chosen implementation?
• Is lock contention measured as wait and hold time rather than inferred from CPU use?

Message ownership is often clearer when one task can serialize state. A mutex is often clearer when callers need a short in-process transaction. Sharding reduces unrelated contention but makes cross-shard snapshots and updates harder. Atomics fit observation counters and small state transitions; they should not be scattered around a record whose invariant reviewers must reconstruct.

Give atomics one sentence of authority

Every atomic field needs a sentence describing what its value authorizes. A relaxed active_requests gauge may support telemetry but not reclamation. A release/acquire flag may publish earlier work only when the acquire observes the relevant release. A compare-exchange state machine needs legal transitions and separate success/failure ordering rationales.

If the sentence mentions several ordinary fields, heap reclamation, fairness, a complex rollback, or “flush,” stop and draw the happens-before graph. Prefer a channel, lock, or one-time cell when the proof no longer fits on a review card. Appendix N supplies the litmus tests for that review.

Treat cancellation as an input, not an undo button

Dropping a pending future drops its owned state, but that fact alone does not reverse external effects. The future may have sent bytes, committed a transaction, spawned independent work, or handed a request to a thread. Cancellation safety is therefore a property of each suspension point and resource protocol.

Classify each operation:

1. Drop-safe before commit: no externally visible effect, or owned cleanup restores the local invariant.
2. Resumable: partial progress has a durable cursor or idempotency key.
3. Outcome-ambiguous: the effect may have committed although the caller did not observe success.
4. Must run to cleanup: cancellation requests stop, but the owner must finish a bounded cleanup or reconciliation phase.

A timeout decides how long the caller waits. It does not, by itself, decide whether underlying work stops. Use an absolute deadline propagated through queueing, attempts, and backoff so each layer cannot spend a fresh full timeout. Reserve time for cleanup and response delivery.

Retries require all of the following: a classified transient failure, a replay-safe operation or idempotency mechanism, a remaining end-to-end deadline, bounded attempts and delay, jitter policy, and observability that connects attempts to one logical operation. Never retry an outcome-ambiguous write merely because the client saw a timeout.

Rehearse shutdown from the outside in

Shutdown is complete only when no new work can enter, owned work has reached a documented terminal state, and all resources that matter have been acknowledged or joined.

A useful order is:

1. publish not ready and close admission;
2. stop or reject producers;
3. close queue senders so consumers can detect the end;
4. request cancellation for work that is safe to cancel;
5. drain committed or cleanup-required work within a deadline;
6. escalate according to policy, preserving enough evidence to reconcile ambiguous outcomes;
7. join tasks/threads and release listeners, files, permits, and foreign callbacks;
8. record a terminal event with remaining-work counts.

Dropping every sender can be a clean channel shutdown signal. A boolean flag alone is weaker if sleeping workers have no wake path. Aborting task handles may skip async cleanup and leave external work uncertain. A JoinHandle or explicit acknowledgement is evidence of completion; a cancellation request is only evidence of intent.

Test shutdown at every lifecycle position: before admission, queued, lock-waiting, active before commit, active after an ambiguous effect, retry delay, cleanup, and already terminal. Inject panic and receiver closure. Assert bounds and final ownership, not a sleep-based hope that tasks finished.

Demand observability that explains the lifecycle

Minimum signals should distinguish:

• admission accepted, rejected, and wait duration;
• queue items, retained bytes when material, and oldest age;
• active work and permit utilization;
• lock wait/hold time at important contention points;
• cancellation requested, observed, ignored past commit, and completed;
• timeout stage and remaining deadline;
• retry reason, attempt, delay, and logical operation identifier;
• shutdown phase, remaining owners, escalation, and join result;
• panic, task loss, channel closure, and outcome ambiguity.

Avoid using sampled gauges as lifecycle proof. A relaxed active count is useful operations evidence, but the join/acknowledgement protocol must decide quiescence. Correlate events with operation and attempt identifiers so retries do not look like independent requests.

Reproduce the reference service evidence

The canonical fixture is examples/rust-engineering-handbook/part-10/relay-service-lab. It models admission before active ownership, absolute deadlines, bounded permits, idempotent storage, explicit ambiguous outcomes, event traces, and drainable shutdown.

cd examples/rust-engineering-handbook/part-10/relay-service-lab
cargo +1.97.0 fmt --all -- --check
cargo +1.97.0 check --locked --all-targets --all-features
cargo +1.97.0 test --locked --all-targets --all-features
cargo +1.97.0 clippy --locked --all-targets --all-features -- -D warnings

Paused-time tests make deadline and cancellation paths reproducible without equating wall-clock sleeps with proof. These tests validate the fixture’s asserted behaviors on the recorded toolchain and dependency graph. They do not prove starvation freedom under every executor schedule or cancellation safety for a different I/O driver.

Review exercise: the shutdown that never finishes

Audit a thumbnail service with 64 active permits, a 10,000-item async channel, tasks spawned before permits, a five-second timeout at each of three downstream calls, automatic retries for every error, and shutdown that flips an atomic flag then waits for the active gauge to reach zero.

Produce:

• an admission-to-quiescence ledger including retained-byte estimates;
• a revised spawn/permit order and overload response;
• one absolute-deadline calculation with cleanup reserve;
• a cancellation classification for read, upload, and database commit paths;
• a retry table separating replay-safe, idempotency-keyed, and ambiguous effects;
• a lock/atomic/channel ownership map;
• a shutdown sequence with wakeups, joins, deadline, and escalation;
• six metrics or events that distinguish overload from deadlock and slow cleanup;
• deterministic tests for full queue, cancelled waiter, ambiguous commit, panic, closed receiver, and shutdown during retry delay.

The design passes when every admitted item has one owner and terminal evidence, not when the happy-path test finishes.

Concurrency and async review card

• Do Send and Sync follow from field semantics, with no unjustified manual implementation?
• Does each thread/task own or borrow data for exactly its documented lifetime, and is completion joined or acknowledged?
• Are spawned tasks, queued items, retained bytes, active attempts, retry delays, and cleanup all bounded?
• Does overload propagate to producers through an explicit wait, reject, persist, coalesce, sample, or drop policy?
• Does every lock name its invariant, acquisition order, hold interval, panic policy, and contention evidence?
• Does every atomic name what it authorizes and the ordering edge or observation-only scope?
• Is every suspension point classified for cancellation, cleanup, and outcome ambiguity?
• Are timeouts derived from one absolute deadline, with time reserved for cleanup and response?
• Are retries bounded, jittered, deadline-aware, and safe for the operation’s effect semantics?
• Does shutdown close admission, wake waiters, drain or cancel by policy, join owners, and prove quiescence?
• Can traces distinguish queue delay, lock wait, execution, retry, cancellation, cleanup, and lost-task failure?

Concurrent correctness is a lifecycle property. Review ownership and bounds from admission to quiescence; use language and runtime mechanisms to enforce that path, then collect evidence that exposes where it stalls.

That lifecycle tells you where an atomic participates, but not whether its ordering argument is valid. For each atomic that authorizes publication or a state transition, carry its one-sentence authority into an explicit allowed-outcome and happens-before proof.

Sources and version notes

• Send documentation
• Sync documentation
• std::thread::scope
• Future polling contract
• std::sync::mpsc::sync_channel
• Rust atomics and Ordering

The language examples target Rust 2024 and the manuscript snapshot is Rust 1.97.0. The executable async fixture pins its dependency graph in Cargo.lock; executor scheduling, task-handle cancellation, channel fairness, timer behavior, and I/O cancellation are runtime- and version-specific. Verify those properties against the selected runtime and supported targets. The stated Rust MSRV is 1.85.0 and should be tested independently before editorial acceptance.