Shared State: Mutexes, Read-Write Locks, Condition Variables, and One-Time Initialization

Suppose relay-service can either send every cache operation to one owner thread or let request threads read and refresh the cache directly. Chapter 54 made the first option concrete: a queue serializes commands, but capacity, overload, replies, and shutdown become part of correctness. The second option removes that queue and its handoff latency. It does not remove the serialization point. It relocates it into a guard around a shared relationship.

That relationship—not the block of code—is the object of design. A cache entry, its freshness generation, and the fact that a refresh is in flight may form one invariant. If those fields are protected separately, each individual access can be race-free while readers still observe a state that the component declares impossible.

The useful review question is therefore not “Did we wrap the map in Arc<Mutex<_>>?” It is:

Which facts must change and be observed together, which lock protects them, how long may a guard live, and what happens if acquisition, waiting, work, or recovery does not complete normally?

Shared state is a good fit when several callers need short synchronous operations on the same small invariant. It becomes dangerous when the protected operation quietly includes I/O, callbacks, allocation spikes, another lock, or an async suspension. The mutex prevents simultaneous access; the surrounding design must still provide liveness, bounded latency, and a credible failure policy.

Draw the invariant boundary before choosing the primitive

Rust’s Mutex<T> associates mutual exclusion with T. Calling lock blocks until it can return a MutexGuard<T>. The guard dereferences to the protected value, and dropping it releases the lock. This RAII shape makes exceptional control flow safer than manual unlock calls, but lexical convenience does not decide the right scope.

For a cache with single-flight refresh, a plausible protected state is:

struct CacheState<K, V> {
    entries: HashMap<K, V>,
    loading: bool,
    generation: u64,
}

The invariant might be stated as follows:

1. loading == true means exactly one caller owns permission to produce the next cache value.
2. An inserted value and the increment of generation become visible as one locked transition.
3. Every transition from loading to not loading wakes callers whose predicate may now be false.
4. User-provided loading code never runs while the state guard exists.

The mutex should protect all three fields because their relationship is what readers rely on. A separate atomic loading, mutex-protected map, and atomic generation would be three individually safe locations with no automatic composite snapshot. A reader could see a new generation and an old map, or decide to refresh from a stale combination. Stronger atomic ordering does not invent the missing transaction.

Conversely, putting unrelated telemetry, immutable configuration, and independent tenant shards under the same mutex enlarges the serialization domain for no correctness benefit. Begin with the invariant graph: nodes are fields or resources, and an edge means they must participate in one transition. One connected component is a candidate lock domain. It is a starting model, not a command to combine every field ever read by the same function.

Arc and Mutex solve different problems. Arc<T> shares ownership of one allocation across threads. Mutex<T> coordinates mutable access. Arc<Mutex<T>> is appropriate only when both shared lifetime and exclusive mutation are required. A mutex in a statically owned structure, a scoped thread borrowing &Mutex<T>, or an owned component with non-cloneable access can avoid the extra ownership flexibility.

Guard lifetime is a latency budget

A guard starts when acquisition succeeds and ends when it is dropped. Treat that interval as a visible operational budget. It contains both useful mutation time and every accidental delay introduced before the guard leaves scope.

let value = {
    let mut state = cache.state.lock()?;
    state.hits += 1;
    state.entries.get(key).cloned()
}; // guard drops here

if let Some(value) = value {
    emit_access_event(key); // no cache lock is held
    return Ok(value);
}

The inner block is not cosmetic. It separates the protected transition from downstream work and gives reviewers an obvious unlock point. drop(state) is useful when a longer function needs an explicit boundary, but a small method on the protected state often communicates the transition better.

Do not retain a guard across:

• network, disk, DNS, or process I/O;
• a blocking channel send whose receiver might need this lock;
• a join or condition wait on an unrelated predicate;
• expensive parsing, compression, or user-controlled allocation;
• logging or metrics paths that can call unknown sinks;
• a callback, trait method, destructor, or hook with reentrant behavior;
• .await in an async task.

Copying or cloning a small result out of the guard can be the right price for shorter contention. Cloning an unbounded structure on every read can instead move the incident from lock wait to allocator and memory pressure. Alternatives include immutable Arc snapshots, versioned replacement, a narrower owned result, or changing the API so work happens inside a small, non-reentrant closure controlled by the component. The last option still needs a strict callback contract; “closure” does not mean “cheap or harmless.”

Measure acquisition wait and hold time separately. A high wait with a short hold suggests too many contenders or unfair scheduling. A long hold points to work inside the critical section. Averages hide the queueing effect; use distributions and identify the lock or shard without attaching unbounded request identifiers.

API shape decides whether callers can extend the critical section

A method that returns MutexGuard<'_, T> delegates lock duration to every caller. That can be appropriate for a low-level synchronization wrapper, but it prevents the component from promising a bounded critical section. Callers may retain the guard in a struct, pass it down a deep call chain, or perform work that the lock owner cannot inspect.

Prefer operations that express domain transitions and return owned observations:

impl Cache {
    fn record_hit(&self, key: &Key) -> Option<Arc<Entry>> {
        let mut state = self.state.lock().unwrap();
        let entry = Arc::clone(state.entries.get(key)?);
        state.hits = state.hits.saturating_add(1);
        Some(entry)
    }
}

The method couples lookup and hit accounting under one guard, returns an owned shared handle, and fixes the release point before caller code runs. Whether Arc<Entry> is acceptable depends on eviction semantics: removing the map entry no longer proves destruction because callers can retain clones. If reclamation or secret erasure requires exclusive lifetime control, return a copy, an identifier, or an operation result instead.

Closure-based APIs such as with_state(|state| ...) appear to control access but execute unknown code under the guard. A private closure used by audited internal methods can reduce duplication. A public closure should be treated as a callback boundary: it can panic, block, allocate, or reenter unless the type system and API sharply restrict it. Even a closure that cannot escape the borrow can hold the lock for arbitrarily long.

Returning a reference borrowed from a guard naturally retains that guard’s lifetime. Guard-mapping facilities can narrow which field the caller sees without releasing the underlying lock; they improve encapsulation, not contention. A type named CacheEntryRef should make guard retention explicit rather than masquerading as an ordinary cheap reference.

Failure results deserve the same boundary design. If a method returns a domain error after partially updating protected state, the invariant must already be restored before the guard drops. Do not use the mutex as rollback. A common pattern is validate all fallible inputs before acquisition, compute a proposed change outside the lock, revalidate the relevant generation under the lock, then commit with only non-fallible operations. This is optimistic work with a version check, not an atomic transaction; conflicts need retry or rejection policy.

Guard ownership also affects public auto traits. Standard mutex guards are intentionally scoped to the acquiring thread on supported APIs; do not design work transfer around sending a live guard to another worker. Transfer owned data or a command, then let the destination acquire its own synchronization boundary. This keeps unlock behavior attached to the thread and stack that established the critical section.

At the API review, require an answer for every method: does it acquire, can it block, what invariant does it transition, can it invoke foreign code, what owned value leaves, and is a guard hidden inside that value? Lock behavior is part of the method’s operational contract even when the mutex type is private.

The fixture makes loading a permission, not a long critical section

The companion synchronization-memory-lab implements a deliberately compact single-flight cache. On a miss, one caller changes loading under the mutex, drops the guard, calls the loader, then reacquires the mutex to publish the result and wake waiters.

if !state.loading {
    state.loading = true;
    drop(state);

    let loaded = load(&key);
    let mut state = self.lock_recovered();
    state.loading = false;
    if let Ok(value) = &loaded {
        state.entries.insert(key, value.clone());
        state.generation = state.generation.wrapping_add(1);
    }
    self.changed.notify_all();
    return loaded;
}

This is a three-interval operation with two short locked phases:

1. Reserve under the guard: change loading from false to true and establish one loader.
2. Work without the guard: call the loader while other protected operations remain able to acquire the mutex.
3. Commit or release under the guard: publish success or clear the reservation, then notify waiters whose predicate may have changed.

Correctness depends on every path that completes reservation reaching the third interval. If the loader can panic and the process unwinds, the teaching fixture’s loading flag would remain true because the panic occurs without the state lock held. A production API must choose among catching unwind at this boundary and repairing state, using an RAII reservation whose Drop clears the flag, forbidding unwinding across the callback boundary and aborting, or delegating the work to a supervised owner.

The fixture serializes misses for different keys behind one loading flag. That is intentionally easy to inspect and potentially poor under a diverse key workload. Per-key flight records permit unrelated misses to proceed but create another lifecycle problem: who removes abandoned records, and how are waiters prevented from holding a stale record? Sharding by a stable hash bounds contention and metadata but admits false hotspots. A single-owner channel makes cleanup serial again but reintroduces queue policy. The simplest correct design should be measured before adding per-key coordination.

The lock-scope timeline in the memory aid shows why the loader cannot sit between acquisition and release. Notice also that the mutex surrounds entries, loading, and generation; it does not visually “protect” the loader function.

The two short blue guard intervals are the serialization budget. The long loading lane is owned work outside the mutex. In the state machine, notification is merely a reason to compete for the lock and re-evaluate; it is not proof that the desired value exists.

A condition variable waits for a predicate, not a message

A condition variable lets a thread release a mutex and block until a state change may make progress possible. Condvar::wait atomically unlocks the represented mutex and blocks; before returning it reacquires that mutex. This closes the lost-wakeup window that would exist if code unlocked and then separately registered itself as a waiter.

The condition is still data protected by the mutex. A notification carries no durable payload and grants no reservation. Wakeups may be spurious, another waiter may consume the resource first, and a notification may describe a transition irrelevant to this key. Therefore the correct shape is a predicate loop:

state = changed
    .wait_while(state, |state| {
        state.loading && !state.entries.contains_key(&key)
    })
    .unwrap_or_else(|poisoned| poisoned.into_inner());

wait_while checks immediately, waits only while the closure returns true, and checks again after reacquisition. An explicit while predicate { state = changed.wait(state)?; } expresses the same essential model.

Change the predicate while holding the mutex. Notify after the state transition; otherwise a woken thread can reacquire and find that nothing changed. Whether to call notify_one or notify_all follows from eligibility. If one available item can satisfy one interchangeable waiter, waking one may limit a herd. If a generation change, shutdown, or inserted key can make many distinct predicates false, waking all may be necessary. Even notify_one provides no right to proceed without rechecking.

Timeouts add another outcome, not a clock guarantee. wait_timeout_while accounts for repeated relative waits while the predicate remains true and reports whether the timeout is known to have elapsed. On return, code owns the guard again and must inspect both predicate and elapsed result. Scheduling and platform behavior make it unsuitable for precise deadlines. For a caller deadline, compute a remaining budget from a monotonic clock and define whether timeout cancels only waiting, cancels the refresh reservation, or leaves background work running.

One condition variable should remain paired with its mutex and documented predicate family. The standard Condvar documentation warns that using it with more than one mutex over time may panic. More importantly, splitting predicate fields across mutexes makes the atomic unlock-and-wait contract impossible to reason about.

Poisoning is a signal that demands a domain policy

When a thread panics while holding a standard-library Mutex, later acquisition normally returns PoisonError containing the acquired guard. This is advisory failure propagation, not an access-control wall and not proof of corruption. into_inner allows inspection; clear_poison clears the flag after a deliberate repair. Poison detection has documented edge cases, so unsafe code must never depend on poisoning for soundness.

There are three defensible responses:

1. Propagate or terminate. If an in-memory index cannot be trusted and reconstruction is expensive or ambiguous, fail the operation, stop the component, or restart the process.
2. Validate and continue. Inspect all coupled fields under the recovered guard, repair a known partial transition, record the recovery, then clear poison.
3. Replace from authority. Overwrite the protected value from a durable snapshot or other source of truth, then clear poison.

Blindly calling poisoned.into_inner() and continuing makes the poison result meaningless. Blindly unwraping may be correct for a fail-fast component, but the panic propagation and restart policy should be intentional.

The fixture’s recovery method clears an in-flight reservation, clears poison, and wakes waiters:

self.state.lock().unwrap_or_else(|poisoned| {
    let mut state = poisoned.into_inner();
    state.loading = false;
    self.state.clear_poison();
    self.changed.notify_all();
    state
})

This repair is valid only because loading = false is sufficient to restore the fixture’s small invariant. A real cache may also need to discard a partially built entry, restore generation metadata, or invalidate dependent indexes. Recovery code belongs beside a written invariant and a test that deliberately poisons the lock. If the recovery path can itself panic, define whether the process aborts or a supervisor replaces the component.

Panic-safe mutation is stronger than poison handling. Suppose an update removes an old index entry, calls a fallible allocator-dependent operation, then inserts its replacement. A panic between removal and insertion can leave the state internally valid as Rust values but invalid according to the domain. Poison may alert a later acquirer, yet it does not provide the old entry or describe the intended repair.

Design transitions so every potential panic point sees an acceptable state. Build replacement values before locking. Use swaps where either old or new is valid. Keep enough journal data under the lock to finish or roll back. For critical state, consider abort-on-panic or an owner process that reconstructs from durable authority. catch_unwind is not a universal transaction facility: not all panics should be resumed through a component, foreign code may not be unwind-safe, and an aborting panic profile runs no recovery.

Destructors are part of this analysis. Replacing or removing a value can run its Drop while the guard is held. A destructor may be expensive, acquire locks, flush buffers, or call foreign code. Move the old value out, restore the protected invariant, drop the guard, and only then drop the old value when the API permits. This is especially important for reference-counted values whose final clone can trigger cleanup unpredictably.

Recovery observability should distinguish detection from successful repair. Emit the lock identity, invariant version, repair action, and component disposition without logging protected secrets. Incrementing “poison seen” before validation and “poison cleared” afterward makes repeated failures visible. Never clear poison merely to silence alerts.

RwLock uses related advisory poisoning, but a panic while a write guard is held is the relevant poisoning case; a reader panic does not poison it. That asymmetry follows from exclusive mutation being the likely invariant-changing operation, but it does not absolve readers from side effects performed elsewhere.

Read-write locks optimize overlap only under the right workload

RwLock<T> permits multiple read guards or one write guard. That sounds ideal for a read-heavy cache, but “read-heavy” is insufficient evidence. Consider:

• how long each read guard lives;
• how often writers arrive and how much they update;
• whether reads touch the same cache lines and shared reference counts;
• whether a writer must wait for a convoy of readers;
• whether the platform’s lock policy favors readers or writers;
• whether operations advertised as reads mutate recency, statistics, lazy fields, or reference counts elsewhere.

The standard lock does not guarantee a particular reader/writer priority policy. Code must not assume that a queued writer blocks later readers, nor that readers cannot starve a writer. Recursive acquisition patterns are especially suspect: acquiring another read lock while holding a read guard can deadlock on a policy that gives a waiting writer precedence.

A plain mutex can outperform an RwLock when critical sections are tiny, writes are common, or reader bookkeeping and cache-line traffic dominate. An immutable Arc<Snapshot> can make reads independent, at the cost of rebuilding and retaining versions. Shards reduce contention but weaken cross-shard atomicity. A channel-owned cache makes updates and eviction deterministic but charges every read a request/reply path unless readers use snapshots.

Select with a workload and an invariant, then benchmark on the supported targets. Lock implementation, scheduler, core count, NUMA placement, and read/write distribution all matter. “More parallel reads” is a mechanism, not a throughput result.

One-time initialization is publication with a narrow lifecycle

OnceLock<T> represents a value that is normally written once and then read by shared reference. It is a strong fit for immutable process configuration, a compiled lookup table, or an initialized service handle whose lifecycle genuinely lasts for the static or owner containing the cell.

static PROCESS_LABEL: OnceLock<String> = OnceLock::new();

fn process_label(source: impl FnOnce() -> String) -> &'static str {
    PROCESS_LABEL.get_or_init(source).as_str()
}

Initialization and publication synchronization are provided by the primitive; callers should not add a separate “initialized” atomic flag. get is nonblocking and can return None while initialization is in progress. get_or_init participates in initialization and returns the stored reference. Rust 1.97 also documents wait for callers that must block until initialization completes. Pick the API whose waiting behavior belongs in the caller contract.

One-time does not mean infallible by default. Decide what an initialization panic or error means, whether retry is allowed, and whether configuration should instead be constructed in main and passed down explicitly. A global cell complicates test isolation, multi-instance use, teardown, and reload. If the value changes operationally, a versioned snapshot or locked state is more truthful than pretending reload is initialization.

Recursive initialization is a design error: an initializer that reaches the same cell can deadlock or otherwise fail to make progress. Keep initialization dependencies acyclic and perform expensive fallible validation before committing the process-global value when possible.

More locks require a global acquisition story

One lock can cause contention; two can cause a cycle. If code sometimes acquires accounts then quotas, and elsewhere acquires quotas then accounts, two threads can each hold one and wait forever for the other. Rust prevents data races, not deadlock.

The cleanest fix is often to combine truly coupled state or redesign the operation so it does not need both guards simultaneously. When multiple locks remain necessary, establish a total order based on stable lock identity—subsystem rank, shard index, or ordered key—and acquire only in that order. Document exceptional paths. try_lock with rollback can avoid permanent waiting, but an immediate retry loop creates livelock; it needs bounded backoff, cancellation, and evidence that rollback is safe.

Sharding maps keys to locks and increases independent progress. It also introduces cross-shard operations, skew, resizing, and order. Acquire shard indices in sorted unique order; never lock in caller input order. Hash flooding or a hot tenant can collapse the expected distribution, so security and observability belong in the shard design. Record per-shard wait and hold distributions, but avoid publishing raw sensitive keys.

Lock coupling—holding one node or shard while acquiring the next—can protect a traversal invariant, but its safety case must state the order, deletion protocol, and what happens on failure between acquisitions. It is an advanced algorithm, not a generic way to walk a map.

Callbacks erase local lock-order reasoning. A callback can acquire another lock, call back into the component, block, panic, or await indirectly. Copy the callback inputs under the guard, release it, invoke the callback, then reconcile the result under a new guard if the operation supports optimistic change. If atomicity truly requires running foreign code inside the guard, the API needs a severe reentrancy contract and should usually be redesigned.

Blocking locks do not become async because the function is async

std::sync::Mutex::lock, RwLock::read, and Condvar::wait block the current OS thread. In an async executor, that thread may be responsible for polling many unrelated tasks. Holding a standard guard across .await is worse: the task can suspend while retaining the lock, and the awaited work may need the same state.

Use a synchronous lock inside async code only when the critical section is short, never crosses .await, and contention will not block executor workers beyond the service budget. Otherwise choose an async-aware mutex whose acquisition yields to the runtime, move blocking work to a dedicated blocking pool, use an owner task with a bounded channel, or redesign around immutable snapshots. Async-aware locks still permit deadlock, long holds, cancellation complications, and unfairness; they change how waiting consumes execution resources.

The primitive should follow the ownership of the protected resource, not the syntax of its callers. A memory-only statistics map updated for a few instructions may remain under a synchronous mutex even when called from async request handlers. A connection whose operations must await network readiness needs an async-native ownership design; wrapping the connection in a synchronous mutex merely moves suspension under a guard. A CPU-heavy index rebuild belongs on a blocking or compute pool, with a short synchronized snapshot swap at completion.

Mixed sync/async access needs one authoritative boundary. Maintaining a synchronous mutex for threads and a separate async mutex for tasks around the same logical state creates two gates that do not exclude each other. Route both caller classes through one owner, use one primitive safely accessible from both contexts, or divide the state into genuinely independent invariants. Adapters must not hold one lock while waiting to acquire the other.

Async-aware mutexes can have cancellation-sensitive acquisition queues. Dropping a waiting future may forfeit its place; whether the implementation is fair or cancellation-safe is a property of the named runtime primitive, not of Rust’s language semantics. Avoid encoding correctness in acquisition order unless the primitive explicitly guarantees it. If requests need tenant fairness or deadlines, build an admission policy rather than hoping lock wake order supplies scheduling.

Cancellation must be included. A future cancelled before acquisition changes nothing. Cancellation after acquiring and mutating state runs normal drop cleanup for the guard, but the domain transition may be incomplete unless mutation was commit-safe at every suspension point. The most robust rule is no suspension while a shared invariant is mid-transition.

Exercise: audit a cache that is race-free and operationally unsafe

You inherit a cache with one RwLock<HashMap<Key, Entry>>, a condition variable named updated, and a OnceLock<Client>. On a miss, code takes a write guard, calls a remote loader, inserts the value, invokes an eviction callback, calls notify_one, and releases the guard. A background task holds a read guard across .await while exporting every entry. Eviction locks a quota table after the cache; quota replenishment locks them in the opposite order. Poison errors use into_inner without validation. Operators see only hit rate.

Produce a review packet with these artifacts:

1. an invariant map naming every coupled field and the lock or one-time cell that protects it;
2. a lock-scope trace for hit, miss, failed load, concurrent miss, eviction, export, and shutdown;
3. a wait predicate and state-transition table that handles spurious wakeups and explains notify_one versus notify_all;
4. a total lock order or a redesign that removes the two-lock transaction;
5. a poison policy that identifies authority, validation, repair, logging, and when the component must stop;
6. a decision between one mutex, measured RwLock, immutable snapshots, sharding, or an owner task;
7. a callback and async-boundary revision with no unknown code or .await under a synchronous guard;
8. deterministic tests using barriers or channels, including failed loading, a poisoned transition, waiters, and shutdown—no timing sleeps;
9. metrics for acquisition wait, hold time, waiter count, refresh duration, poison recovery, shard skew, and stale-value age.

Reject a revision that merely shortens the visible block while returning a guard-backed reference whose lifetime retains the lock. Reject one that adds timeouts but leaves the state ambiguous after timeout. The deliverable is complete only when the protected invariant and every release path can be narrated without reading incidental control flow.

Shared-state review card

• Name the data relationship protected by each lock; do not say it protects “this code.”
• Identify the exact acquisition and release points, including values that retain guards.
• Move I/O, callbacks, joins, allocation spikes, and async suspension outside the guard.
• Pair every condition wait with one mutex-protected predicate and a recheck loop.
• Treat notification as a hint to re-evaluate, never as ownership of a resource.
• Choose and test a poison policy; never use advisory poisoning as a soundness boundary.
• Demand workload evidence before replacing a mutex with an RwLock.
• Use OnceLock only for genuinely one-time, long-lived publication.
• Define a total multi-lock order or eliminate overlapping acquisition.
• Measure wait and hold time separately, including shard and tail behavior.

The choice can be summarized compactly. Use a mutex for short exclusive transitions over one invariant. Consider an RwLock only for measured overlapping reads with an acceptable writer policy. Use a condition variable when synchronous threads must sleep until a mutex-protected predicate changes. Use OnceLock for immutable one-time publication. Use sharding when independent keys dominate and cross-shard rules are explicit. Use a channel owner when operations, queueing, and lifecycle are easier to defend than direct access. These are different contracts, not interchangeable performance switches.

A mutex makes access exclusive; it does not explain why a transition is valid, why waiting ends, or how failure repairs the state. Those claims live in the invariant, guard lifetime, predicate, order, and recovery policy. Chapter 56 narrows the primitive further. Atomics remove guards for small machine values, but then every cross-thread ordering edge must be justified directly.

Sources and version note

The behavior described here follows the Rust 1.97.0 standard-library documentation for Mutex, RwLock, Condvar, OnceLock, and PoisonError. The cache fixture uses Rust 2024 Edition, is dependency-free, and declares Rust 1.85.0 as its MSRV. Lock priority, wake order, scheduling, contention cost, and throughput are platform and workload properties rather than standard-library fairness guarantees. Poisoning is advisory and must not be relied on for unsafe-code soundness.