Atomics and the Memory Model

Two fields are atomic. One stores a payload; the other says the payload is ready. The writer stores the payload and then sets ready. The reader sees ready and reads the payload. Is the protocol correct?

“No data race” is not a sufficient answer. “The CPU will not reorder those instructions” is not a portable argument. Even “both operations are atomic” proves only that each atomic location is accessed indivisibly according to its operation; it does not automatically order the payload operation before the reader’s load.

The protocol is correct only under a narrower contract: one publisher, and a reader whose acquire load observes that publisher’s release. That verdict becomes reviewable when it names the edge:

A release operation publishes prior work only to a thread whose acquire operation observes that release or an appropriate release sequence. That synchronizes-with edge, combined with each thread’s sequenced-before order, establishes happens-before for the published work.

Atomics are excellent for observation counters and small coordination state machines. They are unforgiving when used as punctuation around a larger undocumented invariant. This chapter develops a proof vocabulary, not a ladder of “faster” orderings.

Atomicity answers one location-sized question

An atomic load observes a value from an allowed write to that atomic location without tearing. An atomic store replaces the location atomically. A read-modify-write operation such as fetch_add, swap, or successful compare_exchange reads a prior value and writes a new one as one atomic modification.

That property prevents data races on the atomic location when all accesses obey Rust’s atomic memory rules. It does not mean:

• two atomics form a transaction;
• a relaxed counter publishes another allocation;
• the newest wall-clock value must be observed immediately;
• a check followed by a store is indivisible;
• the algorithm is lock-free, wait-free, fair, or faster;
• adjacent atomics avoid cache-line contention;
• an atomic exists or supports the same operations on every target.

Rust’s atomic module describes data races as conflicting nonsynchronized accesses where at least one access is non-atomic. A data race is undefined behavior. Race freedom is still weaker than algorithmic correctness: two threads can atomically debit separate fields and violate assets == liabilities, or both observe an atomic state that permits duplicated work because the transition was split into load then store.

Use a mutex when the state is naturally a record whose fields must change together. Use an atomic when the invariant can be stated as one location or a small, explicit transition graph and every dependent access has a justified ordering edge.

Relaxed ordering is complete for independent observation

Ordering::Relaxed guarantees atomic access but imposes no inter-thread ordering constraints on other operations. That makes it appropriate for a metric whose only meaning is the observed atomic total:

pub fn record(&self) {
    self.0.fetch_add(1, Ordering::Relaxed);
}

pub fn snapshot(&self) -> usize {
    self.0.load(Ordering::Relaxed)
}

This counter can underlie “events recorded so far as observed by this load.” It must not decide whether all event payloads are visible, whether a queue is empty, whether shutdown is complete, or whether another object may be freed. The counter update does not publish those facts.

Relaxed read-modify-write operations on one counter still participate in that atomic location’s modification order, so increments are not lost through a non-atomic read/write race. A snapshot taken concurrently remains transient. If it is exported with another relaxed counter, the pair may not correspond to one instant. Chapter 54’s accepted - completed metric was deliberately approximate operational evidence, not permission to receive a job or reclaim memory.

This distinction is useful in review. Ask, “If this load returns any value allowed by concurrent execution, can it change correctness?” If the answer is no and no other memory is being published, relaxed may be enough. If the value grants access, transfers ownership, completes initialization, or changes a lifecycle, the proof must name additional order.

Release and acquire create a publication edge

Consider the fixture’s one-writer reading:

pub struct PublishedReading {
    payload: AtomicU64,
    ready: AtomicBool,
}

pub fn publish(&self, value: u64) -> Result<(), u64> {
    if self.ready.load(Ordering::Relaxed) {
        return Err(value);
    }
    self.payload.store(value, Ordering::Relaxed);
    self.ready.store(true, Ordering::Release);
    Ok(())
}

pub fn read(&self) -> Option<u64> {
    self.ready
        .load(Ordering::Acquire)
        .then(|| self.payload.load(Ordering::Relaxed))
}

The external contract permits exactly one publisher. That matters: the initial relaxed check is not a multi-writer claim operation. With multiple publishers, two threads could both see false and overwrite the payload. A compare_exchange on the flag would need a state such as empty → writing → ready so no reader observes ready before payload construction completes.

For the one-writer protocol, the ordering proof is:

1. The payload store is sequenced before the release store to ready in the writer.
2. A reader’s acquire load observes true from that release store.
3. The release store synchronizes with that acquire load.
4. The acquire load is sequenced before the reader’s payload load.
5. By transitivity, the payload store happens before the payload load.

The payload is atomic here so the example can remain safe and dependency-free while isolating the ordering edge. In real object publication, safe primitives such as OnceLock, a mutex, a channel, or an Arc handoff should normally own the difficult lifetime and initialization proof. Publishing non-atomic data through raw memory requires unsafe-code invariants beyond this chapter.

The memory aid turns the five statements into one graph. Follow the solid arrows within a thread, then the release/acquire synchronizes-with arrow across threads. No arrow means no visibility argument.

Release applies to operations capable of storing; acquire applies to operations capable of loading. A release store does not force every future acquire load in the program to synchronize with it. The acquire must observe the relevant release value through the atomic’s allowed value history. Loading false gives no publication edge and must not expose the payload.

Acquire/release is directional. It publishes earlier writer operations to later reader operations. It does not turn all operations before and after both threads into one global timeline. Nor does it make an unrelated relaxed flag safe as a second protocol.

Read-modify-write operations carry a load side and a store side

Operations such as fetch_add and compare_exchange both read and potentially write. Their ordering must be interpreted on both halves.

• Acquire on a successful read-modify-write gives acquire behavior to the load; its store half is relaxed.
• Release gives release behavior to the store; its load half is relaxed.
• AcqRel provides acquire behavior for the load and release behavior for the store.
• SeqCst adds the sequentially consistent ordering guarantees described later.

This split explains why blindly selecting AcqRel is not a proof. The operation may need only atomic modification for a counter, only release publication for completion, acquire access after claiming a published node, or both because it closes one phase and opens another.

The fixture’s state machine permits Idle → Running → Finished:

self.0.compare_exchange(
    JobState::Idle as u8,
    JobState::Running as u8,
    Ordering::AcqRel,
    Ordering::Acquire,
)

compare_exchange is one conditional transition. It returns Ok(previous) when it writes the new value and Err(observed) when the comparison fails. The success ordering governs the read-modify-write. The failure ordering governs a load because no store happened; Rust therefore permits only load-compatible failure orderings such as Relaxed, Acquire, or SeqCst, not Release or AcqRel.

The example uses acquire on failure because a caller inspecting an already advanced state might need facts published by the winning transition. If it merely returns false without consulting dependent data, relaxed failure can be sufficient. The code should encode the real contract rather than copying a standard pair.

compare_exchange_weak may fail spuriously even when the expected value matches. It is useful in a loop that already handles observed-value changes; on some targets that can be more efficient. A one-shot claim normally wants strong compare_exchange, because treating spurious failure as “someone else won” would be a semantic error.

CAS loops must reconsider the operation after failure. A loop that computes new once and retries against changing old can overwrite intervening updates. Pointer algorithms face the ABA problem: a location can change from A to B and back to an indistinguishable A while the underlying object identity or generation changed. Solving reclamation and ABA safely requires a named algorithm and memory-reclamation strategy; adding SeqCst does not solve object lifetime.

Sequential consistency adds a global order, not a global transaction

Ordering::SeqCst provides acquire/release behavior appropriate to the operation and adds a single total order observed consistently for sequentially consistent operations. This often makes a litmus outcome easier to rule out and can be the right default while establishing correctness.

It still does not:

• make a sequence of several operations indivisible;
• include relaxed operations in the same strong total-order guarantee;
• repair an incorrect lifecycle or memory-reclamation scheme;
• guarantee progress or fairness;
• guarantee a particular instruction cost on every target;
• mean that a load returns the most recent value by wall-clock time.

Replacing every ordering with SeqCst can make some ordering bugs disappear, but it can also hide that the design lacks one state owner or one invariant boundary. If correctness requires two locations to change atomically, a mutex or a packed, justified single-atomic representation may be the real repair. Packing fields introduces width, overflow, encoding, and compare-exchange retry costs; it is not automatically simpler.

Use the weakest ordering whose proof is clear only after there is a proof. “Weakest” is not a style prize. A team may standardize on SeqCst for rare coordination paths to reduce review risk, or on safe higher-level primitives to avoid local proofs entirely. Performance changes require target measurements because ordering cost varies by architecture and surrounding code.

Several atomic fields do not yield one snapshot

Operational structures often expose accepted, started, completed, and failed as separate atomics. A reader that loads them in that source order has four values, not one atomic sample. Concurrent updates can produce combinations that never existed at a single instant. Even if every load is SeqCst, another thread’s modifications can occur between those loads.

This can be acceptable for dashboards if every field is labeled approximate and derived arithmetic is saturating or otherwise robust to transient inversion. It is unacceptable when equality authorizes reclamation, billing settlement, failover, or shutdown completion.

There are several real repairs:

• Put the related fields behind one mutex and clone a coherent snapshot under one guard.
• Encode the complete bounded state in one atomic word and update it with a justified CAS loop.
• Use a versioned snapshot protocol in which readers detect concurrent mutation and retry, with a formal argument for wrap and memory ordering.
• Let one owner serialize transitions and publish immutable snapshots.
• Weaken the consumer contract so independent approximate values are genuinely sufficient.

Packing values is not free. If two 32-bit counters share a 64-bit atomic, the target must support the width and needed operation. Field overflow can corrupt the neighbor if masks are wrong. CAS retry work grows under contention. Adding a generation consumes bits and eventually wraps. The representation becomes a public maintenance burden even if the type is private because operators and tests depend on its semantics.

A common read-twice scheme loads a generation, copies data, then loads the generation again. It is not automatically safe for non-atomic data: concurrent reads and writes to ordinary memory may already be a data race, and an equal wrapped generation does not prove no mutation. Such schemes require an established algorithm with safe storage and reclamation, not just two acquire loads.

Metrics have another subtlety: ordering cannot create a coordinated timestamp. An acquire load may establish visibility of work published by a release, but it does not mean every unrelated counter is refreshed to the same logical time. If a trace requires causal correlation, carry an operation or generation identifier through the synchronized protocol instead of inferring causality from sampled totals.

The review decision should name the consumer. An alert estimating backlog can tolerate a fuzzy pair. A destructor deciding no worker retains an object cannot. The same atomic code can be adequate observability and invalid lifecycle coordination.

Happens-before is the review graph

Happens-before is not merely chronological prose. It is the relation that lets a reviewer connect operations across threads and decide which conflicting accesses are synchronized and which writes a reader may rely on.

Useful edges come from several places:

• operations sequenced within one thread;
• release/acquire pairs that actually communicate through an atomic;
• mutex unlock followed by a successful acquisition of the same mutex;
• channel send/receive and thread spawn/join according to their documented contracts;
• one-time initialization primitives according to their APIs.

Do not infer an edge from logging order, debugger display, wall-clock timestamps, thread scheduling on one run, or “the writer probably finished first.” A compiler and CPU may exploit freedoms that preserve single-thread behavior and the language model. The program must state cross-thread communication through synchronization.

Draw one node per access that matters. Label the atomic location, operation, ordering, and value observed. Add sequenced-before arrows, then only documented synchronizes-with arrows. Ask whether a path exists from every initialization or mutation to each dependent read. If the graph needs “obviously” as an unlabeled arrow, the protocol is unfinished.

The Rust Reference warns that Rust’s memory model is incomplete and not fully decided. The standard atomic documentation defines its operations in close relation to the C++20 atomic model. That is a reason to keep proofs within documented primitives and avoid inventing conclusions about provenance, mixed atomic/non-atomic access, or unsafe object publication.

Litmus tests explore executions; they do not certify orderings

A litmus test is a small concurrent program designed to ask whether an outcome is allowed or observed. For example, two threads can each store to one atomic and load the other. Under relaxed ordering, an execution where both loads observe the initial value is the classic warning that per-location atomicity does not create a cross-location order.

Running the test a million times and never seeing that result does not prove it impossible. The host architecture may be stronger than the language requirement, the compiler version may make different choices, and scheduling may never expose the window. Conversely, observing a permitted surprising result is useful evidence that an assumption was false.

Use three layers of evidence:

1. Model argument: state why the desired outcome follows from documented ordering and happens-before rules.
2. Concurrency exploration: use an appropriate state-space tool or model to enumerate relevant interleavings when available, with the tool and version recorded.
3. Stress and target testing: look for integration defects, contention, starvation, and platform-specific behavior without presenting absence of failure as proof.

Tests should use barriers, channels, and joins to establish their own setup; sleeps produce scheduling anecdotes. A test that inserts an extra channel may accidentally synchronize the very accesses it intends to study. Keep orchestration outside the measured protocol and audit every helper primitive for unintended edges.

Counters and coordination deserve separate types

Atomics used for metrics should not be reused for safety decisions. A relaxed active_requests gauge can be approximate and may be sampled independently from accepted. A shutdown gate, generation publication, or ownership state must have a precise transition contract.

Separate types make misuse harder:

• ObservationCounter exposes record and snapshot, not is_safe_to_delete.
• PublishedReading exposes a one-writer publish/read protocol.
• AtomicJobState exposes legal transitions rather than a public AtomicU8 and magic constants.

The wrapper is not proof by itself. Its documentation must state number of writers, allowed transitions, dependent data, ordering rationale, overflow behavior, and what a failed CAS means. Keep the atomic private so callers cannot bypass the transition graph.

Counter overflow is part of the contract. Unsigned atomic arithmetic wraps according to the operation’s integer semantics. A process-lifetime telemetry count may accept wrapping only if downstream rate calculations tolerate it; a ticket or generation number may need checked reservation, a larger supported width, epoch handling, or process restart before exhaustion. Sequential consistency does not prevent arithmetic wrap.

Flags must describe a lifecycle, not a mood

An AtomicBool named shutdown, ready, or cancelled compresses a potentially rich protocol into one bit. Before choosing its ordering, ask what each value authorizes and whether transitions can reverse.

A cancellation flag checked with relaxed loads may be sufficient when it is only a best-effort request to stop and no data is published through it. The worker may perform more iterations before observing true; the contract must tolerate that. If setting the flag also publishes a reason, deadline, or cleanup plan stored elsewhere, release/acquire or a higher-level message is needed. If cancellation races with an irreversible commit, one bit cannot by itself decide which side won unless the state machine encodes that boundary atomically.

A readiness flag normally has at least three conceptual states: uninitialized, initializing, and ready. A two-state multi-writer protocol can expose “ready” too early or allow duplicate initialization. OnceLock already implements one-time initialization and publication safely. Replacing it with an atomic flag plus raw or interior storage expands the proof to initialization, panic, retry, aliasing, and object lifetime.

Shutdown is rarely boolean at the component level. Intake-open, draining, forced-stop, and stopped have different permissions. A private AtomicU8 wrapper can encode legal transitions, but a mutex plus condition variable or an owner channel may better express waiters, deadlines, accepted-work accounting, and acknowledgements. The atomic only changes a number; joining workers or observing the stopped state is what proves lifecycle completion under the relevant synchronization contract.

Memory ordering also cannot guarantee prompt observation. A worker polling an acquire flag in a tight loop can waste a core and contend on a cache line. A worker polling infrequently can miss the service’s shutdown deadline. Pair state with an appropriate wake mechanism, blocking primitive, runtime notification, or channel when latency and resource use matter. The wake mechanism and atomic state must form one protocol so a notification cannot be lost between checking and sleeping.

Polling also needs compiler-visible work. A loop that repeatedly loads an atomic uses an observable atomic operation, but inserting spin_loop is only a processor hint; it does not add a synchronization edge or a bounded retry policy. Spin waiting is defensible only for a short, measured interval when the owner is guaranteed to run and oversubscription is controlled. After that interval, park, yield through an appropriate scheduler, or block on a notification. A spin lock in user code inherits panic, preemption, priority inversion, and fairness problems while discarding the operating system mutex’s sleeping behavior.

Fences are another place where compressed syntax hides proof. An acquire or release fence can participate in synchronization patterns involving atomic operations, but it does not act as a universal flush instruction and does not make ordinary racing accesses safe. Prefer acquire loads and release stores on the communicating atomic when they express the protocol directly. Introduce fence only with a graph showing which atomic reads and writes connect it and why per-operation ordering is insufficient. compiler_fence constrains compiler reordering for specialized contexts; it does not provide inter-core hardware synchronization. Signal handlers, device memory, DMA, and FFI add platform contracts outside a routine shared-memory flag.

Name transitions with methods—try_begin_draining, mark_stopped, is_cancel_requested—rather than exposing the atomic. Each method should document whether failure means another participant won, the state was already advanced, or the observed value was invalid. This turns ordering review into lifecycle review, where it belongs.

False sharing turns independence into coherence traffic

Two atomics can be logically independent and physically close enough to share a cache line. If different cores repeatedly write them, the coherence protocol may transfer ownership of that line even though the fields never interact. Throughput falls while the source code appears perfectly sharded. This is false sharing.

Possible repairs include per-thread or per-shard counters with periodic aggregation, separating frequently written fields, batching updates, or using alignment/padding wrappers. Each costs memory, aggregation complexity, snapshot staleness, or portability. Do not hard-code a universal cache-line size without a supported-hardware contract. Layout attributes and padding must be verified on targets, and a larger structure can damage cache density elsewhere.

False sharing is diagnosed with measurement: hardware counters where available, scaling curves by thread count, field-layout inspection, and controlled padding experiments. A benchmark must record toolchain, target, CPU topology, affinity policy, build profile, workload, and variance. A faster padded microbenchmark does not prove end-to-end value if the production path is dominated by allocation or I/O.

Atomics can also create true sharing hotspots. One global fetch_add makes every writer modify the same location. The operation may be lock-free while overall throughput serializes on cache-line ownership. Lock-free describes a progress property of an algorithm, not the absence of a hardware bottleneck.

Portability includes availability and progress

Atomic integer widths and operations vary by target. The standard library documents that available atomic types are lock-free, but not necessarily wait-free; an operation may use a compare-and-swap loop. Some targets lack 64-bit atomics, and some embedded targets support atomic loads/stores without compare-and-swap. cfg(target_has_atomic = "64") and related configuration allow conditional compilation by supported width.

AtomicUsize follows pointer width, so it is not a portable substitute for a required 64-bit generation. If a wire protocol or persistent format requires 64 bits, the fallback must be explicit: a lock, critical section supplied by the platform, different representation, or unsupported target. Do not silently truncate.

Lock-free also does not mean wait-free. Lock-free systems guarantee that the system as a whole makes progress under the algorithm’s conditions; one thread may starve. Wait-free promises bounded completion for each participant and is substantially stronger. A single atomic instruction can still wait on caches or platform machinery. State the progress requirement separately from memory safety and ordering.

Portability review should include target availability, alignment, mixed-size overlap, signal or interrupt context, process-shared requirements, compiler version, and whether the primitive is actually in std on the target. Rust’s safe atomic APIs prevent many invalid accesses, but unsafe conversions or FFI can reintroduce alignment, lifetime, and mixed atomic/non-atomic hazards.

Prefer a lock when the proof stops being small

Replace an atomic design with a mutex, channel, or one-time cell when any of these is true:

• the invariant spans several independently meaningful fields;
• readers need one coherent snapshot rather than independent values;
• ownership or reclamation of heap objects is involved;
• the state graph has many exceptional or rollback transitions;
• fairness, waiting, cancellation, or backpressure is central;
• reviewers cannot explain every ordering without reconstructing the algorithm from code;
• the performance advantage is assumed rather than measured;
• target fallbacks dominate the implementation.

A mutex provides exclusion plus synchronization around arbitrary safe Rust data. It also provides a natural place to validate invariants and instrument wait/hold time. Its costs are blocking, contention, poisoning policy, and possible deadlock. Those costs may be far less than the maintenance risk of an unjustified lock-free protocol.

A channel is often better when one owner can serialize transitions and callers can tolerate an explicit queue and reply lifecycle. OnceLock is better for one-time immutable publication. Atomics remain ideal for compact state such as a cancellation flag, monotonically observed epoch, statistics, reference-count machinery inside a proven abstraction, or a tiny claim/completion machine.

“Lock-free” should never be an aesthetic requirement. Ask which service objective or execution constraint the lock violates, then measure the alternative’s throughput, latency tails, starvation, cache traffic, and code-review burden.

Exercise: put every ordering claim on trial

Review a work registry with these fields:

struct Registry {
    slots: Vec<Slot>,
    initialized: AtomicBool,
    next: AtomicUsize,
    completed: AtomicUsize,
    state: AtomicU8,
}

The implementation initializes slots, stores initialized with relaxed ordering, lets workers claim indices with load followed by store, increments completed with SeqCst, and frees the registry when completed == next. A comment says the design is safe because “all shared numbers are atomic and sequential consistency creates a memory barrier.” The target set includes x86-64 Linux and an embedded target whose atomic widths have not been checked.

For each claim below, mark justified, insufficient, or false, then provide the missing proof or replacement design:

1. Atomic initialized makes preceding writes to slots visible under relaxed ordering.
2. Separate atomic load and store claim a unique index.
3. SeqCst completed makes the independently loaded pair completed == next one coherent snapshot.
4. Seeing completion permits immediate reclamation even if workers retain slot references.
5. A successful CAS with AcqRel and a relaxed failure ordering is always the right pair.
6. Not observing a relaxed litmus outcome on x86-64 proves the protocol portable.
7. Available Rust atomics are wait-free.
8. Padding every atomic to 128 bytes must improve throughput.

Deliver:

• a transition table naming legal states, writers, readers, and failure paths;
• a happens-before graph for initialization and for completion;
• revised Rust code using either a documented atomic protocol or a lock/channel/OnceLock design;
• explicit success and failure orderings for every CAS, with one sentence for each half;
• a lifetime and reclamation argument independent of counter equality;
• a target-availability matrix using target_has_atomic evidence;
• model, concurrency-exploration, stress, and performance test plans, each labeled with what it can and cannot prove;
• a false-sharing experiment that records layout, CPU topology, affinity, toolchain, profile, and variance.

The strongest answer may remove most atomics. A mutex-protected registry plus an observation-only relaxed counter can be easier to verify and fast enough. If an atomic design remains, no arrow in its happens-before graph may be labeled “the barrier handles it.”

Atomic review claims worth keeping

• Atomicity is per access and location; composite invariants need an explicit representation.
• Relaxed is sufficient for independent observation, not publication or ownership.
• A release publishes only through an acquire that observes the relevant release history.
• Read-modify-write ordering has distinct load and store effects.
• Failed compare-exchange is a load and needs a load-compatible ordering.
• SeqCst adds a consistent total order for sequentially consistent operations, not a transaction over arbitrary state.
• Happens-before paths, not source order or wall-clock intuition, justify visibility.
• Litmus and stress runs find evidence; absence of an outcome is not a semantic proof.
• Lock-free does not imply wait-free, fair, uncontended, or fast.
• Portability includes atomic width, operation availability, alignment, and fallback behavior.
• When the invariant or lifetime proof grows, prefer a higher-level synchronization primitive.

One final ordering-selection discipline helps in code review. Start each operation at Relaxed. Add Release only when the operation publishes earlier work. Add Acquire only when later work consumes data from an observed publication. Use AcqRel when one read-modify-write truly consumes an earlier phase and publishes a later one. Use SeqCst when the algorithm needs its operations to participate in the additional consistent total order or when a deliberately stronger, simpler policy is worth the cost. Then test the explanation by deleting each claimed edge from the graph: if correctness supposedly remains unchanged, that ordering was not the reason; if correctness fails but no dependent data or transition can be named, the design is relying on folklore.

Atomics remove the guard that made Chapter 55’s protected interval visible. In exchange, the design must expose a state graph and ordering graph precise enough for another engineer to verify. Chapter 57 uses that precision to ask a broader question: even if race freedom and ordering are correct, does the system remain live and scale under contention?

Sources and version note

The guarantees and terminology here follow the Rust 1.97.0 documentation for std::sync::atomic, Ordering, and the atomic integer operations, together with the Rust Reference memory-model chapter. The atomic documentation relates Rust orderings to C++20 and documents data races, mixed-size access limits, portability, and lock-free availability; the Reference explicitly notes that Rust’s overall memory model remains incomplete. The fixture is safe Rust 2024, verified on Rust 1.97.0 with Rust 1.85.0 as MSRV. Instruction selection, ordering cost, cache-line size, contention, and observed litmus outcomes are target and toolchain properties, not portable performance guarantees.