Fuzzing, Sanitizers, Miri, and Concurrency Exploration

The most valuable output of a week-long fuzz campaign may be three bytes:

00 01 ff

Those bytes are not yet evidence. Without the target revision, features, toolchain, engine configuration, original failure, minimized failure, and a command that reproduces the behavior, they are only a souvenir. If the crash came from undefined behavior, an ordinary regression may stop failing after an optimizer change. If it came from a race, the bytes may reproduce the input but not the schedule. If minimization removed the condition that matters, the tiny artifact may even describe a different defect.

Dynamic analysis is therefore an evidence pipeline, not a menu of powerful commands. Choose a tool for a named defect class, bound the state space it explores, retain enough execution context to replay a finding, and promote the result to the cheapest deterministic layer that can prevent recurrence. Ordinary tests establish expected behavior. Fuzzers vary inputs, interpreters inspect abstract-machine rules, sanitizers instrument concrete executions, and schedule explorers vary ordering. Their overlap is useful, but none makes the others redundant.

Give the target a stronger job than surviving

A fuzz target translates arbitrary engine input into one meaningful operation. The shortest possible target often calls a parser and ignores the result. That can find panics and some memory errors, but it leaves semantic defects invisible. A production target should include a cheap oracle whenever one is available.

The chapter lab uses a length-prefixed frame:

pub fn fuzz_one(bytes: &[u8]) {
    if let Ok(payload) = decode_frame(bytes) {
        assert_eq!(encode_frame(payload).as_deref(), Some(bytes));
    }
}

The target accepts every byte slice, so the engine does not need to construct a domain object first. When decoding succeeds, re-encoding must reproduce the canonical frame exactly. This checks more than absence of panic: accepted length, consumed bytes, payload boundary, and canonical encoding must agree. A second target could mutate valid semantic payloads and assert decode(encode(value)) = value; the two targets search different neighborhoods.

Keep each target narrow enough that coverage feedback reflects progress toward one contract. A target that initializes a database, starts a runtime, parses a frame, authenticates it, and writes a response spends most executions in setup and makes failures hard to classify. Extract pure or in-memory seams, seed expensive state once when the engine permits it, and cap input size, recursion, collection growth, decompression ratios, and operation counts. Resource exhaustion may be a legitimate security finding, but it should not accidentally starve the search.

Do not repair malformed input so aggressively that the target never reaches rejection logic. Structure-aware generation is valuable for checksums, nested syntax, or protocols, yet a raw-byte target still explores broken lengths, truncated headers, invalid tags, and surprising transitions. A useful campaign often contains both: raw targets for hostile boundaries and structured targets for deep valid states.

Treat the corpus as executable search policy

The initial corpus gives the engine starting points. Begin with the smallest valid and invalid representatives: empty input, one-byte truncation, minimum valid frame, maximum header with absent body, known version tags, boundary lengths, dictionary tokens, and previously observed formats. Large production captures can delay mutation and contain secrets; reduce and sanitize them before retention.

A dictionary supplies tokens worth inserting—magic bytes, separators, keywords, enum tags, protocol verbs, and common length encodings. It helps the engine cross shallow syntactic checks, but it is not a specification. Update it when the grammar changes, keep tokens scoped by target, and measure whether it opens deeper behavior rather than assuming more entries are better.

Coverage guidance answers “did this input reach a new instrumented path or comparison?” It does not answer “is this path correct?” Two targets can report similar edge coverage while one has an independent oracle and the other merely avoids a crash. Coverage can also plateau because the harness blocks progress, the search needs a token, checksums reject mutations, state setup is too expensive, or the remaining paths are infeasible. Investigate the frontier; do not turn one percentage into a quality gate detached from contracts.

Store the curated seed corpus in the repository when it is small and non-sensitive. Store a larger evolving corpus in versioned artifact storage with hashes, retention, and an owner. A missing remote corpus must fail or clearly degrade the deep job; silently starting empty wastes accumulated search knowledge.

Minimize without destroying the explanation

When an engine finds a crash, first preserve the original artifact and full run metadata. Then minimize. Input minimization deletes or simplifies bytes while retaining the same observable failure. Stack-trace deduplication and top-frame hashes are triage hints, not identities: one defect can produce many stacks, and unrelated defects can share a terminal assertion.

Re-run the minimized case under the original instrumentation and an ordinary debug build. Classify it before filing:

1. Is the behavior reproducible at the recorded revision and configuration?
2. Is it a target defect, an unsupported tool operation, a resource-budget breach, undefined behavior, a semantic assertion, or a dependency failure?
3. Does the minimized artifact preserve the same failure as the original?
4. What is the smallest owning layer for a regression?
5. Does the finding imply a wider audit, backport, disclosure, or corpus update?

Promote a parser mismatch into an exact unit or integration test with an explicit expected result. Keep the fuzz artifact too when it helps the engine rediscover nearby states. For undefined behavior, retain the Miri or sanitizer invocation because a normal test that happens to return the expected value does not prove the invalid execution disappeared. For concurrency, retain the controlled schedule or model-checker trace; input alone is incomplete.

Use Miri for a selected abstract-machine subset

Miri interprets Rust’s mid-level representation and can detect classes of undefined behavior in executed paths, including invalid memory use and some concurrency violations. It is especially valuable for small unsafe kernels, pointer manipulation, initialization and destruction, slice construction, custom collections, and regressions from an unsafe safety case. Run public safe-API tests through it as well as private unsafe helpers: Chapter 74 established that safe code can violate an unsafe block’s non-local assumptions.

Miri is not a fast alternate test runner. Interpretation is expensive, platform interaction is incomplete, and unsupported operations are not product defects. Select deterministic tests that exercise safety-sensitive state transitions with modest data sizes. Keep file, network, process, FFI, and runtime-heavy tests in their native layers unless the current pinned Miri supports the exact operation and the test earns its cost.

Pin a dated nightly when reproducibility matters, record Miri flags, and distinguish tool-model experiments from established Rust guarantees. Multiple Miri seeds can vary allocation placement and thread interleavings, increasing explored executions without exhausting them. A clean run means no detected violation in those executions under that tool version and model. It is not a proof that all executions are sound.

The lab itself contains no unsafe code, so Miri is not required to justify it. The batch still attempts a small cargo miri test subset to verify tool availability and to demonstrate honest reporting. “Component unavailable” is an environment fact to schedule and fix, not permission to claim the Miri gate passed.

Build a sanitizer matrix around mechanisms and platforms

Sanitizers instrument compiled native code and supporting runtimes. Their value differs by compiler support, target, linkage, dependencies, and workload. Address-oriented instrumentation can reveal invalid accesses in executed native paths. Leak detection can expose lost ownership where supported. Thread-oriented instrumentation can reveal data races in native executions. Memory-initialization tooling may have stricter compiler and platform constraints. Names and availability evolve; verify the pinned Rust and target documentation rather than copying a matrix from another repository.

Do not multiply every sanitizer by every test and platform. Select jobs by where the relevant mechanism exists:

Instrumentation changes timing, allocation, layout, and performance. A sanitizer failure deserves investigation; a sanitizer success covers only the executed workload in that altered environment. Keep at least one ordinary native run for the real build mode, and avoid using instrumented latency as a service performance verdict.

Race detectors apply to concrete executions and supported synchronization mechanisms. They may not understand every custom primitive or foreign runtime. Miri and model checkers reason under different execution models. Agreement across them raises confidence because their blind spots differ; disagreement requires classification, not majority vote.

Explore schedules instead of adding sleeps

Concurrency defects depend on order, not merely input. Repeating a stress test can find rare failures, but it samples schedules chosen by an OS, runtime, workload, and instrumentation. Adding sleeps perturbs probability without describing the ordering being tested.

The lab models two threads that each load, increment, and store a shared counter. It enumerates every interleaving that preserves each thread’s program order. The possible final values are {1, 2}: some schedules lose an update. This is an abstract deliberately racy algorithm, not executable shared-memory Rust. Its value is explanatory: schedule exploration needs a state model, operation boundaries, and an invariant.

Real model checkers can control synchronization choices, atomics, or tasks and replay a failing schedule. Keep models small: two or three threads, short histories, bounded queues, bounded retries, and explicit state. State-space growth is combinatorial. Use symmetry reduction, preemption bounds, targeted yields, or algorithm-specific operation boundaries, and report the bound. “All schedules passed” is meaningful only with the finite model named.

Weak-memory behavior needs special care. Sequential interleavings do not model every hardware/compiler ordering permitted by atomics. Use a tool and model appropriate to the memory-ordering claim, preserve Chapter 56’s happens-before reasoning, and test on real supported architectures where platform behavior matters. No schedule tool rescues an underspecified invariant.

Split fast detection from deep search

Presubmit work should provide useful failures before reviewers stop waiting. Keep deterministic regressions, a bounded fuzz smoke corpus, selected Miri safety tests when affordable, and perhaps one high-risk instrumented job. Move rotating fuzz campaigns, broad sanitizer workloads, multiple Miri seeds, model-checker expansions, and platform races into scheduled capacity.

A credible weekly deep-verification job has a contract, not just a cron expression:

• pin compiler, components, engines, sanitizer flags, target images, dependencies, corpus version, and model bounds;
• allocate per-target time and memory budgets so one pathological target cannot consume the week;
• shard by target while retaining one manifest that says which shards completed;
• upload original and minimized failures, stdout/stderr, symbolized traces, seeds, schedules, coverage deltas, and exact replay commands;
• deduplicate provisionally, then assign human ownership by subsystem and defect class;
• fail conspicuously on missing components, missing corpora, zero executions, or expired credentials;
• promote confirmed failures into stable regressions and track time-to-triage, not merely crash count;
• quarantine only with an owner, reason, expiry, and preserved failing evidence.

Long-running jobs should be resumable. Corpus checkpoints and deterministic model bounds make an interrupted run observable rather than ambiguous. A dashboard that reports “green” after every fuzz worker failed during setup is worse than no dashboard.

Design the weekly campaign

For a zero-copy protocol library with a small unsafe cursor and a bounded concurrent queue, write a one-page campaign:

1. Name three fuzz targets, their input domains, maximum sizes, oracles, seed corpora, dictionaries, and hourly budgets.
2. Select the public tests and unsafe-kernel tests that run under a pinned Miri, including seed count and unsupported exclusions.
3. Choose sanitizer/target pairs only where the current toolchain supports them; name the workload each pair observes.
4. Define a bounded queue schedule model: threads, operations, capacity, preemption or history bound, and invariant.
5. Specify every retained failure field and the deterministic regression destination.
6. Set triage ownership, severity rules, quarantine expiry, and the response when a tool silently executes zero cases.

Reject a proposal that says only “run fuzzing overnight.” The design is complete when another engineer can reproduce a finding, understand the explored boundary, and tell which cheaper gate should retain it.

Audit dynamic-analysis evidence

Before trusting a campaign, ask:

• Does each target name a contract and include the strongest cheap oracle available?
• Are valid, invalid, boundary, and historical inputs represented without sensitive production data?
• Can coverage guide search without being mistaken for correctness?
• Are original and minimized artifacts both retained and classified as the same failure?
• Is every Miri claim scoped to executed tests, flags, model, and pinned nightly?
• Does each sanitizer job match a supported target and a workload that reaches the mechanism?
• Are stress results distinguished from controlled, replayable schedules?
• Are model bounds and weak-memory assumptions explicit?
• Do scheduled jobs fail visibly when tools, corpora, or workers are missing?
• Does every confirmed discovery become a stable regression at the cheapest correct observer?

Generators, models, and oracles establish what correctness means. Dynamic tools extend their reach into hostile bytes, abstract-machine violations, native instrumentation, and alternative schedules. The durable outcome is still an evidence artifact a maintainer can replay. The remaining release decision is which jobs belong in fast gates, which require scheduled capacity, and which should remain advisory or release-only.

Sources and version notes

Miri and sanitizer behavior is toolchain- and target-sensitive. Pin actual versions, consult current support documentation, and record unavailable attempts. Third-party fuzzing and model-checking engines are implementation choices; the target, oracle, corpus, bound, triage, and regression principles are independent of one engine.

• Miri repository and usage guide
• The rustup book: components
• The Rustonomicon: working with unsafe code
• The Rust Unstable Book