Test Architecture: Unit, Integration, Documentation, and UI Tests

A repository can contain ten thousand passing tests and still leave its most important promises unobserved. A private helper may be exhaustively checked while the public API wires it incorrectly. A library test may call main-adjacent functions while the shipped binary mishandles exit status. A compile-fail example may prove that some program is rejected while failing to protect the diagnostic that users depend on. A Linux-only suite may quietly turn “portable” into an aspiration.

Part XIII changes the question from “does the code compile?” to “what evidence supports each contract?” Rust’s type system, ownership rules, and unsafe boundaries establish strong facts, but production confidence is layered. The first architectural decision is therefore not which test framework to adopt. It is which observer must witness each promise, and where that observer can run without crossing a false boundary.

That principle yields a practical rule: put a test at the lowest layer that can observe the entire contract, but no lower. Test a private state transition beside its implementation. Test a public compatibility promise from another crate. Test command-line behavior as a process. Test rejected source with the compiler. Test platform behavior on the platform. Duplication is warranted only when two layers protect different consequences.

Map contracts before arranging directories

The familiar testing pyramid is useful only after adapting it to Rust’s contract surfaces. “Small, medium, large” says little about what has been proved. A unit test and an integration test may execute the same instructions but observe different APIs. A doctest may be tiny yet protect a public teaching path. A one-line UI fixture may defend a central type-level guarantee.

Use five fields for every important requirement:

This is contract-to-test traceability. It is more useful than a raw coverage percentage because it exposes missing observers and unjustified environments. Coverage can tell you a line executed; it cannot tell you that an external caller, a process supervisor, or a downstream MSRV build saw the promised behavior.

Keep implementation invariants close

Unit tests compiled inside a library or binary target can access private items in their module hierarchy. That privilege is valuable when the contract itself is private: an internal state transition, normalization step, parser cursor, cache eviction rule, or representation invariant. The lab checks that a rejected debit changes neither balance nor accepted-command count:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn rejected_debit_preserves_private_state() {
        let mut ledger = Ledger::new();
        assert!(!ledger.apply(Command::Debit(1)));
        assert_eq!(ledger, Ledger::default());
    }
}

The name states condition and consequence. If the test fails, the structural equality shows the whole small state rather than reporting only an unexplained boolean. This test belongs beside Ledger because an external caller should not acquire accessors merely to inspect bookkeeping.

Private access becomes a liability when it lets a test counterfeit a public guarantee. If a persistence test inserts rows by calling an internal decoder, it has not proved that the supported import API accepts them. If an integration test needs pub(crate) hooks to assemble every state, first ask whether the production boundary is too entangled. A narrow private unit test is legitimate; widening production visibility only for tests creates a second API that can drift from real use.

#[cfg(test)] is appropriate for test modules and helpers required only while the target is built in test mode. It is not active when the crate is built merely as a dependency of an integration test, so public items hidden behind cfg(test) are not a general bridge into tests/. A test-support feature makes helpers available more broadly but becomes a feature and maintenance contract; use a separate support crate when multiple packages genuinely share builders, fake services, or protocol fixtures.

Make public promises cross a crate boundary

Each file under tests/ is compiled as a separate crate linked against the library. It sees what a downstream crate sees: public names, trait implementations, feature configuration, and externally observable behavior. That makes integration tests the correct home for public API composition and compatibility claims.

The lab’s reference-model check imports only public items:

use testing_strategies_lab::{Command, Ledger, generated_commands};

#[test]
fn public_ledger_matches_simple_reference_model() {
    let mut system = Ledger::new();
    let mut model = 0_u32;
    for command in generated_commands(0x5eed, 128) {
        let expected = /* checked arithmetic in the model */;
        assert_eq!(system.apply(command), expected.is_some(), "seed=0x5eed");
        // update and compare the externally visible balance
    }
}

The model and generator can grow independently. Architecturally, the important fact here is that the oracle does not reach into Ledger.balance. If a refactor preserves the public behavior, this test should survive even if internal representation changes.

Do not make every case a separate integration-test file. Cargo compiles each file as a distinct test crate, which can add linking and startup cost. Group tests by public contract or expensive fixture boundary, then use modules within a test crate where that improves compile and execution time. Conversely, do not collapse unrelated platform or process tests into one enormous binary whose shared global state makes isolation impossible.

Shared fixture code under tests/common/mod.rs is ordinary test code, not a magical scope. Keep it deterministic and narrow. If it grows network clients, migrations, clocks, builders, and cleanup policy, give it an owner and explicit API or extract a support crate. Fixtures are dependencies: a defect in a builder can make hundreds of tests agree on the same false world.

Let documentation prove the first successful path

A doctest is compiled from a Rust code block in documentation and can run assertions. It answers a public question that ordinary integration tests do not: does the example a reader copies still work through the documented interface?

/// Creates an empty ledger.
///
/// ```
/// use testing_strategies_lab::{Command, Ledger};
/// let mut ledger = Ledger::new();
/// assert!(ledger.apply(Command::Credit(12)));
/// assert_eq!(ledger.balance(), 12);
/// ```
pub fn new() -> Self { /* ... */ }

Use doctests for concise, deterministic public usage. Hidden setup lines can reduce noise, but excessive hidden machinery creates a demonstration the reader cannot reconstruct. no_run still checks compilation and is appropriate when executing would require a network or destructive action; it does not prove runtime behavior. ignore removes even routine compilation and should have a specific, reviewed reason. compile_fail is useful for compact negative API examples, but a dedicated UI suite is stronger when the diagnostic, compiler version, features, or multiple files matter.

Doctests link against the crate’s public interface. They are not a path to private unit testing. They can also behave differently from ordinary source because rustdoc extracts and wraps snippets. Run cargo test --doc explicitly in important library gates rather than assuming an all-target command communicated the intended coverage.

Treat examples as maintained programs

Files under examples/ are product-adjacent artifacts: tutorials, smoke clients, interoperability probes, and operational utilities. Cargo normally compiles examples during cargo test, protecting type correctness, but it does not necessarily execute each example’s main. If runtime output matters, run it explicitly or configure an appropriate harness.

The lab includes examples/replay_seed.rs. Its job is not to duplicate a unit assertion; it gives a maintainer a stable command for replaying a generated sequence:

cargo run --example replay_seed -- 5eed

A useful example has the same dependency and feature discipline as a consumer, avoids workstation-specific files, reports actionable errors, and is small enough to read end to end. If an example becomes the only place a safety or compatibility rule is checked, promote that rule into an automated test while retaining the example’s teaching purpose.

Test binaries from outside the process

Calling a binary’s internal function can test parsing or domain logic, but it cannot prove process-level contracts: argument decoding, environment handling, signal behavior, current-directory assumptions, stdout versus stderr, exit status, or partial output before failure. A black-box test launches the built executable and observes it as an operator or parent process would.

Cargo exposes CARGO_BIN_EXE_<name> to integration tests for a binary target. The lab uses it without guessing a target/ path:

let output = Command::new(env!("CARGO_BIN_EXE_ledgerctl"))
    .args(["credit:20", "debit:3"])
    .output()?;
assert!(output.status.success());
assert_eq!(output.stdout, b"accepted=2\nbalance=17\n");

Assert the channel as well as the text. Diagnostics on stdout can corrupt pipelines; ordinary results on stderr can confuse supervisors. For failures, check a stable domain phrase and exit-code class rather than freezing every punctuation mark unless exact output is a compatibility promise. Bound process time, isolate environment variables, provide a temporary working directory, and kill descendants on timeout. The standard library alone does not provide a universal ergonomic timeout wrapper, so production suites often need a harness with explicit lifecycle ownership.

Make rejected programs first-class evidence

Rust APIs frequently promise that invalid states, illegal trait implementations, non-Send captures, or private operations cannot compile. Runtime tests cannot prove compiler rejection. A compile-fail or UI test presents source to rustc and asserts failure.

The lab’s deliberate probe calls a private function. Its script requires nonzero compilation and matches the core diagnostic phrase. That is a lightweight fixture, not a complete UI framework. Larger libraries commonly use specialized harnesses to compile cases, normalize platform/version noise, and compare stderr snapshots.

Choose assertion strength deliberately:

• Exit failure alone is robust but may pass for the wrong syntax error.
• A diagnostic code can be precise when rustc assigns a stable one, but not every error has one.
• A domain phrase or relevant span protects user experience with moderate churn.
• A full stderr golden catches exact regressions but couples the suite to compiler wording, paths, and formatting.

Pin the toolchain when exact diagnostics matter, normalize only known nondeterminism, and review changes rather than regenerating expected stderr blindly. Test one reason for rejection per small fixture. A file with five independent errors can keep failing after the contract under test disappears.

UI tests also need positive neighbors. If every generic example fails, the suite may protect an accidentally overconstrained API. Pair a rejected program with the nearest accepted program and run both on the declared stable and MSRV channels where compatibility is promised.

Put platform claims on real platforms

Conditional compilation can remove the code that a host cannot exercise. Mocking Windows path syntax on Linux does not test Windows filesystem calls; emulating a target can verify some instruction behavior without proving kernel integration. Classify each promise:

• pure target-independent logic can run everywhere;
• conditional compilation should at least compile for each supported target;
• filesystem, process, socket, dynamic-library, and ABI behavior needs the relevant OS or a justified environment;
• privileged, hardware, or timing behavior may require a dedicated scheduled lane.

Keep platform assertions close to the contract and make unsupported versus temporarily unavailable distinct outcomes. An ignored test without an owner and reason is not evidence. Record the platform, architecture, features, toolchain, and external service versions for failures; only then can a cost-aware CI matrix allocate them honestly.

Design failures for the person on call

Test names and assertions are diagnostic interfaces. test_transfer says less than duplicate_request_id_does_not_apply_second_debit. Prefer messages containing the violated invariant, minimal relevant inputs, deterministic seed, operation index, and observable mismatch. Avoid dumping megabytes of structures or secrets.

Arrange assertions around one causal claim. A test that performs forty operations and ends with twelve unrelated assertions makes triage guesswork. This does not mean “one assertion per test”; several observations may jointly define one outcome. It means one failure should point to one contract and its owner.

Parallel test execution exposes global-state mistakes. Environment variables, fixed ports, process current directory, global tracing subscribers, and shared fixture names can make tests order-dependent. Serialize only the narrow resource that requires it. Better, allocate unique resources and pass dependencies explicitly. A suite that passes only with --test-threads=1 has documented contention, not eliminated it.

Classify ledger-core before adding another test

For each requirement below, write the observer, layer, fixture boundary, failure message, platform set, and owning team:

1. A rejected debit does not change balance or accepted-operation count.
2. Downstream crates can create an account without naming an internal storage type.
3. The first documentation example compiles on the MSRV.
4. Importing the same request twice yields one durable posting after process restart.
5. ledgerctl writes machine-readable results to stdout, diagnostics to stderr, and returns documented status codes.
6. Callers cannot construct a posting with an unchecked negative amount.
7. The on-disk rename is atomic on every supported filesystem/OS combination—or the product explicitly weakens that promise.

Then challenge each placement. A private unit test cannot prove item 2. A doctest cannot prove crash recovery in item 4. A compile-fail test may protect item 6 if the type design makes construction impossible; otherwise a runtime boundary test must prove validation. Item 7 cannot be inherited from a Linux laptop.

Review the evidence architecture

Before approving a repository test plan, ask:

• Is every critical contract mapped to an observer and an owner?
• Does any test bypass the boundary whose promise it claims to verify?
• Are private helpers private for design reasons, or widened for tests?
• Do doctests exercise the documented feature set and MSRV?
• Are examples compiled, and are behaviorally important examples executed?
• Are binaries checked as processes for status and output channels?
• Does each rejection fixture fail for one intended reason with a positive neighbor?
• Are fixture builders smaller and more trustworthy than the systems they create?
• Do platform claims run on their actual platforms?
• Will a failure report enough deterministic context to reproduce without leaking secrets?

The durable model is not a pyramid of test sizes. It is a graph from contract to observer to evidence. Once those edges exist, generators and models can enlarge the behavior space each observer explores without confusing random volume for better architecture.

Sources and version notes

The fixture targets Rust 2024 Edition, stable Rust 1.97.0, and MSRV 1.85.0. Cargo target defaults and harness behavior can evolve; verify them against the toolchains in the project’s support policy.

• The Cargo Book: Cargo targets
• The Cargo Book: cargo test
• The rustdoc book: documentation tests
• The Rust Reference: testing attributes