Designing for Testability and Hermeticity

At domain time t=1000, relay message 41 reaches the output adapter. The adapter returns Unavailable. A retry policy consumes the next deterministic jitter draw, 7, and schedules attempt two for t=1107. A call at t=1106 does nothing. At t=1107, the second delivery succeeds. Shutdown then rejects new work, drains what was already accepted, and produces one observable Stopped transition.

That entire trace runs in less than a millisecond. It does not sleep, open a socket, mutate the machine clock, consult the operating system’s random source, or race a runtime timer. More importantly, it tests a production contract: the service reads time, obtains entropy, and performs output through boundaries that production also needs. The test did not gain a privileged method that skips the real state machine.

The architecture behind that trace is the last layer in a quality system. Tests, fuzzers, dynamic tools, and CI gates can only control what the design makes controllable. Keep policy as deterministic transformation wherever practical; concentrate effects in a narrow shell; and place seams at volatile, consequential boundaries rather than around every function. This gives tests authority over inputs while leaving production logic intact.

Make the decision core ordinary data

The most useful meaning of “pure core, effectful shell” is not “functional programming everywhere.” It is a placement rule. Code that decides what should happen should accept explicit facts and return explicit decisions. Code that discovers the current time, reads a packet, chooses random jitter, writes a file, or sends a request belongs at an effect boundary.

For a relay, the decision state can be plain data:

struct Scheduled {
    message: Message,
    attempt: u8,
    due: Millis,
}

enum Observation<E> {
    Delivered { id: u64, attempts: u8 },
    RetryScheduled { id: u64, attempt: u8, due: Millis },
    Abandoned { id: u64, error: E },
    Stopped,
}

This model says more than a method named retry(). It exposes the retry attempt, domain deadline, terminal failure, and lifecycle transition as values a test can inspect. The service shell still has work to do: read the clock, call the output, obtain jitter after a failure, and arrange the next wake. Those effects should not leak back into the policy as hidden globals.

Purity is a spectrum. A reducer that mutates a VecDeque supplied by its owner can still be deterministic. An adapter can cache a connection and remain testable. The decisive question is whether equal state and explicit inputs produce equal decisions, not whether every function is mathematically pure.

There are also costs. Moving every line into command-returning reducers can make straightforward resource code indirect. A local parser over a byte slice does not need a “byte provider” abstraction. Use the split where effects obstruct important evidence or where policy deserves independent reasoning: scheduling, retries, authorization, idempotency, routing, admission, and lifecycle transitions are common examples.

Give time a domain, not a wall-clock shortcut

Production code often reaches directly for Instant::now(), SystemTime::now(), or an async runtime’s sleep function. The first direct call seems harmless. The tenth creates a state machine whose timing cannot be advanced without waiting and whose boundary cases depend on scheduler luck.

First decide which kind of time the contract uses:

• monotonic elapsed time for deadlines, backoff, leases, and duration measurement;
• wall-clock time for human timestamps, calendars, certificates, and cross-system records;
• logical or domain time for event ordering and simulation.

Do not substitute wall time for elapsed time. Wall clocks can jump because of synchronization or administrative changes. A fake must preserve the semantics of the production clock it replaces: a monotonic fake should never move backward unless the test explicitly exercises invalid adapter behavior.

The chapter lab uses the smallest adequate port:

pub trait Clock {
    fn now(&self) -> Millis;
}

pub trait Entropy {
    fn next_u64(&mut self) -> u64;
}

pub trait Output {
    type Error;
    fn deliver(&mut self, message: &Message) -> Result<(), Self::Error>;
}

Relay::step asks the clock once, processes at most one due item, and returns an observation. A test owns a FakeClock backed by Rc<Cell<Millis>>, so advancing from 1,106 to 1,107 is a data operation rather than a sleep. Production can supply a monotonic implementation. The public behavior remains the same on both paths.

The fake clock should not automatically advance whenever it is read. Auto-advance hides accidental extra reads and can create impossible interleavings. Advance it explicitly at a named point. Likewise, avoid using a frozen clock for every test; code that accidentally requires progress may deadlock against a fake that production never sees. Test both a fixed instant and deliberate advancement.

Inject entropy as input to policy

Randomized retry jitter, identifiers, sampling, shuffling, and security tokens do not have identical requirements. A deterministic pseudo-random sequence is appropriate evidence for retry policy. It is not a substitute for a cryptographically suitable operating-system source when unpredictability is the production contract.

Keep the seam at the level the policy consumes. The relay asks for a u64 draw and maps it into a bounded jitter interval. A test supplies [7, 0, 20] and can prove exact due times. An alternative is to pass the already-selected jitter duration into a pure scheduling function. That function boundary is often better when only one decision needs entropy:

fn retry_due(now: Millis, base: Millis, jitter: Millis) -> Millis {
    Millis(now.0.saturating_add(base.0).saturating_add(jitter.0))
}

Use a trait when a long-lived component owns repeated draws or when several production implementations are meaningful. Use a function parameter or explicit value for a localized decision. Use a generic parameter when static dispatch and concrete ownership are useful; use dyn Trait when runtime composition is part of the architecture. Testability does not choose dispatch for you.

Seeded generators deserve care. A seed enables replay only if the algorithm, version, input construction, and draw order are also stable. Record them for discovered failures. Do not make a test assert an incidental sequence from a third-party generator unless that sequence is itself part of the contract. Prefer assertions about bounds, invariants, and reproducible failure cases.

Model adapters by contract, not by private calls

Mocks tend to encode conversations: “expect connect, then write, exactly once, then flush.” Sometimes order is the protocol, but often those expectations merely restate today’s implementation. A refactor from one write to two buffered writes breaks the test while preserving caller behavior.

An in-memory adapter is stronger when it implements the observable contract. For a repository, it might preserve uniqueness, version checks, transactions, and query semantics. For the relay output, it records attempted message IDs and consumes a sequence of outcomes. Tests assert delivery, retry, abandonment, and shutdown observations—not whether a private helper was called.

Adapter fidelity has limits:

Keep multiple layers. A memory repository cannot prove a PostgreSQL transaction isolation assumption. A local TCP server can prove framing and cancellation without depending on a public endpoint, but it may not reproduce a proxy’s half-close behavior. Contract tests should run the same behavioral suite against the in-memory and real adapter where their promises overlap.

Ports should use domain types. A UserRepository with six meaningful operations communicates more than a generic interface that accepts arbitrary SQL strings. Conversely, an interface with fifty methods usually mirrors a vendor client and gives fakes an impossible compatibility burden. Wrap the smallest capability the component genuinely owns.

Make failure a first-class adapter outcome

Happy-path dependency injection is incomplete. Production effects can fail before work begins, after partial progress, during cleanup, or ambiguously after the remote side committed. A useful seam preserves these distinctions.

For retryable operations, define a fault vocabulary from the caller’s decisions:

• unavailable before acceptance;
• timeout with unknown remote outcome;
• explicit rejection;
• malformed or unauthenticated response;
• partial write or truncated read;
• shutdown or cancellation;
• permanent local configuration failure.

The lab uses one Unavailable variant to keep its mechanism small, but production code should not compress every failure into a Boolean. Retry eligibility, idempotency, and observability depend on what is known. Inject failures at meaningful boundaries: after N accepted items, on a named message, during commit, or after response bytes. “Fail every third method call” is reproducible but may couple the test to unrelated internal calls.

Fault injection needs its own assertions. Verify that the fault actually fired; otherwise a green test may have missed the intended branch. Bound every retry test by attempts and domain time. Assert the final state and retained evidence—dead-letter record, error classification, metric, or audit event—not merely that an error was returned.

Panic injection has a narrower role. It can test unwind cleanup where unwinding is supported, but an adapter error is not interchangeable with a panic, process abort, power loss, or kill signal. Crash-consistency evidence generally needs process-level tests and durable storage inspection.

Isolate files without pretending all filesystems agree

Temporary resources should give each test exclusive names, explicit ownership, and cleanup on both success and failure. A robust test creates a fresh directory, places all paths beneath it, and removes it through an RAII guard. The test should never depend on the repository working directory, a developer’s home directory, or a previous run’s leftovers.

Use the platform temporary directory only as a parent; create a unique child and treat collisions as errors. Avoid predictable names alone when tests can run concurrently. Keep a failed directory when forensic value outweighs cleanup, but print its path and bound retention. Tests that intentionally inspect crash leftovers need a separate cleanup policy.

An in-memory filesystem may be useful for path-independent policy, yet real filesystem semantics vary across platforms and filesystems: case sensitivity, rename behavior, permissions, locking, symlinks, timestamps, path encoding, durability, and deletion of open files all differ. Put path construction and content policy under fast tests, then run targeted native tests for promised platforms and durability behavior.

Security tests must keep the hostile boundary real enough. Normalize and validate paths at the owned boundary; exercise traversal, symlink, and race cases in isolated real directories where applicable. A map keyed by strings cannot establish operating-system containment.

Contain processes, networks, and environment

Subprocess tests should build an explicit world. Resolve the executable to an absolute path, set a fresh working directory, close or pipe standard streams deliberately, bound execution time, and kill plus reap children on timeout. Supply an allowlist environment rather than inheriting credentials, proxy settings, locale, and tool configuration from the test runner.

Rust’s std::process::Command::env_clear prevents ordinary environment inheritance, after which the test can add only required variables. The standard-library documentation also warns that relative executable paths combined with current_dir are platform-sensitive; canonicalize the program path before changing the child directory. On Windows, executable resolution has additional PATH behavior, so verify the actual platform contract rather than assuming Unix semantics.

Never mutate process-global environment in parallel tests to configure code under test. In Rust 2024, mutating environment through std::env::set_var and remove_var is unsafe because other threads and foreign libraries may read it concurrently. Parse configuration once from an explicit provider or map, pass the typed result inward, and reserve real-environment tests for a serialized process boundary.

Network tests should bind loopback port 0 and read the assigned address instead of guessing a free port. Own the listener before starting the client, give every operation a deadline, shut the server down explicitly, and join its task or thread. Do not call public services from the deterministic suite. DNS, proxies, certificates, rate limits, and Internet availability are external variables; test them in a separately governed integration lane.

Isolation is not emulation. A loopback test omits packet loss, cross-host clocks, load balancers, NAT, certificate rotation, and real deployment policy. Preserve a small set of environment-realistic tests for those risks, but do not force every state-machine test to pay their latency and flakiness.

Turn environment into validated configuration once

Environment variables, command-line arguments, configuration files, secret stores, and service discovery are input adapters. Business logic should not know which one supplied a value. Read them at startup, preserve source-aware diagnostics, validate relationships, and pass a typed configuration inward.

Suppose retry policy arrives as RELAY_MAX_ATTEMPTS, RELAY_BASE_DELAY_MS, and RELAY_JITTER_MS. A boundary test should cover missing text, nonnumeric text, zero attempts, overflow, and a jitter bound that violates service policy. Once parsed, inner tests construct:

RetryPolicy {
    max_attempts: 3,
    base_delay: Millis(100),
    jitter_bound: Millis(20),
}

They do not need to coordinate a process-global environment. This separation also clarifies reload semantics. An immutable startup configuration is owned by the component for its lifetime. A reloadable policy needs an explicit snapshot or update channel, versioning, and a rule for in-flight work. Re-reading an environment variable on every retry is neither a coherent reload design nor a safe test seam.

Test precedence at the boundary: command line over file, file over environment, or whatever the product promises. Test secret redaction in errors and debug output. Then keep the parsed domain value free of source concerns. A fake “configuration service” that can return a different answer on every getter call models behavior the production system may never support.

Locale and timezone deserve the same treatment. If parsing or formatting is locale-independent, choose and pass that policy rather than inheriting the test runner’s locale. If local calendar behavior is required, make timezone data and daylight-saving edge cases explicit test inputs. Hermeticity begins when ambient context becomes named data.

Treat builds as functions of declared inputs

A hermetic test has the same result from the same declared inputs and toolchain, independent of undeclared machine state. A hermetic build adds a stronger artifact concern: source, dependencies, compiler, target, flags, build scripts, environment, and external tools must be controlled enough to explain the output.

Cargo provides mechanisms, not an automatic hermeticity guarantee. --locked prevents lockfile changes; --offline prevents network access but relies on the local cache; source replacement can constrain registries; pinned toolchains and targets constrain compilation. Build scripts remain arbitrary host programs. They can read files and environment, execute tools, inspect timestamps, and access the network unless the execution environment prevents it.

For build scripts:

• declare input changes with cargo::rerun-if-changed and relevant environment with cargo::rerun-if-env-changed;
• write generated artifacts under OUT_DIR, not into source directories;
• distinguish HOST from TARGET during cross-compilation;
• pin or checksum external generators and make their invocation visible;
• prohibit undeclared network access in the build sandbox;
• test regeneration from a clean checkout and compare expected outputs.

Reproducible artifacts and hermetic execution overlap but are not identical. A hermetic build can embed its current timestamp and produce different bytes. A reproducible build might accidentally succeed on two similarly configured machines while still reading an undeclared input. Record both goals explicitly.

Cache correctness is another contract. A shared compiler cache can improve speed without becoming an input authority. Validate clean builds periodically, key caches by every relevant compiler and configuration dimension, and treat cache deletion as a supported operation. If removing target/ changes test semantics, the build has hidden state.

Reconnect deterministic policy to the concurrent shell

A deterministic reducer proves lifecycle decisions, not that an async task wakes at the right time or that a blocked send observes cancellation. Keep a thin set of runtime tests around the shell. Start the component with a paused or controllable runtime clock when the runtime documents one, drive it to an observable ready state, advance time, and await a bounded completion signal. Never assume that yielding once means every spawned task has reached a particular line.

Synchronization in the test harness should express milestones: listener bound, worker accepting, shutdown requested, input closed, queue drained, task joined. Arbitrary sleeps express guesses. A timeout remains necessary as a harness safety bound, but it should fail the test rather than serve as the mechanism that makes progress.

Test cancellation at each owned await boundary that can retain a resource or half-completed operation. Check that senders are closed, children are joined, reservations are returned, and accepted work follows the declared drain-or-abort policy. Then use controlled schedule exploration or stress evidence for interleavings the deterministic policy test does not cover.

This layered design prevents two opposite mistakes. Pure tests do not claim runtime liveness, and every policy edge does not require a real scheduler. The shell suite proves wiring and lifecycle integration; the data-level suite explores exact domain transitions cheaply; deployment evidence covers the runtime and platform behavior neither can emulate.

Delete seams that exist only to satisfy tests

Testability can become architecture theater. Warning signs include one-method traits for every concrete helper, public constructors used only by tests, production branches guarded by “test mode,” fakes that reimplement the entire dependency, and assertions against private call order. Each abstraction adds names, generic parameters, object-safety questions, documentation, and evolution cost.

Apply a deletion test:

1. Name the production volatility or policy boundary the seam represents.
2. Name the high-consequence behavior it makes controllable.
3. Ask whether an explicit value or function parameter is smaller.
4. Ask whether a state/result assertion can replace interaction expectations.
5. Remove the seam if its only justification is “the test framework wants it.”

There are credible alternatives. A concrete component can be tested through a local server instead of a trait. A deterministic function can accept now and jitter values directly. A database layer may use transaction rollback and isolated schemas rather than an in-memory rewrite. A large legacy binary may first gain process-level characterization tests before internal seams are safe to introduce.

The goal is not maximum unit-test coverage. It is a small set of stable control points that let each evidence layer exercise its contract at the cheapest truthful fidelity.

Refactor the relay without introducing a mock framework

Take a loop that calls Instant::now(), sleeps, reads random jitter, sends through a concrete client, and exits when a global shutdown flag changes. Produce a refactoring dossier with these artifacts:

1. Effect inventory: mark every read of time, entropy, configuration, filesystem, process state, network, and shutdown state.
2. Decision boundary: rewrite retry and lifecycle rules as state plus explicit observations. List the commands or observations visible to callers.
3. Seam choice: choose explicit values, functions, traits, a local server, or process isolation for each effect. Justify why each is the smallest truthful boundary.
4. Deterministic trace: prove a first failure at t=1000, jitter draw 7, no retry at 1106, delivery at 1107, shutdown rejection, drain, and one stopped transition.
5. Fault table: cover pre-accept failure, unknown-outcome timeout, permanent rejection, and cancellation. State which cases are safe to retry.
6. Fidelity plan: identify what the memory adapter cannot prove and assign those claims to protocol, real-dependency, and deployment tests.
7. Hermetic envelope: list child environment, temporary paths, loopback addresses, toolchain, dependencies, and build-script inputs.
8. Deletion pass: remove every abstraction that has no production boundary or consequential test behind it.

The exercise is complete when tests use no wall-clock sleeps, no public network, no shared directory, and no process-global environment mutation; every injected fault is observed; and the production path uses the same decision logic and ports.

Review testability as an operational property

Before accepting a design, ask:

• Are policy inputs explicit, or does the core discover them from global state?
• Are monotonic, wall-clock, and logical time distinguished?
• Can retries, expiry, and shutdown advance without sleeping?
• Does deterministic entropy preserve the production security boundary?
• Do fakes implement caller-visible contracts instead of private conversations?
• Is every fake paired with evidence for the real adapter assumptions it omits?
• Can faults occur before, during, and ambiguously after an effect?
• Does the test prove the intended fault actually fired?
• Are temporary resources unique, owned, and cleaned or deliberately retained?
• Are subprocess executable, directory, streams, environment, deadline, kill, and reap behavior explicit?
• Are network addresses dynamically allocated and operations bounded?
• Can builds run from a clean checkout without undeclared network or host inputs?
• Would deleting a seam simplify production without losing consequential evidence?

Evidence layers can control reality honestly only when the architecture exposes the effects they need to vary. With that boundary established, the governing question changes from “does the system obey its contract?” to “what does the system cost under a specified workload?” The same discipline carries forward: explicit inputs, controlled environments, and evidence whose limits remain visible.

Sources and version notes

The lab targets Rust 2024 with an explicit Rust 1.85 MSRV and uses only the standard library. Environment mutation became unsafe in Edition 2024; verify platform-specific process and filesystem behavior on each promised target. Cargo flags constrain particular inputs but do not by themselves sandbox build scripts or prove reproducible bytes.

• Rust standard library: std::process::Command
• Edition Guide: unsafe functions for process environment mutation
• Cargo Book: build scripts and declared inputs
• Cargo Book: environment variables
• Cargo command reference: cargo test