Cancellation, Idempotency, and Cleanup Contracts

The timeout is not the outcome

A client sends ingestion request event-42, waits until its deadline, and closes the connection. The service reports a timeout. On disk, event-42.record already exists.

Did ingestion fail? The transport did. The operation may have succeeded. Retrying without a stable operation identity might append the event twice; refusing all retries might lose the only usable response. Cancellation exposes a fact that ordinary return values hide: the caller can stop observing before the system stops producing effects.

The useful contract divides execution at one commit point:

• before commit, cancellation may discard tentative work and report that no durable effect was selected;
• after commit, cancellation cannot honestly report rollback, even if the response is lost;
• a retry identifies the same logical operation and retrieves its recorded result instead of performing the effect again.

This is not automatic transactionality. Rust ownership can make staging and cleanup precise, but a database, filesystem, remote service, or message broker defines what becomes visible and durable. The design must connect those external semantics to the operation’s Rust state.

Cancellation is cooperation, not preemption

The chapter fixture uses a small token backed by Arc<AtomicBool>:

#[derive(Clone, Debug, Default)]
pub struct CancellationToken(Arc<AtomicBool>);

impl CancellationToken {
    pub fn cancel(&self) {
        self.0.store(true, Ordering::Release);
    }

    pub fn is_cancelled(&self) -> bool {
        self.0.load(Ordering::Acquire)
    }
}

Calling cancel does not interrupt a write, seize another thread, or unwind a stack. It records a request. The worker decides where it is safe and useful to observe that request. A CPU loop may check every bounded chunk. A synchronous worker blocked in an operating-system call may not observe it until the call returns. A separately spawned task may continue after the handle or response future is dropped unless its owner explicitly propagates shutdown.

That latency belongs in the contract. “Cancellable” without an observation bound is weak operationally. Prefer statements such as:

The importer checks cancellation before opening each object and after each validation batch of at most 1,000 records. An in-progress filesystem sync is not interrupted. Once the manifest rename begins, the importer finishes commit and records the result.

The memory ordering in the tiny token only publishes a boolean. It does not make unrelated state transactional. If cancellation must accompany a reason, deadline, or ownership transfer, put those facts behind an appropriate synchronization design instead of assuming the flag orders the whole operation.

Cancellation also differs from panic. A panic generally signals a violated assumption and may unwind or abort. Cancellation is an expected control request. Both can stop the happy path between two instructions, so both require valid intermediate states and explicit side-effect analysis. A cancellation path should normally return a typed outcome, not panic.

Dropping work is a control-flow edge

In async Rust, a future is a value representing a computation. Executors poll it until it returns Ready; a pending future makes progress only when polled again, unless it merely observes work running elsewhere. If its owner drops it, the future’s owned fields are dropped. Code after its current suspension point does not run just because it appeared later in the async block.

That gives every .await a review question: if the future is dropped here, what remains true?

Consider an async method that removes an item from a queue, awaits sending it, then acknowledges it. Dropping the future during the send can leave the item removed but unacknowledged. Memory remains safe; the workflow may not. A lock guard held across .await may be released on drop, yet remote work already submitted can continue. A spawned child task is independently owned and may outlive the parent future. An ordinary destructor cannot perform arbitrary asynchronous cleanup to completion.

An operation is cancellation-safe at a suspension point when dropping it there leaves the surrounding system within the documented contract. That may mean no effect, a valid resumable staging state, a committed effect discoverable by key, or a quarantined job for recovery. It does not always mean “nothing happened.”

Synchronous code has the same problem under different triggers: a shutdown token observed between loop iterations, a caller abandoning a worker, process termination, a lease expiring, or an I/O API returning after partial progress. Async syntax increases the number of visible suspension points; it does not create the underlying contract.

Place checks around the commit point

The fixture stages bytes in a temporary file. Its central sequence is deliberately short:

let mut staging = StagingFile::create(staging_path)?;
staging.file.write_all(payload)?;

if cancellation.is_cancelled() {
    return Ok(IngestOutcome::CancelledBeforeCommit);
}

staging.commit(&destination)?; // sync, then rename
Ok(IngestOutcome::Committed { path: destination })

The cancellation check is useful before rename: dropping staging removes the temporary file, and the destination remains absent. A check immediately after rename cannot produce CancelledBeforeCommit; the visible effect already exists. The implementation may return Committed, record a “response abandoned after commit” observation, or finish a result record for later retrieval. It must not lie.

A commit point has four properties worth writing down:

1. Visibility: which observers can see the new state after it?
2. Durability: which failures can still erase it?
3. Uniqueness: what prevents the same logical operation committing twice?
4. Result recovery: how does a retry learn the first outcome?

One instruction rarely supplies all four. A filesystem rename can change a name atomically under applicable filesystem and platform semantics, while application idempotency still requires an operation record. A database transaction may commit rows atomically, while publishing a message afterward creates a second commit boundary. A remote API may accept a request before the client receives its response. Name each boundary instead of describing the entire workflow as “atomic.”

Too many cancellation checks can be as misleading as too few. A check inside a non-rollbackable commit sequence may abandon necessary finalization. Once commit begins, mask or defer cancellation for the bounded critical tail, then report the committed outcome. The critical tail should be small, observable, and protected by a timeout or recovery process appropriate to the external system.

Idempotency identifies work; it does not make code pure

An idempotent operation produces the same intended effect when the same logical request is repeated. That is different from a naturally idempotent state assignment such as “set desired version to 7,” and different again from a pure function with no effects.

For non-natural operations, use an idempotency key whose scope and ownership are explicit. A useful record contains at least:

• the key and tenant or security principal that owns its namespace;
• a fingerprint of the canonical operation, so the key cannot silently name different work;
• state such as staged, committed, or failed-permanently;
• the committed result or enough information to reconstruct it;
• retention and conflict policy.

The fixture uses the final file as a deliberately small result record. If the destination exists and its bytes equal the retried payload, it returns Replayed. If the same key names different bytes, it returns KeyConflict. Production systems should not compare secrets or enormous payloads naïvely; they can store a versioned canonical digest and approved result fields. The important rule is that equality is defined by the operation contract, not by whatever JSON byte ordering arrived.

Keys need scope. A user-chosen key in a global namespace lets one tenant probe or block another tenant’s work. Guessable keys can disclose whether an operation occurred. Authenticate the lookup, bind the key to the caller and operation class, bound its size, and apply retention. Expiring records too early reopens duplication; retaining them forever creates cost and privacy obligations.

Idempotency also cannot erase non-idempotent sub-effects performed before the record. If ingestion charges an account, publishes an event, and only then stores its key, a crash between those actions permits a duplicate charge. Put the deduplication decision and authoritative mutation in one transactional boundary where possible. Otherwise use a durable state machine, outbox/inbox pattern, or downstream idempotency key and admit that the system spans multiple commits.

Read the operation as a state machine

Figure 37-1 separates two concerns often collapsed in review. The upper path is logical: a same-key retry returns the recorded result. The lower path is physical: a temporary file is written and synchronized, renamed within the destination directory, and followed by any parent-directory synchronization required by the deployment’s durability policy.

Cancellation before the amber gate takes the discard path. Cancellation after it takes the lookup path. There is no arrow from committed state back to “never happened.” That absence is the most important part of the figure.

The diagram is a retrieval aid, not a portable filesystem specification. std::fs::rename exposes cross-platform rename behavior, but replacement, crash durability, open-file interactions, network filesystems, and directory synchronization vary by platform and filesystem. Verify the exact target environment. Keep temporary and destination paths on the same filesystem, and do not infer durable persistence merely because rename returned successfully.

Cleanup guards own residue, not success

The fixture’s staging guard is armed when it creates a unique file:

impl Drop for StagingFile {
    fn drop(&mut self) {
        if self.armed {
            let _ignored = fs::remove_file(&self.path);
        }
    }
}

Normal early return, observed cancellation, and unwinding all drop the guard. Commit synchronizes the file, renames it, then disarms cleanup. This converts a scattered set of remove_file calls into one ownership rule.

The guard is best-effort. Drop cannot report its removal error through the operation’s return value, and it may never run under abort, power loss, process kill, or deliberate leak. Therefore a production design also needs startup reconciliation: enumerate staging names created by this protocol, validate their age and ownership, then resume or delete them. Never run a broad “delete every dotfile” cleanup. Namespace temporary artifacts, prevent path traversal, use restrictive permissions, and avoid following attacker-controlled links.

When cleanup failure matters immediately, provide an explicit method that returns a result, then retain Drop as a fallback. Do not make Drop the commit operation: callers cannot observe a failed final flush, and async network cleanup cannot be awaited from ordinary drop. A guard should release local ownership cheaply; a recovery worker should handle slow or fallible reconciliation.

Temporary naming is also concurrency control. create_new(true) avoids the check-then-create race for a single path, but the fixture intentionally assumes one active writer per idempotency key. Two writers can race between the destination lookup and rename. Production code must serialize by key, use an atomic conditional insert in the authoritative store, or accept one winner and reconcile the loser. A random suffix avoids temp-name collision; it does not enforce one logical commit.

Partial writes require loops and policy

The Write::write contract permits writing fewer bytes than supplied. write_all loops until the buffer is written or an error occurs. Even after it returns, data may reside in memory or an operating-system cache. File::sync_all asks the operating system to synchronize file content and metadata and can surface errors that drop would ignore; it is more expensive than closing. sync_data may be sufficient when metadata durability is not required, subject to platform behavior.

A robust file replacement protocol normally considers:

1. create a unique temporary file in the destination directory with correct permissions;
2. write the complete representation and validate any length or checksum;
3. flush language-level buffering, if present, and apply the selected file synchronization policy;
4. rename to the final name under verified same-filesystem semantics;
5. synchronize the parent directory where the target platform and durability promise require it;
6. record or expose the committed outcome.

Atomic visibility and crash durability are separate. So are record replacement and append. Appending a multi-field record can tear or stop partway; readers need framing, checksums, length prefixes, or a journal recovery rule. For a large object, reconstructing on retry may be cheaper than retaining a full temporary file, but that choice changes cancellation latency and resource cost.

Designs that survive compilation but not interruption

Return cancelled after commit. The caller retries and repeats an irreversible effect. Return or record the committed outcome once the gate has been crossed.

Generate a new key on every retry. The service cannot correlate attempts. The logical caller supplies or durably retains one operation identity across transport retries.

Check cancellation after every write. The method abandons a file halfway through a commit protocol and calls that responsive. Define cancellable staging and a bounded non-cancellable commit tail.

Use Drop as proof of cleanup. A killed or aborting process leaves residue forever. Add namespaced startup reconciliation and age/ownership validation.

Write directly over the destination. Readers observe truncation or partial content. Stage beside the destination and replace under verified filesystem semantics.

Store the key after the side effect. A crash opens a duplication window. Couple deduplication and effect transactionally or propagate the key downstream.

Claim every future is cancelled when dropped. Separately spawned or externally executing work continues. Document ownership of child tasks and propagate cancellation explicitly.

Treat cleanup failure as success. Disk pressure accumulates staging files until later operations fail. Emit bounded cleanup metrics and provide reconciliation.

Shutdown turns cancellation into ownership transfer

Service shutdown is not one token broadcast followed by hope. A process usually has at least three phases: stop admitting new work, drain or cancel owned work, then close shared resources. Reversing them produces noisy failures: the database pool closes while ingestion tasks are still inside their commit tail, or listeners keep accepting requests after the drain deadline began.

Give each spawned task an owner and a join policy. The owner should know whether shutdown asks the task to finish, cancel before commit, finish a bounded commit tail, or detach work to a durable queue. Waiting forever is not graceful; set a drain budget, report remaining operation identities, and define the supervisor action when the budget expires. A forced process exit may leave staging, so startup reconciliation is part of shutdown design even when graceful drain usually succeeds.

Backpressure interacts with cancellation. If a request is cancelled while waiting to enter a bounded queue, it should not consume a slot later. If it has already transferred ownership to a worker, dropping the sender’s response handle does not necessarily revoke the job. The queue protocol needs an acceptance point distinct from the business commit point. Metrics should distinguish rejected-before-acceptance, cancelled-while-staged, committed-response-lost, replayed, and cleanup-failed; collapsing them into cancelled_total hides capacity and correctness problems.

Deadlines are data, not merely timers. Pass the remaining budget only where downstream work can use it honestly. Reusing an expired request context for the bounded commit tail can terminate required finalization; discarding all deadlines can let cleanup consume the outage. Use separate budgets for admission, execution, commit, and reconciliation, and make their ownership visible in traces without placing unbounded operation keys in metric labels.

At a distributed boundary, cancellation is another message that can be delayed, duplicated, or reordered. A remote worker may receive “cancel” after it committed. Its response must describe the actual state rather than echo the requested intention. Protocol states such as cancel_requested, cancelled_before_commit, and committed are more truthful than one ambiguous boolean.

Document the interruption surface

A reviewable API answers these questions:

• Who may request cancellation, and how is authority revoked?
• Where is cancellation observed, and what bounds observation latency?
• Which suspension points or blocking calls are cancellation-safe?
• What externally visible effects may exist on every early-return path?
• Where does commit begin and end? Is cancellation deferred inside that interval?
• What identity makes a retry the same logical operation?
• How are key conflicts, retention, authentication, and result replay handled?
• Which guard owns tentative resources, and what reconciles them if drop never runs?
• Which atomicity and durability claims depend on a database, filesystem, target, or third-party service?
• Do tests inject interruption before, during, and after commit?

For an async method, state whether dropping the returned future cancels local work, requests cancellation from a child task, detaches it, or leaves a remote operation running. For a synchronous method, state which calls may block beyond the deadline and whether a worker continues after its caller leaves. “Supports cancellation” is not enough for either model.

Exercise: make ingestion restart-safe

Level: System design and proof. A service parses an uploaded file, writes normalized records, publishes one notification, and returns a generated batch id. The client retries on timeout. Today it writes directly to the final file, creates a fresh batch id per attempt, and deletes partial files only on handled Err paths.

Deliver:

1. an operation state machine naming every tentative, visible, committed, and reconciled state;
2. a commit-point table for the file, batch record, and notification, including atomicity and durability authority;
3. an idempotency-key schema with tenant scope, canonical request fingerprint, conflict behavior, result retention, and authentication;
4. a Rust staging guard plus explicit cleanup/reconciliation path;
5. a decision on notification delivery: same transaction, transactional outbox, or downstream idempotency, with its duplication window;
6. sync tests that cancel before write, after partial staging, immediately before commit, and after commit;
7. an async audit marking every .await, owned child task, and drop outcome;
8. a crash-recovery test that begins with stale staging and a committed result whose response was never delivered.

Reject a solution that merely wraps the method in a timeout or retries the whole closure. Accept different storage mechanisms when the same-key replay, commit evidence, and residue recovery are executable.

Durable conclusions

• Cancellation is an expected cooperative request; responsiveness depends on explicit observation points and bounded work.
• Dropping a future drops its local state, not necessarily every remote or spawned effect it initiated.
• A cancellation-safe point leaves the system in a documented state, which may be tentative, resumable, or already committed.
• Commit divides “safe to abandon” from “must finish or recover the recorded result.”
• Idempotency requires stable logical identity, operation equality, conflict policy, and retained outcome—not just a retry loop.
• Cleanup guards own tentative local resources; reconciliation covers paths where destructors do not run or fail.
• Atomic visibility, durable persistence, exactly-once business effect, and response delivery are different guarantees.

Part VI began by classifying failure and ends by classifying interruption. The same discipline—typed outcomes, retained evidence, valid intermediate state, and explicit commit—now moves outward. The next part asks where those contracts live as a Rust system grows from a file into packages, crate targets, module trees, and public boundaries.

Sources and verification notes

• Rust standard library, Future, including polling, pending work, and independently executing underlying computations.
• Rust standard library, Write::write_all, File::sync_all, OpenOptions::create_new, and std::fs::rename.
• Executable source: examples/rust-engineering-handbook/part-06/restart-safe-ingestion-lab/, Rust 2024 Edition, no third-party dependencies, rust-version = "1.85".
• The fixture’s rename example is intentionally qualified by target and filesystem semantics and assumes one active writer per idempotency key. It does not claim portable crash durability or distributed exactly-once execution.
• Fixture verification covers formatting, all-target compilation, tests, doctests, Clippy with warnings denied, release execution, and the stated MSRV.