Error, Cancellation, and Async Contracts in Public APIs

The upload future has returned Pending. The caller’s deadline expires, so the task drops it. What is true now?

let receipt = timeout(deadline, client.upload(request, progress)).await??;

The type signature does not answer whether bytes reached the network, a temporary object remains, the object became durable, the progress callback can still run, a retry will duplicate data, or cleanup blocks an executor thread. async says that the operation is represented by a future. Result says that completion can report failure. Neither construct supplies the operational contract.

A public operation is complete only when callers know what may happen before completion, which effects survive interruption, and which evidence distinguishes retry from reconciliation. That statement governs errors, cancellation, timeouts, retries, callbacks, destructors, and execution context as one design problem. Treating them as separate documentation chores produces the most dangerous state: every individual sentence sounds plausible, but no caller can make a safe recovery decision.

This chapter specifies an upload API around a visible durable-commit boundary. Its fixture uses only std so cancellation behavior can be polled and observed without importing a runtime’s policy. The model transfers to database writes, message publication, device commands, and remote control-plane changes.

Audit the operation in a fixed order: trace effects, name success evidence, interrupt at every suspension point, classify recovery, account for resource ownership, and finally state execution context. Errors, timeouts, retries, callbacks, and destructors then attach to one lifecycle instead of becoming five independent policy lists.

Trace effects, not merely future states

An async operation usually crosses several effect phases:

1. Validate owned inputs without external effects.
2. Prepare local staging, credentials, or request metadata.
3. Transfer data to a peer or temporary resource.
4. Commit the externally visible durable result.
5. Acknowledge the result to the caller.

The important line is not “before or after .await.” It is the commit boundary. A suspension point may occur during any phase, and one phase may contain many suspension points. Cancellation before commit can still require cleanup. Cancellation after commit but before acknowledgement can leave the caller with an unknown outcome. The server succeeded; the client did not receive proof.

The diagram deliberately labels ownership. The caller transfers the request body into the future. While transfer is in progress, the operation owns staging. After commit, the durable service owns the object and the caller should receive a receipt that identifies it. A design that cannot say who owns each resource at interruption cannot claim cancellation safety.

Rust’s Future trait defines polling and completion, not transaction rollback. Dropping a future runs ordinary destruction for the future value and its fields. That can release memory, file descriptors, guards, or runtime-specific handles, but it does not recall packets, undo a database commit, or prove that a remote worker stopped. Cancellation behavior belongs to the particular operation and the systems it drives.

Publish an operation contract card

Before choosing an error enum or executor integration, write a contract card for each effectful public operation. Here is the card for UploadClient::upload:

This card is more useful than “cancellation safe: yes.” Cancellation safety is conditional on where interruption occurs and what the caller does next. The table also exposes incompatible promises early. If destruction must wait for a remote abort acknowledgement, then ordinary Drop cannot be the complete cleanup mechanism; the API needs an explicit async shutdown or abort handle.

Stabilize error decisions, not every diagnostic detail

Callers branch on some failure distinctions and merely report others. A public error type should preserve the former while leaving room to improve the latter. The fixture exposes a non-exhaustive category:

#[non_exhaustive]
pub enum UploadErrorKind {
    InvalidRequest,
    Unavailable,
    OutcomeUnknown,
}

pub struct UploadError {
    kind: UploadErrorKind,
    detail: &'static str,
}

UploadError::kind is a compatibility promise. Display is concise operator-facing context, not a parsing protocol. Private fields allow the implementation to add a source chain, request identifier, or redacted diagnostic without forcing callers to construct or exhaustively destructure the error. The type implements std::error::Error; a production implementation should expose meaningful sources where underlying causes help diagnosis without leaking secrets or transport internals.

“Transient” must be a caller-relevant policy, not a guess derived from string matching. Even a transient category needs conditions. Unavailable might permit retry only before the caller’s overall deadline, with backoff, a retry budget, and the same idempotency key. InvalidRequest needs correction, not delay. OutcomeUnknown needs a query by operation or idempotency key because repeating a non-idempotent operation may amplify the incident.

Avoid exporting every HTTP code, database code, or dependency error variant as the top-level API. That freezes implementation choices and makes callers depend on distinctions the library cannot preserve. Conversely, one opaque “request failed” category prevents automated recovery. Stable domain decisions plus inspectable sources usually form the better boundary.

There are three credible public shapes. A concrete non-exhaustive enum makes categories easy to match but makes every public variant and field a compatibility decision. A private error struct with a stable kind() method, as in the fixture, permits richer internal evolution at the cost of one more call. A boxed dyn Error + Send + Sync preserves maximum implementation freedom but supplies no domain branching unless paired with another classification channel. Choose from caller needs, not library convenience. Application binaries can often use broad contextual errors because they own both ends; reusable libraries usually owe downstream code a more deliberate decision surface.

Panic behavior belongs beside ordinary failure

Result does not describe panic behavior. Public docs should say which caller-controlled conditions can panic, which callback panics propagate, and whether the library catches unwinding at a containment boundary.

The fixture promises no intentional internal panic during upload, but its progress callback is caller code. If that callback panics while the future is polled, the panic propagates through the polling task. The caller then needs the same phase analysis as cancellation: had commit occurred, which resources drop during unwinding, and can another task observe partial state?

Do not casually catch every panic and translate it into an error. A panic may signal a violated invariant, and continuing with corrupt logical state can be worse than terminating a worker. Catching unwinding is appropriate at explicit isolation seams such as a plugin invocation, job boundary, or FFI adapter, followed by a defined quarantine or shutdown policy. It is not a substitute for documenting ordinary recoverable errors.

Destructors must remain safe during unwinding. They should not panic, because a second panic during unwinding can abort the process. They should also avoid taking contended locks or performing blocking network work. Release local ownership synchronously; put fallible, asynchronous finalization in an explicit method whose result can be observed.

Say whether polling can block

An async signature is not a non-blocking guarantee. Code before the first .await runs when the future is first polled. Code between suspension points runs on the polling thread. DNS resolution, filesystem calls, compression, cryptography, mutex acquisition, and destructor cleanup can all monopolize an executor worker if implemented synchronously.

Document relevant behavior in operational terms:

• whether a poll can execute CPU work proportional to input size;
• whether it acquires a contended synchronous mutex;
• whether any platform path invokes blocking I/O;
• whether progress callbacks run on the polling thread;
• which work is delegated to a blocking pool or dedicated thread;
• whether cancellation waits for that delegated work to stop.

“Non-blocking” should mean polling does not wait synchronously for external progress. It does not mean free, constant-time, lock-free, or immune to scheduler starvation. If a method legitimately blocks, make that explicit in its name or documentation and keep it out of an async facade whose callers reasonably expect cooperative scheduling.

Runtime assumptions and Send are separate choices

Futures are inert until polled. An async API may be runtime-neutral, require a named reactor, require a local task set, or return a handle owned by an internal runtime. State the choice. Hiding a runtime dependency until the first panic or stalled socket is a contract failure.

Send answers a narrower question: can ownership of the future move across thread boundaries? A multithreaded executor often requires spawned futures to be Send + 'static, but the Future trait itself does not. A library may deliberately support thread-affine state and return a non-Send future for local execution. Neither is universally superior.

The fixture requires P: FnMut(usize) + Send, and a test proves that the returned upload future is Send. Its compile_fail doctest passes an Rc-capturing callback and verifies rejection. This is more durable than assuming an async fn remains Send after internal refactoring. An apparently harmless non-Send value held across an .await can change the generated future’s auto traits and break downstream spawning.

If future mobility is public, keep a compile-time assertion in the test suite. If it is intentionally local, document the executor context and avoid accidentally implying spawn compatibility in examples.

A timeout cancels waiting, not reality

Timeout combinators commonly stop polling and drop an inner future. That is a local control-flow event. The remote service may still be processing a request, a blocking worker may still run, or the commit may already have happened.

Separate three deadlines:

• a caller wait deadline, after which the caller no longer waits;
• an operation deadline, carried to cooperating components and the remote service;
• a resource lease, after which staging or ownership expires independently.

Passing a deadline through the protocol can bound useful work. It still cannot erase a commit that raced with expiry. For mutating operations, return or preallocate an operation identifier so the caller can query outcome. Use clocks consistently: durations from a monotonic clock for local expiry; explicit timestamps only where a cross-system protocol needs them and clock skew is addressed.

Timeout errors also need classification. “Timed out before dispatch,” “remote deadline rejected,” and “acknowledgement absent after possible commit” lead to different actions. Collapsing them into one retryable timeout invites duplication.

Retry policy starts with idempotency

A retry repeats an attempt, not necessarily an intention. Safe retry requires a stable identity and server-side semantics that recognize equivalent attempts. The fixture’s IdempotencyKey is caller-chosen, validated, and retained with the request. The store maps the key to the first durable receipt, so completing the same logical upload twice returns the same evidence rather than creating a second object.

That toy mechanism omits production questions that the public contract must answer:

• How long is the key retained?
• Is identity scoped to tenant, account, or endpoint?
• What happens when the same key arrives with different content?
• Does a failed validation consume the key?
• Can two concurrent attempts race?
• Is the stored response replayed exactly or reconstructed?
• Which authentication principal may query an outcome?

Automatic retry also needs a budget, backoff with jitter, and observability. A library that retries internally can obscure latency, consume a caller’s deadline, and multiply load during an outage. Expose attempt counts or tracing, accept a bounded policy where appropriate, and let callers disable retry when they coordinate it at a higher layer.

Never retry a callback merely because the surrounding transport retried unless the callback contract permits repetition. A progress callback, credential provider, signing hook, or “before send” observer may have effects of its own.

Cancellation safety is a state invariant

The fixture’s first poll invokes progress with zero, reaches YieldOnce, and returns Pending. Dropping the future there destroys the owned UploadRequest; the store remains unchanged. A unit test observes exactly that. On the second poll, the future resumes and commits under the store lock. There is no suspension point while the lock is held and no cancellation gap between mutation and receipt construction.

This narrow design supports a precise claim: cancellation at its only pending point is safe and leaves no durable object. It does not justify a universal statement about real uploads. A production transfer may need a staged object with a lease, an explicit abort request, or a background reaper. The invariant could instead be:

At every suspension point, durable state is either absent, recorded as incomplete under the operation key, or committed with a queryable receipt.

That invariant survives dropped futures and lost acknowledgements. It also dictates storage schema and protocol design, which is why cancellation cannot be repaired solely inside an async wrapper.

Audit every .await and every call that can yield indirectly. Record the owned resources and externally visible effects immediately before it. Then test cancellation at those points with deterministic fakes, fault injection, or a model appropriate to the system. A single happy-path timeout test proves very little.

Resource ownership must survive every exit path

Ownership in a Rust signature covers values in one process. An operation also acquires logical and remote resources that the borrow checker cannot track: multipart-upload identifiers, temporary files, database reservations, permits, leases, authentication sessions, and metrics cardinality. List them explicitly against every exit.

Suppose upload staging allocates a remote multipart object. Four designs are defensible:

• The future owns an abort handle whose destructor submits a best-effort local cancellation signal; a background worker performs remote cleanup.
• The API returns an UploadHandle, and callers must invoke an async abort or finish; leaked staging expires under a server lease.
• The server owns incomplete staging from creation and guarantees time-bounded garbage collection, so client drop needs no remote action.
• The operation is detached after submission, and dropping only releases the local observer; callers query the handle later.

These designs expose different semantics. The first cannot promise that Drop observed remote deletion. The second makes cleanup visible and fallible but relies on a server backstop for crashed clients. The third trades storage and reaper work for simple client cancellation. The fourth is not cancellation at all; it is detachment. Name it honestly.

On reported failure, say whether owned input can be recovered. Moving a gigabyte buffer into a future and returning only UploadError destroys or retains the buffer according to the implementation. If retry should avoid rereading or reallocating it, return an error wrapper containing the request, accept a replayable source, or persist staging behind a handle. Each alternative changes memory pressure, lifetime, security, and API complexity. “The request failed” is insufficient when the request is itself an expensive resource.

Security belongs in this ledger. Credentials and plaintext buffers should have defined retention and redaction behavior; cancellation must not extend a secret’s lifetime indefinitely in an abandoned queue. Do not promise zeroization merely because a buffer drops—ordinary Vec<u8> destruction releases allocation ownership but does not establish a guaranteed data-erasure mechanism.

Finally, make cleanup observable. Metrics for incomplete operations, staging age, abort attempts, reaper backlog, unknown outcomes, and idempotent replays let operators verify that the documented lifecycle holds under outage conditions. Cancellation without observability becomes accumulated state, not a solved problem.

Callbacks create a reentrancy boundary

A callback is foreign control flow entering library state. Specify when it runs, how often, on which thread or task, whether invocations can overlap, which locks are held, whether it may retain arguments, what a panic does, and whether it may call the originating object again.

The fixture runs progress synchronously on the polling thread, never while holding its store mutex. It says the callback may repeat and prohibits recursive client calls. A richer client might allow reentrancy, but then its invariants must tolerate nested operations and callbacks arriving in surprising order. “We release the lock first” prevents one deadlock pattern; it does not establish a complete reentrancy policy.

Prefer data streams or channels when notifications have independent lifetime, buffering, backpressure, or concurrency. Prefer a callback for a narrow synchronous capability. Do not retain a borrowed callback beyond the documented call, and do not invoke user code from Drop.

Exercise: specify and break an async upload

Write a complete public contract for an upload that supports a 1 GiB body, chunk progress, a 30-second caller deadline, and an idempotency key. Deliver all of the following:

1. A phase diagram that marks validation, staging, each suspension point, durable commit, and acknowledgement.
2. An ownership ledger for input bytes, buffers, temporary objects, sockets, credentials, and the final receipt at success, reported failure, timeout, callback panic, and dropped future.
3. A stable error classification with the exact caller action for every category: correct, retry, reconcile, abort, or escalate.
4. Runtime and mobility statements: blocking paths, executor dependency, Send/'static expectations, and callback thread/reentrancy rules.
5. An idempotency protocol covering key scope, retention, content mismatch, concurrent attempts, and outcome query authorization.
6. Destructor and explicit-shutdown behavior, including what cleanup is bounded and what a background reaper owns.
7. Tests that cancel before transfer, during staging, immediately before commit, after commit before acknowledgement, during callback execution, and during shutdown.

Adversarially inject a lost response after server commit, a poisoned local worker, a callback panic, duplicate concurrent attempts, a full staging volume, and a caller that drops the future without ever polling it. The contract is complete only when each witness has a defined durable state and recovery action.

Review questions for public operations

• Can callers enumerate stable failure decisions without parsing text or dependency errors?
• Does “transient” include retry conditions, key reuse, budget, backoff, and deadline interaction?
• Which code can block the polling thread, including callbacks and destruction?
• Is runtime dependence explicit? Is future Send behavior intentional and tested?
• Does every suspension point preserve a named state invariant?
• Does dropping stop only polling, or also delegated and remote work?
• What evidence distinguishes no commit, committed, and unknown outcome?
• Who owns input, staging, handles, and durable resources after every exit?
• Can callback invocation overlap or reenter, and are internal locks released first?
• Can a destructor panic, block, or attempt fallible remote cleanup?

Make interruption a first-class API result

Errors describe reported completion. Cancellation and timeout describe an observer ceasing to wait. Retry begins another attempt. None alone reveals durable reality. A senior API makes them coherent around a commit model, operation identity, ownership ledger, and queryable evidence.

The Rust type system can enforce owned input, meaningful receipts, callback bounds, and future mobility. Tests can witness cancellation points and idempotent replay. Protocol and storage design must supply the remaining guarantees. Once those guarantees exist, callers still need to find them without reconstructing the system from source. The next chapter turns this operation contract into rustdoc, examples, feature documentation, search paths, and release notes.

Sources and verification notes

• Standard library documentation: Future, Send, and Drop.
• The Asynchronous Programming in Rust book provides the official executor, future, pinning, and cancellation background; runtime-specific semantics still belong to the selected runtime.
• Rust API Guidelines: meaningful error types and Send/Sync.
• Executable source: examples/rust-engineering-handbook/part-08/public-operation-contracts-lab/. Its deterministic future, cancellation-before-commit test, idempotent replay test, Send assertion, runnable doctest, and rejected non-Send callback are evidence for this fixture only. Network, runtime, and distributed-commit claims require system-specific tests.