Timeouts, Retries, Racing, and Idempotent I/O

At 10:14:03.000, relay-service accepted event evt-4821 with a one-second client budget. It waited 310 ms for a commit permit, sent a storage request at 10:14:03.360, and returned a timeout at 10:14:04.001. The caller retried immediately. The first storage operation committed at 10:14:04.090; the second committed at 10:14:04.220. One logical event became two durable records.

Nothing in that sequence requires a broken clock or a data race. The service confused four different facts:

• its local waiter stopped waiting;
• the local future representing the attempt was dropped;
• the transport may or may not have stopped sending;
• the storage system may already have accepted the effect.

A timeout proves only that a result was not observed within a chosen interval. It does not, by itself, prove that underlying work stopped or that no externally visible effect occurred. Once a request can cross a process boundary, temporal failure and effect ambiguity become part of the correctness model.

The central rule is therefore stricter than “add a timeout”: carry one absolute deadline, spend it deliberately, classify failures by what may have happened, and retry an ambiguous effect only behind a durable idempotency protocol. Racing and hedging must use the same rule. The first answer wins the latency race; it does not magically erase losing operations.

Reconstruct the operation in two dimensions

An incident timeline needs both waiting state and effect state. A single span marked timeout cannot answer whether the remote side parsed, accepted, committed, or replied.

For each attempt, record at least these transitions:

local:  admitted → queued → sending → awaiting → result | timed_out | cancelled
remote: unknown  → received → validated → committing → committed → replied

The states are not synchronized. A local timeout may occur while the remote operation is still unknown, while it is committing, after it committed but before the reply arrived, or after the reply was lost. Cancellation is a request to stop work according to some local or remote protocol; it is not retroactive erasure. Some operations are cancel-safe before a commit boundary and necessarily non-cancel-safe after it.

That distinction changes the response to failure:

“Probably absent” still needs a protocol-specific justification. Buffered transports, proxies, multiplexing, and client libraries can make application-level statements such as “nothing was sent” harder to establish than a local byte counter suggests.

One deadline crosses every layer

A relative timeout created independently at each layer expands the caller’s budget. If an HTTP handler allows one second, a storage adapter allows one second per attempt, and a retry wrapper permits three attempts plus backoff, the actual operation can live far longer than the caller’s one-second intent. Queueing from the bounded pipeline consumes time too.

Represent the request’s temporal contract as an absolute deadline in a monotonic clock domain. At every boundary, compute remaining time:

remaining = deadline - monotonic_now

Then reserve time for work that must happen after an attempt: cancelling children, reconciling an ambiguous result, releasing capacity, recording the outcome, and serializing a response. The lab makes that policy executable:

let deadline = Deadline::from_now(1_000, Duration::from_millis(900));
let timeout = deadline.attempt_timeout(
    1_250,
    Duration::from_millis(500),
    Duration::from_millis(100),
);

assert_eq!(timeout, Some(Duration::from_millis(500)));

At 1,250 ms, 650 ms remain. Reserving 100 ms for cleanup leaves 550 ms, so the 500 ms per-attempt cap applies. At 1,850 ms, only 50 ms remain and the function refuses to start an attempt that would consume the cleanup reserve.

An absolute deadline is a value, not a universally portable timestamp. Instant-like monotonic values are normally meaningful only inside their originating process or clock domain. Across a network, protocols commonly transmit a duration, a wall-clock expiry with bounded skew assumptions, or both. A service receiving a deadline should cap it by local policy rather than trusting an arbitrary client to reserve resources indefinitely.

Wall clocks can jump because of synchronization or operator action. They are appropriate for audit timestamps and protocol expirations whose semantics require civil time, but elapsed-time measurement should use a monotonic source. Even monotonic clocks do not guarantee useful progress while a process is suspended or a machine is overloaded. The deadline bounds willingness to wait, not scheduler latency.

Timeout cancellation ends a local future, not an effect history

Tokio’s time::timeout wraps a future and returns an elapsed error if the duration expires. Dropping the wrapper cancels the wrapped future by dropping it. That statement is local and precise. Whether drop closes a socket, releases a permit, unregisters an I/O interest, sends a protocol cancellation, or leaves a remote operation running depends on the future and the systems beneath it.

The runtime also cannot preempt code that never yields. Tokio checks the timeout before polling the inner future; a poll that performs long synchronous work can exceed the duration and still complete. Timeouts therefore require cooperative async boundaries and the blocking-work discipline established at the runtime boundary.

The fixture proves one deliberately narrow property:

#[tokio::test(start_paused = true)]
async fn timeout_drops_the_local_loser_and_releases_its_guard() {
    let active = Arc::new(AtomicUsize::new(0));
    let operation = async {
        let _guard = InFlightGuard::new(Arc::clone(&active));
        tokio::time::sleep(Duration::from_secs(60)).await;
    };

    assert!(tokio::time::timeout(Duration::from_secs(2), operation)
        .await
        .is_err());
    assert_eq!(active.load(Ordering::SeqCst), 0);
}

Paused time keeps the test deterministic, and the drop guard proves local resource release. It does not prove that a hypothetical remote write was undone. A production test needs a fake or integration peer that records request receipt, cancellation frames, commit transitions, and late replies.

Design every timed operation around a cleanup matrix:

The cleanup owner must survive long enough to perform cleanup. Spawning an attempt and timing out only its join handle can recreate detached work if the task itself continues.

Retry the condition, not the error string

Retries are justified by a hypothesis: another attempt has a meaningful chance of succeeding without violating the effect contract and before the deadline expires. Classify failures using at least four axes:

1. Permanence: transient overload differs from invalid input.
2. Effect ambiguity: did the prior attempt possibly change state?
3. Scope: is the failure local to a connection, endpoint, shard, or whole dependency?
4. Budget: can another attempt, its backoff, and cleanup fit?

The lab’s small classifier makes the ambiguity visible:

match failure {
    Overloaded | ConnectionResetBeforeWrite => Retry,
    AmbiguousAfterWrite => RetryOnlyWithIdempotency,
    Rejected | InvalidRequest => DoNotRetry,
}

Real taxonomies include protocol status, request method, transaction state, authentication, quota, circuit state, and dependency-specific guidance. Keep the classification typed and observable. A blanket “retry I/O errors” policy combines failures with different effect histories.

Limit amplification along three dimensions:

• attempt cap: a hard maximum for one logical operation;
• time budget: the absolute deadline and cleanup reserve;
• population budget: a retry rate or token budget shared by callers so an unhealthy dependency is not flooded by synchronized retries.

Exponential backoff reduces repeated pressure, but deterministic backoff can synchronize a fleet. Add jitter from a well-defined distribution and cap it. The fixture uses deterministic symmetric jitter for test evidence; production should inject a random source so policy tests can fix a seed without global nondeterminism.

delay = min(cap, base × 2^attempt) × jitter_factor

Backoff consumes the same deadline as attempts. Do not sleep first and discover afterward that no useful attempt can fit. Expose attempt_number, failure_class, backoff, remaining_budget, and retry_suppressed_reason. A retry counter without its logical-operation denominator hides amplification.

Coordinate retry ownership across layers

Independent retry loops multiply. Three client attempts through a proxy that makes two attempts to a storage SDK configured for four attempts can create as many as twenty-four physical calls for one logical operation. The layers may also use different error taxonomies and deadlines, so an outer layer interprets an inner layer’s exhausted budget as a fresh transient failure.

Assign retry ownership at the layer with the best combination of effect knowledge, endpoint choice, and remaining-budget visibility. A storage driver may safely replay a connection setup that cannot produce an application effect. The application knows whether an append is idempotent and whether another replica is equivalent. A service mesh may know endpoint health but not whether a request body can be replayed. No layer has enough information merely because it can intercept an error.

Publish attempt metadata across the boundary where practical:

logical_operation_id
idempotency_key
absolute_deadline or remaining budget
attempt ordinal and maximum
previous failure class
hedge versus sequential retry

Do not trust caller-supplied ordinals or deadlines without local caps. Their value is coordination, not authority. If an SDK has mandatory internal retries, include them in the application budget and metrics. Prefer disabling overlapping automatic policies when the application owns the semantic decision.

Circuit breakers and concurrency limits solve adjacent problems. A breaker stops sending ordinary calls when recent evidence suggests a dependency cannot serve them, then probes recovery. A concurrency limit bounds active pressure. Neither makes mutation replay safe. A breaker that opens after a request was committed still leaves an ambiguous result, and a limit that admits a retry does not establish idempotency.

Recovery traffic needs its own restraint. When a dependency returns, thousands of callers can wake from similar backoffs. Jitter, a shared retry token budget, gradual admission, and server-advertised delay can spread the load. Honor server retry hints only within the caller’s deadline and local policy; a hint longer than the remaining budget is a reason to stop, not to extend the operation silently.

Idempotency is durable result identity

An idempotency key is not merely a duplicate-detection header. It binds a logical operation identity to a request fingerprint, execution state, and stable result. A robust record commonly includes:

(tenant, operation, idempotency_key)
request_fingerprint
state: in_progress | committed | permanently_failed
canonical result or result reference
retention/expiry policy

The key’s namespace prevents cross-tenant collisions. The fingerprint prevents accidental reuse of one key for different input. The stored result lets a duplicate receive the original semantic outcome instead of executing again or returning a vague “already processed.” Record creation and the protected effect must share a transaction or an equivalent atomic protocol; otherwise a crash between them reopens the ambiguity.

The in-memory fixture illustrates the interface:

assert_eq!(
    ledger.apply_once("req-7", "tenant=a;amount=5", || "offset-91".into()),
    ApplyResult::Applied("offset-91".into())
);
assert_eq!(
    ledger.apply_once("req-7", "tenant=a;amount=5", || panic!("duplicate")),
    ApplyResult::Replayed("offset-91".into())
);

It also rejects the same key with a different fingerprint. This proves local decision logic, not durability. In relay-service, the idempotency record must be colocated with the append transaction or mediated by a storage primitive that provides the needed atomicity.

Retention is part of the contract. Expire records too early and a delayed retry may duplicate an effect; retain them forever and the index becomes unbounded. The lifetime should cover the maximum replay and redelivery window plus clock and operational margins. Document what happens after expiry.

Idempotency is not always available. Streaming partial writes, non-transactional side effects, and calls to systems without operation identity may require reconciliation, compensating action, or an explicit at-most-once choice that sacrifices automatic retry.

There are three common delivery/effect choices, each with a cost:

• At-most-once attempt: do not retry an ambiguous operation. Duplication risk falls, but a possibly absent effect may be lost.
• At-least-once delivery: retry until acknowledged or budget exhaustion. Availability improves, but consumers must tolerate duplicates.
• Effectively-once effect: allow repeated delivery under one durable operation identity and return the canonical result. This requires storage, retention, atomicity, and conflict policy; it is not a transport flag.

Exactly-once language should name its boundary. A broker can deliver one record once within one log transaction while an external email is still sent twice. An idempotent database append can coexist with duplicate metrics or webhook emission. Inventory every side effect under the logical operation and either place it in the atomic boundary, derive it from committed state, or give it its own idempotency identity.

Hedging trades spare capacity for tail latency

A hedge starts a second equivalent attempt before the first has failed, usually after a percentile-based delay. A race may start alternatives together. Both can reduce tail latency when slow attempts are weakly correlated. Both also multiply load and create intentional overlap.

Hedging is defensible only when:

• the operation is read-only or protected by the same idempotency identity;
• the dependency has capacity reserved for hedges;
• endpoints or replicas are independent enough for a second attempt to help;
• losing attempts have an explicit cancellation and result-reconciliation owner;
• the hedge fits the absolute deadline;
• metrics separate original and hedge load.

select!-style racing returns one completed branch and drops non-selected branch futures when the selection expression ends. As with timeout, drop is local cancellation. If branches spawned tasks, issued non-cancel-safe remote writes, or transferred ownership elsewhere, additional cleanup is required. Prefer racing owned futures directly when their drop behavior is understood. If tasks are necessary, retain their handles and finish the lifecycle.

Fairness matters in a loop. Selection order, branch readiness, and runtime-specific behavior can bias who wins. A branch that performs substantial work in one poll can also defeat the intended deadline. Treat selection behavior as third-party runtime behavior, not a Rust language guarantee, and test the exact pattern under the pinned dependency version.

Partial writes make protocol framing part of correctness

Async write methods may write fewer bytes than requested. A timeout or error after a prefix has been written leaves the peer’s state dependent on protocol framing and connection policy. Reusing a stream after abandoning a partially written frame can corrupt message boundaries unless the protocol has a way to resynchronize.

Track write progress as part of the attempt state:

not_started → prefix_written(n) → complete_frame_written → reply_observed

For a length-prefixed protocol, a partial header or body may require closing the connection. For a multiplexed protocol, cancellation may be request-scoped. For an append log, the server may assign an operation identity before payload transfer. The Rust future alone cannot supply these semantics.

Reads can be partial too. A client that times out after receiving part of a response must decide whether to drain, cancel, or close. Connection pools need to know whether a connection remains protocol-clean. Instrument discarded bytes, poisoned connections, late responses, and reconciliation queries; otherwise timeout tuning hides correctness work as transport churn.

Operate the temporal contract

Latency telemetry should decompose, not merely total:

• admission and queue time;
• attempt service time by attempt number;
• backoff time;
• time remaining at attempt start and finish;
• cancellation request-to-cleanup latency;
• ambiguous outcomes and reconciliation latency;
• duplicate keys, replays, conflicts, and record expiry;
• hedge issue rate, hedge win rate, and losing-attempt lifetime;
• bytes written before failure and late-result count.

Alert on retry amplification and cleanup debt. A dependency may appear to meet throughput while total attempts per logical operation climb and hedges consume the headroom needed for recovery. A flat timeout rate can still be dangerous if more timeouts occur after remote commit.

Configuration needs linked bounds. Increasing the attempt timeout without changing the outer deadline may remove all room for retry or cleanup. Increasing retries without a population budget may turn a minor slowdown into overload. Shortening idempotency retention without reducing redelivery windows silently weakens exactly-once-effect behavior.

Timeout values should come from latency distributions and system objectives, but percentiles are not a policy by themselves. A timeout near a dependency’s observed p99 will deliberately abandon roughly the slowest percentile under that workload; if those operations continue remotely, the hidden concurrency can exceed the visible concurrency. Measure the latency distribution of cleanup and late completion as well as replies.

Use separate budgets for different operation classes. A health probe, interactive read, durable append, and shutdown flush have different value and cleanup obligations. One global timeout either wastes latency on cheap failures or cancels work whose reconciliation cannot fit. Keep the number of classes small enough to operate, and attach each to an SLO and owner.

Test the policy under overload and clock discontinuities. Pause task progress while monotonic time advances, exhaust the retry token pool, delay replies beyond the local deadline, and deliver a late successful result after the caller has retried. A system that passes only the prompt-response path has not exercised its temporal contract.

Configuration rollout needs the same care as code rollout. Change one budget dimension at a time, canary it by operation class, and compare logical success, physical attempts, ambiguous outcomes, late commits, and dependency saturation. A lower visible p99 is not an improvement if it comes from returning errors sooner while abandoned remote work grows. Preserve the previous policy for rapid rollback, and include the active timeout/retry policy version in traces so an incident timeline can explain which callers made which decisions.

Incident exercise: the timeout that kept working

Reconstruct the evt-4821 incident using evidence rather than assuming the first attempt stopped.

1. Build a two-lane timeline from admission through both durable commits. Mark queue age, send progress, local timeout, remote receipt, commit, reply, and retry.
2. Name the commit boundary and the owner of each local future, task, permit, transport request, and remote effect after 10:14:04.001.
3. Produce a retry decision table for overload, pre-write reset, partial write, ambiguous reply loss, validation rejection, and confirmed commit.
4. Design the idempotency record, including namespace, fingerprint, atomicity mechanism, replayed result, and retention interval.
5. Allocate the one-second deadline among admission, attempts, backoff, and a cleanup/reconciliation reserve. Reject any attempt that cannot fit.
6. Add one hedge policy and then calculate its worst-case request amplification. State the capacity and correlation evidence required to enable it.
7. Write a deterministic test that proves local losing-future cleanup and an integration test that observes what the server does after client cancellation.

A satisfactory reconstruction does not claim that a shorter timeout fixes duplication. It changes the effect protocol, proves lifecycle cleanup, and makes ambiguous outcomes measurable.

Temporal review

• One absolute deadline reaches every queue and dependency boundary.
• Attempt timeouts reserve budget for cancellation, reconciliation, and response work.
• Elapsed time uses a monotonic source; cross-process expiry states its clock assumptions.
• Failure classification includes effect ambiguity and permanent failures.
• Attempt, time, and population retry budgets are all finite.
• Backoff is capped, jittered, observable, and charged to the deadline.
• Ambiguous mutations require durable idempotency or explicit reconciliation.
• Idempotency keys are namespaced, fingerprinted, atomically tied to effects, and retained for a stated window.
• Races and hedges own losing futures, spawned tasks, remote effects, and late results.
• Partial writes define whether the connection can be reused.
• Timeout metrics distinguish local observation from remote completion.

Temporal correctness now has a place in the interface: callers need to pass deadline and operation identity; implementations need to state cancellation, readiness, Send, and dispatch behavior; tests need controlled time and inspectable I/O. Those are API design questions, not wrapper details.

Sources and version note

Rust’s Future contract defines polling and wake behavior; it does not define a remote cancellation protocol. Tokio 1.52.3’s time::timeout documents cancellation by dropping the wrapped future and warns that non-yielding work can exceed the timeout. Tokio’s select! documentation describes branch cancellation and runtime-specific fairness behavior. These Tokio statements are pinned third-party behavior, not language guarantees.

The async-resilience-lab lockfile records Tokio 1.52.3, declares Rust 1.85 as its MSRV, and tests local cleanup with Tokio’s paused clock. Its deadline uses integer milliseconds only to make the policy reproducible; production code should carry an appropriate monotonic deadline type and define serialization at process boundaries. Revalidate the remote-effect, transport, atomicity, and clock assumptions whenever the protocol, storage boundary, runtime, or deployment clock model changes.