Channels and Ownership Transfer

A worker pool accepts 40,000 records per second and completes 32,000. Its channel is described as “nonblocking,” so producers never wait. Ten minutes later, 4.8 million records are pending—before counting message envelopes, allocator overhead, retained buffers, and retry metadata. Nothing has raced. Nothing has deadlocked. The ownership transfers were valid, and the process is still on a path to memory exhaustion.

Message passing simplifies concurrency when it gives mutable state one owner and turns cross-thread interaction into owned commands. It hides danger when the queue is treated as plumbing rather than a resource with capacity, admission, lifecycle, and observability contracts.

The controlling decision is:

A channel is a queue plus an ownership handoff. Its design is incomplete until capacity, behavior at capacity, receiver multiplicity, disconnect, reply cancellation, fairness, and shutdown are explicit.

Keep three planes separate while reviewing that design. The ownership plane says who owns each message and reply after success or failure. The capacity plane says which budget admits work and what saturation returns. The lifecycle plane says which endpoints may still exist, how intake closes, what accepted work does, and what proves shutdown complete. Most channel incidents come from proving one plane and assuming the other two.

The worker lanes from Chapter 53 now persist across jobs. Channels can keep those lanes free of shared mutable domain state, but the queue becomes shared operational state. The important failures move from aliasing to overload and liveness.

Capacity is part of correctness

For a sustained arrival rate λ greater than service rate μ, backlog grows at approximately λ - μ items per second. Bursts can fill a queue even when the long-run average is safe. A queue bound converts unbounded memory growth into a decision point: wait, reject, shed, degrade, redirect, or replace older work.

Rust’s standard std::sync::mpsc module makes the distinction visible:

• channel() creates a conceptually unbounded multi-producer, single-consumer queue. send does not wait for queue capacity, although allocation and scheduling still have runtime costs.
• sync_channel(n) creates a bounded multi-producer, single-consumer queue. send waits when the buffer is full; try_send returns Full(message) immediately.
• sync_channel(0) is a rendezvous channel: a send and receive must meet. It stores no pending message, although the participating threads and message still consume resources.

“Asynchronous channel” in this API means sending can enqueue without rendezvousing with a receiver; it does not mean the channel integrates with Rust futures. Both recv and a blocked SyncSender::send block the calling OS thread. An async runtime needs runtime-aware channel operations or a deliberate blocking boundary.

Bound the queue from a budget, not a round number. If a message retains up to 256 KiB and the component has a 128 MiB pending-work budget, 4,096 entries are impossible before overhead; 512 entries already consume the full payload budget in the worst case. If messages vary sharply, an item count alone may be inadequate. Enforce byte, work, tenant, or time budgets at admission.

Blocking a producer is not automatically safe backpressure. If the producer holds a lock needed by the consumer, a full queue can deadlock. If it is a request-handling thread, waiting may spend the caller’s entire deadline. try_send makes overload a result the caller must classify, but repeated immediate retries can become a busy loop. The saturation action belongs in the operation contract.

Admission policy determines where overload appears

A bounded queue exposes capacity, but its send method selects who experiences the limit. There are four common policies:

The standard SyncSender offers blocking send and nonblocking try_send; a deadline-bounded admission loop needs careful coordination or a channel implementation with an explicit timed-send contract. Sleeping and retrying is not backpressure design. It adds timing races, wastes capacity, and makes tests flaky.

Choose policy at the layer that knows the loss and retry semantics. A telemetry sampler may drop and count low-priority events. A payment command may persist before acknowledgement. A cache refresh can coalesce duplicate keys. A request handler may return a retryable overload response only if clients have bounded, jittered retry budgets and idempotency. The channel primitive cannot infer these choices.

Capacity reservation can also become unfairness. A single large producer can fill every slot, forcing small critical requests to wait behind bulk work. Separate queues, admission classes, reserved slots, or weighted scheduling may be needed. Those mechanisms should still share one total memory budget; splitting one unbounded queue into several unbounded queues multiplies the hidden risk.

Overload must be observable at the rejection boundary. Count attempts and reasons, but also record which policy followed: dropped, returned upstream, spilled, retried, or coalesced. An accepted counter without a rejected counter can make a saturated service look quiet. A rejected counter without payload size can hide the memory pressure avoided—or the business value lost.

Ownership makes the protocol inspectable

Sending a non-Copy message moves it. After a successful send, the producer cannot mutate or drop that value; the receiver becomes its owner. This is the strongest reason to use a channel for a state-owning worker: commands arrive serially, and only that worker touches its private state.

On failure, ownership can return. TrySendError::Full(T) and TrySendError::Disconnected(T) contain the rejected message. That enables a caller to retry elsewhere, persist the batch, return it to an upstream owner, or account for its drop. A wrapper that maps both cases to a string and discards T destroys an important recovery contract.

Messages should name domain operations and carry the minimum ownership needed:

enum Command {
    Process {
        batch: Vec<Record>,
        reply: SyncSender<Result<Receipt, ProcessError>>,
    },
    Reload(ValidatedConfig),
}

Avoid sending Arc<Mutex<State>> through a channel when the purpose was to establish one owner. That message transfers another shared handle, not ownership of the protected state, and callers can still mutate outside the worker’s protocol. References can cross scoped channels in carefully bounded designs, but long-lived worker queues usually need owned messages because their receive time is not tied to the producer’s stack.

A message type is a public or internal API. Version it with the same discipline as any other boundary: define validation, large-payload limits, sensitive fields, retry safety, and what dropping an unprocessed message means. An enum can make the protocol closed and reviewable. A boxed callback can be flexible, but it obscures operation identity, resource estimates, observability fields, and compatibility.

Multi-producer does not imply multi-consumer

The standard module is named MPSC: multiple producers may clone a Sender or SyncSender, while one Receiver owns consumption. The receiver is not cloneable. That shape naturally implements fan-in: many sources transfer events to one state owner.

Multi-consumer or fan-out designs need another topology:

• A dispatcher owns the single receiver and sends work to per-worker queues.
• Producers choose among per-worker senders directly, as the fixture does.
• Workers steal from a specialized shared queue whose library contract defines multiplicity and fairness.
• Each subscriber gets its own channel when every subscriber must receive every event; this is broadcast, not work distribution.

These are not interchangeable. In a work queue, one consumer handles each message. In broadcast, every active consumer may need a copy or shared handle. In request routing, a key may require affinity to one owner. “Fan-out” must state whether it means load distribution, duplication, sharding, or speculative racing.

Fan-in also needs attribution. Once many producers clone the same sender, the receiver may observe messages in an order influenced by scheduling. FIFO reasoning is safest within the ordering guarantees explicitly provided by the chosen channel and sender usage; do not infer a global fairness contract among producers. If per-key order matters, route a key consistently, attach sequence numbers, or centralize ordering. If priority matters, a single FIFO queue may encode the wrong policy.

A bounded sequence has three outcomes

The fixture exposes nonwaiting admission with try_submit. Each successful request owns a one-shot reply receiver:

pub fn try_submit(
    &self,
    values: Vec<u64>,
) -> Result<Receiver<Result<u64, JobError>>, SubmitError> {
    let (reply_tx, reply_rx) = mpsc::sync_channel(1);
    let index = self.next_worker.fetch_add(1, Ordering::Relaxed)
        % self.workers.len();
    let job = Job { values, reply: reply_tx };

    self.metrics.accepted.fetch_add(1, Ordering::Relaxed);
    match self.workers[index].try_send(job) {
        Ok(()) => Ok(reply_rx),
        Err(TrySendError::Full(job)) => {
            self.metrics.accepted.fetch_sub(1, Ordering::Relaxed);
            Err(SubmitError::Saturated(job.values))
        }
        Err(TrySendError::Disconnected(job)) => {
            self.metrics.accepted.fetch_sub(1, Ordering::Relaxed);
            Err(SubmitError::Closed(job.values))
        }
    }
}

There are three protocol outcomes:

1. Accepted: ownership moves to exactly one worker queue, and the caller owns a reply receiver.
2. Saturated: no transfer occurs; the caller regains the batch and applies overload policy.
3. Closed: no transfer occurs; the caller regains the batch and treats the component as unavailable or shutting down.

The capacity slots in the figure are a hard state boundary, not decorative boxes. A successful send occupies or rendezvouses with capacity. A full try_send returns the same owned batch to the producer. When all senders disappear, the receiver eventually observes disconnection after buffered messages are drained. When the receiver disappears, future sends fail and return their message.

The fixture uses one bounded queue per worker. Round-robin selection is inexpensive and preserves single-consumer ownership, but it is not load-aware: one slow job can fill its selected lane while another lane is idle. Trying every queue may reduce false rejection but changes fairness and admission cost. A central dispatcher can inspect lanes but adds a scheduling hop and its own capacity. Work stealing improves utilization for suitable tasks but weakens affinity and requires a more complex queue contract.

The right behavior depends on job variance and ordering constraints. The fixture is evidence for ownership, bounded admission, replies, and shutdown—not a claim that round robin is universally efficient.

The fixture deliberately returns saturation from the selected worker rather than searching every lane. That makes admission cost constant and affinity predictable, but it can reject while another lane has space. If the contract instead promises “accept when any worker has capacity,” the implementation must coordinate that fact without a check-then-send race. A dispatcher, shared queue, or select-capable channel may express it; reading approximate queue counters and then choosing a sender does not reserve a slot.

Changing topology changes ordering. Direct producer-to-worker routing may preserve order only for messages sent through the same sender to the same lane. A dispatcher can impose one receive order before fan-out, but completion still varies. A shared work queue provides a common acceptance order while worker completion remains concurrent. If callers observe completion order, the reply protocol must explicitly reorder or expose out-of-order results.

Request/reply creates two lifecycles

A reply channel turns a command into a request without sharing a result slot. The producer creates the reply endpoint, sends the response sender inside the owned job, then waits or polls according to its execution model. The worker computes and sends exactly one result.

This pattern separates several failures:

• Request admission can fail before the worker owns the job.
• The worker can return a domain error.
• The worker can panic or exit, disconnecting the reply without a value.
• The requester can lose interest and drop its receiver before the worker replies.

The last case is cancellation of observation, not necessarily cancellation of work. In the fixture, the worker ignores a failed reply send because the batch has already been processed. If processing has external effects, dropping the reply receiver must not imply rollback. If work should stop when the requester leaves, add an explicit cooperative cancellation signal and define its check and commit points.

A capacity-one reply channel is enough for one response and prevents accidental multiple replies from accumulating. A zero-capacity reply would make the worker wait until the requester is receiving; that couples worker availability to caller behavior and can deadlock shutdown if callers disappear without dropping correctly. An unbounded reply channel is unnecessary for a one-shot protocol.

Timeouts require the same care. recv_timeout can stop the caller waiting, but it does not remove a job already queued or interrupt a worker. The system needs a deadline inside the message if stale work should be skipped, an idempotency key if retries can duplicate effects, and accounting for late completion.

Closure and shutdown messages encode different policies

A receiver returns disconnection after every sender has been dropped and its buffered messages have been received. A sender observes disconnection after the receiver is dropped. These rules make endpoint ownership a lifecycle protocol.

The fixture uses closure-driven graceful shutdown:

pub fn shutdown(self) -> thread::Result<()> {
    drop(self.submitter);
    for join in self.joins {
        join.join()?;
    }
    Ok(())
}

fn worker_loop(receiver: Receiver<Job>, metrics: Arc<PoolMetrics>) {
    while let Ok(job) = receiver.recv() {
        // process and reply
    }
}

When the owner drops its last submitter—and all external clones are already gone—each worker drains accepted jobs, observes recv disconnection, and exits. Joining proves shutdown completion. The crucial phrase is “all external clones.” A forgotten sender clone keeps the receiver open forever. Clone ownership must therefore follow component ownership, and shutdown needs a way to stop or revoke producers before waiting for workers.

An explicit Shutdown message expresses a different protocol. It can carry a drain mode, deadline, generation, or acknowledgement. In a FIFO queue it normally sits behind earlier messages; it is not an out-of-band interrupt. With multiple producers, another producer may enqueue work after a shutdown command unless intake is separately closed. With multiple workers, sending one message may stop one worker and leave others waiting.

Use closure when “no producers remain; drain then exit” exactly matches the lifecycle. Use an explicit control message when shutdown is one state transition within a longer-lived connection or when workers need parameters and acknowledgement. Many robust systems use both: revoke admission, send or signal the shutdown mode, drop senders, drain according to policy, and join.

Never rely on process exit as the normal worker cleanup path. It can truncate replies, buffered writes, metrics, and foreign cleanup. Conversely, do not promise indefinite draining: shutdown needs a deadline and a policy for accepted work that cannot finish.

Fairness is a property to verify, not infer

A safe channel can starve a producer, tenant, priority, or worker while maintaining memory safety. FIFO queue order does not prove scheduler fairness. Round robin does not prove equal work when job costs differ. A bounded shared queue can let a hot producer occupy every slot before a quiet producer arrives.

Name the required fairness unit:

• per producer: each source eventually gains admission;
• per tenant: one customer cannot consume the whole budget;
• per key: operations retain order or affinity;
• per priority: urgent work progresses without starving normal work;
• per worker: load is sufficiently balanced for the latency objective.

Then select a mechanism: reserved capacity, per-tenant queues, weighted scheduling, key sharding, admission tokens, aging, or separate pools. Each adds state and failure modes. If strict fairness is not required, say what weaker behavior is acceptable and measure tail latency by class.

Fan-out amplifies this problem. Broadcasting one large message to ten subscribers may retain ten owned copies or one shared allocation until the slowest subscriber releases it. A slow mandatory subscriber applies backpressure to the publisher; a lossy subscriber needs a gap or lag protocol. Work distribution avoids duplication but provides no copy to observers. The topology must follow delivery semantics, not the convenience of a channel constructor.

Queue observability needs events at both ends

“Channel send succeeded” is not enough telemetry. At minimum, observe:

• attempts, accepted, saturated, and disconnected counts;
• queue or outstanding depth, with a precise definition;
• enqueue-to-start wait and end-to-end completion latency;
• active and idle workers;
• processing outcome, panic, and reply-disconnect counts;
• shutdown intake-close, drain, forced-stop, and join durations;
• payload bytes or cost units when item count hides resource use.

The fixture records accepted and completed totals. Their difference is named outstanding, meaning accepted work that is either queued or executing. It is not queue depth. That distinction prevents an operator from interpreting busy workers as buffered backlog. Precise queue depth requires enqueue and dequeue instrumentation, and its snapshot is immediately stale under concurrency. Metrics are evidence for trends and alerts, not synchronization primitives.

Update counters around the actual ownership transition. The fixture increments accepted before the send attempt so a fast worker cannot complete before acceptance is visible, then rolls the count back if try_send returns the job. The atomics use relaxed ordering because they report counts and do not publish job memory; the channel provides the message synchronization. A monitoring counter must never be used to decide whether a receive is safe.

Queue latency is often the earliest overload signal. A queue can remain below capacity while old work violates deadlines. Attach enqueue time or a deadline to the job, measure age at dequeue, and reject stale work when the operation contract allows. Avoid high-cardinality labels such as request IDs in metrics; put identities in sampled structured events or traces.

Design alternatives expose where pressure moves

Message passing is strongest when operations naturally serialize around one owner and when queue policy is easier to defend than lock composition. Shared state is stronger when callers need small atomic operations on one invariant and synchronous access is natural. Hybrids are common: a channel transfers commands, workers share immutable configuration, and a small atomic exposes health. The architecture should identify each mechanism’s invariant rather than declaring one concurrency style superior.

Exercise: make overload and shutdown explicit

Level: Review board. You inherit a pool with eight workers and one unbounded channel<Vec<Event>>. Four ingestion threads clone the sender. Workers write results through a shared callback. Shutdown sends eight Stop values, but producers may still send, reply ownership is implicit, and the only metric is total jobs submitted.

Produce an architecture decision record and a verified revision with these constraints:

1. bound pending memory for worst-case 128 KiB batches under a 64 MiB queue budget;
2. choose one admission result for saturation—bounded wait with deadline, rejection, or spill—and defend its effect on upstream retries;
3. define whether distribution is central, per-worker, keyed, or work-stealing, including the fairness consequence;
4. replace the callback with an owned request/reply or result-stream contract;
5. distinguish domain failure, receiver loss, caller cancellation, worker panic, and shutdown rejection;
6. close intake before drain, account for every sender clone, and join all workers;
7. specify what happens to accepted work at the shutdown deadline;
8. instrument acceptance, saturation, disconnect, queue age, active work, completion, late reply, and drain time;
9. test saturation and disconnect with deterministic coordination, never sleeps.

The arithmetic, endpoint ownership map, and state machine are required artifacts. A design that replaces the unbounded queue with sync_channel(10_000) without a payload budget or caller policy does not pass. Neither does a shutdown plan whose correctness depends on producers “probably having stopped.”

Channel protocol review

• Is this fan-in, work distribution, broadcast, sharding, or request/reply?
• Which endpoint is cloneable, and who proves the last clone is dropped?
• What exact resource does capacity bound: items, bytes, cost, tenant share, or time?
• What happens on full capacity without holding a dependency the receiver needs?
• Does a failed send return ownership for retry or accounting?
• What ordering and fairness are guaranteed, measured, or deliberately absent?
• Does dropping a reply receiver cancel observation, work, or external effects?
• Is shutdown encoded by endpoint closure, control messages, or both?
• Can every accepted job be classified after panic, disconnect, and deadline expiry?
• Do metrics distinguish queued, executing, completed, rejected, and stale work?

Channels do not eliminate shared-state reasoning. They relocate it into ownership, capacity, and lifecycle planes, with ordering and fairness spanning them. Make those contracts visible and a worker can remain a simple single owner. Hide them and a race-free program can still exhaust memory, starve tenants, lose replies, or wait forever during shutdown. Chapter 55 turns to the complementary design: multiple threads accessing one protected invariant directly, where lock scope becomes the central boundary.

Sources and version note

The standard-library documentation for std::sync::mpsc, channel, sync_channel, Sender, SyncSender, Receiver, and TrySendError defines the behavior used by the fixture. The example is Rust 2024 Edition, verified with Rust 1.97.0, and declares Rust 1.85.0 as its MSRV. Scheduling fairness, allocation cost, wakeup behavior, and throughput are not language guarantees; production claims require a named channel implementation, version, target, workload, and measurement.