Executors, Reactors, Runtimes, and Runtime Boundaries

At 02:10, relay-service stops acknowledging events. CPU is nearly idle, every socket is open, and the process still answers its health probe. “The async runtime stalled” names no actionable hypothesis. Four very different failures can produce that dashboard:

• an event was ready, but its future lost the waker registration;
• the reactor observed readiness and woke the task, but the executor’s ready queue was saturated;
• a task reached a worker and then blocked that OS thread in a synchronous client call;
• all tasks are progressing, but application backpressure deliberately prevents new ingestion.

Each failure has a different owner, evidence trail, and repair. A runtime often packages several mechanisms together, but collapsing them into one black box destroys the distinctions needed for design review and incident response. The durable model is a chain of responsibilities:

The language defines futures and polling. Event sources expose readiness. Reactors or drivers translate readiness into wakeups. Executors schedule ready tasks and poll them. Blocking pools isolate work that cannot cooperate. A runtime bundles some or all of those facilities. The application still owns task lifetime, concurrency limits, backpressure, and shutdown.

The exact packaging is runtime-specific. The boundaries are still useful even when one crate implements all of them. The chapter’s controlling thesis is ownership, not taxonomy: a runtime may supply mechanisms, but the application must assemble them into explicit task, capacity, blocking, and shutdown boundaries.

Classify the stall before touching configuration

Take the four symptoms in order.

Lost notification. The future returned Pending, but no valid path will wake its current task. Executor queue depth may be zero because the executor has nothing to schedule. Inspect registration, readiness, and cancellation state as Chapter 58 did.

Ready-queue delay. Wakeups occur and tasks enter a runnable state, but workers do not reach them promptly. Look for queue delay, worker saturation, long polls, self-wake storms, unfair task budgets, or a flood of higher-priority work.

Blocked worker. A polled future calls a blocking filesystem, DNS, database, compression, or foreign-function operation. The future has not returned Pending; the executor cannot reclaim that worker. A current-thread executor stops entirely. A multithreaded executor loses capacity one worker at a time.

Application backpressure. The system intentionally stops intake because a bounded channel, semaphore, memory budget, or downstream admission rule is full. Queue delay may be healthy. The correct signal is capacity utilization plus the reason producers are waiting or rejecting work. Raising executor threads can violate the resource invariant rather than improve throughput.

This classification prevents a common operational error: tuning worker count for a notification bug or removing backpressure to conceal a slow dependency.

The layers and their contracts

Rust’s language and standard library define async, .await, Future, Poll, Context, and Waker. They do not select a network driver, timer wheel, work-stealing policy, blocking pool, or process lifecycle. Calling an async function creates a future. Something else must poll it.

An executor owns runnable-task scheduling. A minimal executor needs task storage, a ready queue, wakers tied to task identity, and a loop that polls a ready task until it returns Ready or Pending. A production executor may add multiple workers, local queues, work stealing, priorities, cooperative budgets, panic isolation, and task metrics. These are implementation and product choices, not properties of Future.

A reactor or I/O driver owns event-source registration and readiness delivery. It associates sockets or other handles with interests, waits efficiently for operating-system events, updates readiness state, and wakes tasks that may now progress. The future is polled later by the executor. The reactor does not call application async functions to completion, and readiness does not contain the future’s output.

Timers follow the same conceptual path. A timer facility records a deadline, an OS or runtime clock source observes expiry, and the timer future’s task is woken. Some runtimes integrate timers with the reactor loop; others package them separately. Do not turn the conceptual label into a claim about an implementation’s thread topology.

A runtime is a bundle and an integration contract. It commonly combines an executor, I/O and timer drivers, task APIs, runtime-bound resource types, entry functions, and a blocking-work facility. It may own worker threads or run on the caller’s thread. “Runtime” does not mean “multithreaded,” and “executor” is not a synonym for the whole bundle.

Read the numbered flows independently. An OS event reaches the reactor, which wakes a task into the ready queue. The executor later polls that task. A blocking call is admitted by application policy into a separate pool; its completion produces another notification path back to a ready task. Neither path guarantees immediate polling or completion on the next poll.

A ready queue is not a thread queue

A task is a schedulable future plus runtime bookkeeping. A worker is an OS thread currently polling tasks. A single worker may drive thousands of mostly waiting tasks. A task may be polled many times, and a multithreaded executor may poll successive states on different workers when the task is Send and its scheduler permits migration.

The lab makes the smallest distinction executable. CurrentThreadExecutor stores tasks and a FIFO ready queue. Each TaskWake contains a task identifier and the shared scheduler state:

struct TaskWake {
    id: usize,
    scheduler: Arc<Mutex<SchedulerState>>,
}

impl Wake for TaskWake {
    fn wake(self: Arc<Self>) {
        self.scheduler.lock().unwrap().ready.push_back(self.id);
    }
}

wake enqueues identity; it does not poll. The executor loop removes an identifier, builds its Waker, and polls the stored future once. A deterministic test proves that yield_once produces the trace [task, task]: first poll returns Pending after self-waking, then the ready-queue entry causes the second poll to return Ready.

This fixture deliberately supports !Send task futures because all polling happens on one thread. The waker itself is thread-safe, as required by the standard Wake interface, but task storage can contain Rc state. A production local executor needs a disciplined ingress mechanism; another thread cannot simply move a !Send future into it.

Cooperative scheduling makes poll duration a system property

Executors cannot preempt ordinary Rust code in the middle of poll. A task that parses a huge payload, loops without an await, blocks on a syscall, or repeatedly finds immediately-ready children can monopolize its worker. The source code may contain many async functions and still behave synchronously between suspension points.

The lab’s chunked_work performs one unit, calls a deterministic yield_once, then resumes later. Two tasks produce A, B, A, B on the FIFO model. This proves only the fixture’s policy. It illustrates the contract the application needs: long work must be divided at meaningful boundaries, or moved to a facility intended for blocking or CPU-bound work.

Do not scatter yields through hot loops without a budget model. Yielding too often adds queue operations, waker traffic, lost cache locality, and latency variance. Yielding too rarely increases tail queue delay. Choose a work unit such as records decoded, bytes scanned, or elapsed CPU budget; measure poll duration and peer latency; and make the budget visible in code review.

Runtime fairness guarantees vary. FIFO, local-first scheduling, work stealing, cooperative budgets, and timer priority are not language contracts. Even a fair ready-queue policy cannot help when a poll never returns.

Blocking work needs isolation and admission control

Blocking is not defined by whether a function is syntactically async. Calling a synchronous method from an async function still blocks the executor worker until the call returns. Destructors can block too.

A runtime’s blocking API typically moves a closure to threads on which blocking is acceptable and returns a future or handle for its result. That boundary has its own capacity and lifecycle:

• the closure and result commonly need Send + 'static because they cross thread and ownership boundaries;
• queued work consumes memory even before it starts;
• running synchronous work may not be abortable by dropping its async waiter;
• runtime shutdown may wait for blocking work or abandon the wait while the work continues;
• a pool sized for occasional filesystem calls can be overwhelmed by unbounded CPU work.

The lab uses std::thread::spawn only to make ownership transfer explicit. It is not a replacement for a managed blocking pool. In relay-service, short bounded legacy calls can use the runtime’s blocking facility behind a semaphore. Persistent consumers may deserve dedicated owned threads. Heavy CPU work may belong in a CPU-oriented pool with an explicit concurrency budget. The decision depends on duration, cancellation, admission, and shutdown—not on an API name.

Never hold an async permit or lock accidentally while waiting for blocking work. Decide whether the permit bounds end-to-end requests, only queued blocking jobs, or running blocking jobs. Those policies produce different memory and latency behavior.

Runtime entry is an ownership boundary

An executable must decide where the runtime begins and ends. A runtime entry macro is convenient, but the architectural questions remain:

• Who constructs configuration and validates it before worker threads start?
• Which thread owns the runtime value?
• Which resources must be created inside an entered runtime context?
• Who initiates shutdown, waits for children, and drops runtime-bound handles?
• Can synchronous callers enter async code, and from which threads?

Nested runtime failures arise when code that already runs on an executor tries to create and synchronously block on another runtime, or blocks the current worker waiting for work scheduled onto that same constrained environment. Some runtimes detect particular forms and panic; other combinations deadlock or starve. There is no language-level “nested async” permission.

Put synchronous-to-async bridging at owned edges: process main, a dedicated adapter thread, a test harness, or a documented foreign-function boundary. Pass a runtime handle when a component must submit work to an existing runtime. Avoid library methods that silently construct a runtime, because they hide threads, I/O-driver lifetime, shutdown behavior, and incompatibility with the caller’s context.

Current-thread runtimes make this especially visible. The thread calling block_on drives the executor and often its drivers. If it returns or blocks elsewhere, tasks do not progress. A handle that can spawn may not by itself drive timers or I/O when no thread is actively running the driver. Treat that behavior as runtime-specific and verify it from primary documentation.

Runtime-specific types spread coupling through signatures

Coupling is not binary. Classify each dependency by what it imports into the library’s public contract:

Runtime-neutral does not mean dependency-free or maximally generic. It means the portability claim matches the abstraction. A network protocol library can keep parsing, state transitions, retries, and error taxonomy neutral while providing a runtime-specific transport adapter. An application binary can intentionally standardize on one runtime and use its types freely inside that integration layer.

Beware “neutral” traits that erase necessary semantics. A timer abstraction still needs a clock, cancellation behavior, deadline rules, and test control. A byte-stream abstraction needs ownership, backpressure, half-close, and error semantics. Portability that discards these contracts merely moves bugs into adapters.

A boundary-shaped relay-service

A maintainable service can divide ownership like this:

relay-domain
  owns: Batch, Commit, validation, retry classification, state transitions
  knows: no runtime types

relay-ports
  owns: async operation traits, clock and storage contracts
  knows: Future-level semantics and domain types

relay-runtime-adapter
  owns: sockets, timers, channels, spawn, blocking bridge, telemetry hooks
  knows: selected runtime and driver types

relay-app
  owns: runtime construction, task tree, concurrency budgets, shutdown
  knows: deployment policy and every adapter

The direction matters. Domain code does not call a global spawn because a storage write happens to be asynchronous. It returns a future to the application-owned task. The adapter may expose a runtime-specific listener because that type is useful at the binary edge, but it converts incoming data into owned domain requests before crossing inward.

This architecture also sharpens tests. Domain and state-machine tests use ordinary values. Port-contract tests use deterministic fake clocks and scripted operations. Adapter tests run under the chosen runtime and exercise real cancellation and I/O readiness. End-to-end tests construct the same runtime topology as production. “Runtime-neutral” claims are tested separately from “this adapter works on runtime version X.”

Observe every handoff, not just task count

Useful runtime telemetry follows the arrows in the architecture:

• event registration count, readiness notifications, and stale-registration cleanup;
• wake count, coalescing indicators, and wake-to-ready-queue delay;
• ready queue depth and age, polls per task, poll duration, and long-poll outliers;
• active and parked worker count, worker utilization, and steal activity when applicable;
• blocking queue depth, wait time, active threads, task duration, and shutdown residue;
• timer count, expiry lag, and clock-adjustment policy;
• application permits, bounded-channel occupancy, rejected work, and downstream latency.

Metrics must preserve boundary meaning. “Task latency” that combines time awaiting I/O, time ready but unscheduled, poll CPU time, blocking-pool wait, and application admission cannot direct an incident. Trace identifiers should survive wake and adapter boundaries without assuming a task remains on one thread.

A thread dump can reveal blocked workers, but an idle thread pool does not prove health. The system may have a stranded future with no wake path. Conversely, high ready-queue depth does not prove executor failure if the application submitted more work than its CPU budget permits.

Exercise: remove ambient runtime ownership

Audit a library with this public method:

pub fn start(&self) {
    runtime::spawn(self.retry_loop());
}

The retry loop creates runtime timers, opens a runtime-specific socket, sends to an unbounded channel, and has no join handle. Produce:

1. A dependency map classifying every imported type or ambient function as language, executor, reactor/driver, blocking, runtime, or application coupling.
2. A revised domain API that returns an owned future or exposes an explicit run loop instead of spawning internally.
3. One runtime adapter that owns timers, I/O, and bounded channels while preserving the retry policy’s deadline and backpressure semantics.
4. An application assembly that constructs the runtime once, attaches the task to an explicit parent, records a handle, and defines shutdown. Chapter 61 will deepen the task-tree protocol; here, make ownership visible.
5. Tests for the neutral core without a runtime, adapter tests under a current-thread scheduler, and a multithreaded integration test that makes no assertion about which worker polls a task.
6. An operations sheet mapping “no wake,” “ready but delayed,” “worker blocked,” “blocking pool saturated,” and “application capacity full” to distinct evidence.

Then evaluate three alternatives: expose selected runtime types publicly, hide them behind a port, or standardize the entire product on one runtime. A strong decision names the product boundary and maintenance cost. Runtime-specific code at an application edge is often simpler and more honest than generic abstractions throughout; runtime-specific ambient behavior inside a reusable domain library is much harder to control.

Runtime review in one pass

• The language supplies future and wake contracts, not a scheduler or I/O implementation.
• The executor polls ready tasks; the reactor or driver turns event readiness into wakeups.
• A runtime packages facilities, but the application owns task lifetime, resource budgets, and shutdown.
• Wake means “schedule reconsideration,” not “operation complete.”
• Cooperative executors require bounded poll work; worker count cannot repair a poll that blocks forever.
• Blocking pools need admission, cancellation expectations, and shutdown policy.
• Current-thread execution is valid and enables !Send tasks, but progress depends on the driving thread.
• Runtime entry and bridging belong at explicit owned edges; hidden and nested runtimes are hazards.
• Public runtime types create intentional coupling; ambient runtime calls create hidden coupling.
• Tests and telemetry should follow readiness, ready queue, poll, blocking, and application-admission boundaries separately.

With these roles separated, “the runtime stalled” becomes a finite investigation. The next question is no longer where a task happened to run. It is who owns that task, how cancellation reaches it, what happens to its children, and how the whole task tree joins system shutdown.

Sources and version note

The executor/future boundary follows the official Future, Waker, and Wake documentation. Tokio is used only as a current primary example of runtime packaging: its official Runtime documentation describes a scheduler, I/O driver, timer, and blocking pool; Handle, spawn_blocking, and LocalSet document runtime entry, blocking limitations, and local !Send task execution. Those APIs and policies are runtime- and version-specific, not Rust language guarantees. The dependency-free lab is a teaching model verified on Rust 1.97.0 with Rust 1.85.0 as MSRV; its FIFO trace is not a claim about production scheduler fairness.