Production I/O, Configuration, Secrets, Shutdown, and Deployment

A release review for relay-service reaches a deceptively simple question: what, exactly, is being deployed?

The executable is only one answer. Operators receive a process that reads bytes from streams and files, resolves values from several authorities, handles credentials, reacts to platform events, advertises lifecycle state, consumes finite resources, and eventually reports an exit status. A binary can be memory-safe and functionally correct while losing a configuration file during a power failure, printing a token in an error, accepting traffic before initialization, or being killed halfway through its drain budget.

The process contract is therefore: resolve and validate all operating inputs before readiness; make every I/O completion and durability boundary explicit; keep secrets narrow and non-renderable; translate platform termination into one bounded drain; and make the built artifact, supervisor, probes, limits, and exit semantics agree.

Begin at the operating boundary

Write the process contract before the deployment manifest. It should answer six questions.

1. Which configuration source wins, and can the effective non-secret configuration be inspected?
2. Which bytes may be partially consumed or produced, and what counts as durable completion?
3. Where do secrets enter, how long are they retained, and which renderers are forbidden?
4. Which event stops admission, how is existing work drained, and what deadline wins?
5. Which lifecycle state drives startup, readiness, liveness, and exit?
6. Which target, libraries, filesystem, limits, and supervisor behavior were actually tested?

These are coupled decisions. Queue capacity from configuration becomes a memory commitment. The supervisor’s termination grace period bounds application drain time. A dynamically linked binary depends on the runtime image’s ABI and libraries. A readiness probe can remove traffic only if the application first changes state and the routing system has time to observe it.

Resolve configuration as one typed value

Configuration sources are not interchangeable bags of strings. Defaults are a product decision; a file is a deployable baseline; environment variables are process-level overrides; command-line flags are an explicit invocation. Choose and document one order. The lab uses:

compiled defaults < configuration file < environment < command line

Later sources override earlier sources field by field. Parse each source into a partial typed value, merge once, and validate the complete result. Do not let subsystems independently read environment variables: that hides precedence, makes tests order-dependent, and permits two components to disagree about the effective setting.

let config = Config::resolve(file_values, env_values, cli_values)?;

The fixture validates relationships, not only syntax. concurrency must be nonzero and no larger than queue_capacity; shutdown time must fit an operational range. A string that parses as usize can still promise more memory than the container owns. Convert capacities into bytes, file descriptors, tasks, connections, and worst-case drain time before marking the process ready.

Unknown keys need a policy. Rejecting them catches misspellings but can complicate mixed-version rollout. Ignoring them aids forward compatibility but can silently disable an intended setting. A practical service can reject unknown keys in its owned file, allow a namespaced compatibility set during rollout, and report deprecated keys. Record the effective non-secret configuration with source attribution and a stable hash. Never include secret values in that snapshot.

The process map is a rehearsal order, not merely a component inventory: ordinary configuration gains authority from left to right, secrets remain on a separate rendering path, and a stop event must close admission before drain and exit.

Treat reads and writes as progress, not transactions

Read::read and Write::write report progress. A successful write can consume only a prefix. A successful read can return fewer bytes than requested without reaching the logical message boundary. Stream protocols therefore need framing and loops; file formats need length and integrity checks; callers must distinguish interruption, temporary unavailability, end of stream, truncation, and malformed content.

Rust’s write_all loops until the buffer is consumed or a non-interrupted error occurs. The fixture supplies a writer that accepts at most three bytes per call:

let mut out = ShortWriter { bytes: vec![], max_chunk: 3 };
out.write_all(b"relay-state")?;

Its test fails if code assumes one write means complete output. The same discipline applies to async APIs, though readiness and cancellation semantics come from the chosen runtime and resource. Cancellation after partial network output may leave a valid prefix visible to the peer. Protocols need an idempotency key, transaction identifier, or explicit resume/reject rule; “the future was dropped” does not retract bytes.

Buffering changes syscall frequency and latency, not the logical completion contract. BufWriter::flush pushes user-space buffered bytes to its underlying writer. It does not universally prove that a file survived power loss or that a remote peer processed data. Flush errors also matter: relying only on destructor cleanup loses the ability to report them. Size buffers from workload evidence and memory budgets, and bound flushing during shutdown.

Replace files without claiming more than the platform guarantees

Updating a state or configuration file in place exposes torn content to readers. A stronger pattern is:

1. create a temporary file in the destination directory;
2. write the full new representation and verify its integrity;
3. flush user-space buffers;
4. request file synchronization when the durability contract requires it;
5. atomically replace the destination using an operation supported by that filesystem and platform;
6. synchronize the containing directory when required to persist the name change;
7. retain or remove recovery artifacts according to a tested policy.

Do not compress that into “rename is atomic.” Rust documents platform-specific behavior and errors for std::fs::rename; filesystems, operating systems, replacement semantics, open handles, network mounts, and crash durability differ. Same-directory temporary placement avoids a common cross-filesystem failure but does not erase platform distinctions. Test crash recovery and concurrent readers on every supported storage class.

For append-only records, define record framing, checksums, maximum record length, and handling of a truncated tail. For stdout and stderr, remember that pipes can close and writes can fail. Decide whether a broken output pipe is normal termination for a CLI, a failed export for a worker, or a process fault for a service.

Keep secrets out of general-purpose values

Environment variables and mounted files are delivery mechanisms, not secrecy guarantees. Environment values may be inherited by child processes or exposed through diagnostic surfaces depending on the operating system and permissions. Files bring ownership, mode, path traversal, replacement, and mount semantics. Command-line secrets can appear in process listings and shell histories. Prefer a narrow secret source supported by the deployment environment and document its exposure boundaries.

Parse secrets into a type whose Debug and Display implementations redact. Avoid Clone unless duplication is required. Pass a borrow or capability only to the component that needs it. Do not include secrets in URLs, panic messages, tracing fields, configuration snapshots, metrics, or error source strings. The lab’s redact_secret only demonstrates the rendering boundary; it does not promise secure erasure:

<redacted:32 bytes>

Rust ownership can shorten logical lifetime, but dropping a String does not guarantee that every prior copy, allocator page, core dump, swap location, kernel buffer, TLS library, or remote system erased the bytes. If zeroization is a requirement, select and verify a mechanism against compiler, allocator, crash, and platform behavior. Rotation also needs two-version overlap, reconnect behavior, and observable failure without value disclosure.

Convert one stop request into one drain

Unix signals, Windows console control events, service managers, and container runtimes expose different mechanisms. Keep the platform adapter small: translate supported termination events into an internal cancellation or lifecycle transition, then run one portable shutdown state machine.

The relay lifecycle is Starting → Ready → Draining → Stopped. A termination request must be idempotent.

• Change readiness or routing state and close new admission.
• Stop producers before consumers so no new work appears behind the drain.
• Give in-flight work a deadline smaller than the supervisor’s remaining grace period.
• Flush only bounded, named resources; do not wait forever for telemetry or a peer.
• Cancel, persist, or reject remaining work according to its ownership contract.
• Join owned tasks and report forced cancellations.
• Exit success only when the documented completion condition was met.

The fixture refuses to enter Stopped while in_flight != 0. Production code also needs a second-stop policy. A repeated termination event might shorten the deadline or force exit, but it must not start a second competing drain. An uncatchable kill provides no cleanup opportunity, so correctness cannot depend exclusively on graceful shutdown.

Exit codes are an API to supervisors and scripts. Reserve zero for the documented successful outcomes. Distinguish invalid startup configuration from transient dependency failure and from internal invariant failure when the supervisor benefits from different restart policy. Avoid returning zero merely because an error was logged.

Align probes with lifecycle rather than implementation detail

A startup probe asks whether initialization finished. Readiness asks whether this instance should receive new work. Liveness asks whether restart is the desired recovery. They are control inputs, not monitoring substitutes.

During startup, load and validate configuration, acquire essential resources, recover durable state, and bind listeners before advertising readiness in the order your routing environment expects. A slow startup should not be mistaken for deadlock; Kubernetes startup probes can defer liveness and readiness checks until startup succeeds. A saturated queue may justify temporary unready state only if removing this instance actually reduces harm. Restarting every overloaded replica through liveness usually shifts more load to fewer survivors.

During termination, Kubernetes and other orchestrators have their own endpoint-removal and grace-period ordering. Measure the actual sequence rather than assuming a readiness change is instantly observed everywhere. Budget propagation delay, request duration, application drain, telemetry flush, and a safety margin inside the configured grace period.

Deploy the artifact you verified

Containers package a filesystem and process contract; they do not make Linux behavior portable to every target. Run the service as the intended user, with a read-only root filesystem where practical, explicit writable paths, a real init/supervisor strategy when it spawns children, and a deterministic working directory. Do not rely on a shell wrapper to forward signals unless it is tested.

Static linking can simplify runtime dependencies but may change resolver, TLS, locale, plugin, licensing, security-update, and debugging behavior. Dynamic linking can reduce duplication and allow library servicing but creates an ABI and image-content dependency. Neither is inherently “production.” Record target triple, libc choice, linked libraries, CPU assumptions, build ID, panic strategy, and debug-symbol retention from the artifact chapter’s evidence.

Resource limits are design inputs. Memory limits include allocator behavior, thread stacks, code, mapped files, queues, buffers, telemetry, and native libraries—not just Rust heap objects. File-descriptor limits cover listeners, connections, files, pipes, and observability exporters. CPU quotas can turn runnable time into throttled queue age. Test near the actual limits and make allocation or spawn failure behavior explicit.

Platform differences belong in the support matrix: path and rename semantics, case sensitivity, signals/control events, dynamic libraries, certificate stores, DNS behavior, clock behavior, line endings, file locking, permissions, and available resource controls. Conditional compilation should isolate mechanisms behind one process contract, not silently remove obligations.

Rehearse the relay deployment

Use this checklist as an executable review artifact:

• Resolve defaults, file, environment, and command line in the documented order; reject an invalid cross-field combination.
• Start with a missing, unreadable, malformed, unknown-key, and deprecated configuration.
• Verify effective configuration output contains source attribution and no secret value.
• Force partial reads/writes, interrupted operations, disk-full behavior, broken pipes, and a truncated state record.
• Crash before and after each file-replacement step; verify old/new/recovery outcomes on the supported filesystem.
• Rotate a credential while requests are active; inspect logs, errors, panic output, and dumps for disclosure.
• Send the supported stop event during startup, idle service, full queue, active write, and dependency stall.
• Prove admission closes, work reaches its declared outcome, joins finish, and exit status matches that outcome.
• Measure routing propagation and drain under the supervisor’s grace period; then force termination at the deadline.
• Exercise startup, readiness, and liveness failures separately and observe the platform response.
• Run at CPU, memory, descriptor, and storage limits using the exact release artifact and runtime image.

An operable process is not one that usually shuts down cleanly. It is one whose partial work, configuration authority, credential exposure, state transitions, artifact dependencies, and forced termination outcomes are reviewable and repeatable. Those finite resource budgets are now hard inputs: when demand exceeds them, the system must reject or degrade without abandoning the process contract.

Sources and version notes

• std::io::Read, Write, and BufWriter define partial-progress and buffering behavior.
• std::fs::rename, File::sync_all, and OpenOptions document Rust’s portable surface while retaining platform-specific qualifications.
• Tokio signal handling documents feature-gated Unix and Windows adapters and their limitations; pin the crate version used by production.
• Kubernetes container lifecycle hooks and startup, readiness, and liveness probes define current orchestration behavior; verify the deployed cluster version and configuration.
• Linux signal(7), rename(2), and getrlimit(2) are Linux-specific references, not portable Rust guarantees.

The fixture targets Rust 2024 with declared MSRV 1.85 and no third-party dependencies. It proves deterministic configuration, short-write, lifecycle, admission, and capacity properties; it does not simulate filesystem crash durability, real signals, containers, or platform resource enforcement.