Production Readiness and Performance Case Study

The review board’s answer is not yet.

relay-service has bounded admission, a measured parser, observable request context, explicit cancellation, a threat model, and a governed dependency policy. Its deterministic fixtures pass. None of those facts proves that the release candidate can survive the production workload, drain before the platform kills it, or restore compatible state after rollback. The board therefore records a no-go with explicit conditions for reopening the review: execute the release-candidate fault campaign, prove rollback against production-shaped data, and assign every accepted risk an owner and expiry.

That answer is more useful than enthusiasm or blanket rejection. It says exactly which evidence supports launch, which evidence is only a teaching-model result, and which missing observations can still stop the release. The readiness contract is: each launch claim must bind a versioned release candidate to a workload, environment, observable result, decision threshold, accountable owner, and retained artifact; uncertainty remains visible and can veto launch.

The production argument is complete only when its separate contracts become one reviewable case. A launch packet must show where capacity, failure, observability, security, and supply-chain evidence joins, without inventing a green dashboard for observations that were never made.

Put a release identity at the center

A readiness packet names one candidate, not “the main branch lately.” Record the source revision, Rust toolchain, dependency lockfile digest, build profile, target, image or package digest, configuration schema, database or durable-format version, SBOM identity, provenance record, benchmark bundle, and deployment manifest. If any material input changes, determine which evidence must be rerun.

The reference architecture has five ownership boundaries:

1. The API adapter authenticates the caller, validates the schema, assigns a request identifier and deadline, and owns the response contract.
2. Admission owns concurrency, queue, tenant-fairness, and memory budgets. A rejected request has not secretly entered background work.
3. Workers own accepted work until one terminal outcome: committed, rejected, expired, or cancelled. They do not detach unaccounted tasks.
4. The durable sink owns idempotency keys, commit visibility, and the acknowledgement boundary. A success response means the documented durable condition, not merely “queued locally.”
5. The lifecycle coordinator owns readiness, drain initiation, cancellation propagation, worker completion, telemetry flush limits, and final exit status.

Teams own boundaries too. API owners approve schemas and client compatibility; the platform team owns admission and deployment; service owners own processing and cancellation; data owners own sink semantics and migration; security owns threat-model review without becoming the sole owner of secure implementation. A box without an accountable team is not an architecture.

Read the board as a claim map, not as the release record. Its before/after percentages are conceptual labels—not fixture measurements—and the red block states preserve the written veto: fault injection and rollback evidence are still missing.

Review the external contracts before the internals

The launch board should be able to state what clients observe without reading worker code.

For ingestion, the request schema is versioned; unknown or malformed input fails without enqueueing; size and item limits are enforced before expensive parsing; authentication and tenant-scoped authorization precede mutation; and an accepted request receives an idempotency identity. The response taxonomy distinguishes invalid, unauthorized, overloaded, deadline-expired, dependency-unavailable, and internal outcomes. Retry guidance is part of the protocol: an overload response may be retryable with bounded jitter, while invalid input is not.

Success has a precise persistence meaning. If the public contract acknowledges only after the durable sink commits, latency and availability budgets must include that sink. If it acknowledges after an in-memory queue accepts work, the service must describe loss on process failure and cannot call the response durable. The reference decision is at-least-once processing with idempotent sink writes and acknowledgement after the commit boundary. Duplicate delivery is expected; duplicate effect is not.

Cancellation is also observable. Disconnect, deadline expiry, operator drain, and internal dependency failure enter one task tree. Cancellation before commit stops expensive work and yields no success acknowledgement. Cancellation racing with a completed commit consults the sink’s idempotency record instead of guessing. Once commit begins, an operation may become deliberately non-cancellable for a short bounded region; that boundary must be measured and surfaced during shutdown.

Compatibility covers more than JSON fields. It includes default values, validation limits, error codes, retry semantics, ordering, idempotency scope and retention, authentication requirements, metrics labels used in alerts, configuration precedence, durable formats, and rollback behavior. The candidate packet needs consumer tests or recorded compatibility probes for each supported client generation.

Turn capacity assumptions into inequalities

The service declares a workload envelope before it reports a throughput number. A defensible envelope includes typical, escaped, and maximum-size inputs; tenant distribution; burst length; arrival shape; sink latency distribution; duplicate rate; deadline mix; worker count; CPU and memory limit; deployment topology; and acceptable rejection rate. One average request size is not enough.

For a first-order memory bound, let Q be queued items, P95_payload the guarded payload size, C active workers, W95 per-worker working memory, and F fixed process cost:

memory_guard = F + Q × guarded_payload + C × guarded_working_set + telemetry_budget

Use checked arithmetic in configuration validation. The Chapter 89–90 fixture exercises precisely this failure: capacity_bytes returns None on overflow rather than wrapping an unsafe budget into a small number. In the deployed service, retained request metadata, allocator fragmentation, TLS buffers, runtime state, library caches, page cache assumptions, and crash-report headroom also belong in the model.

Capacity is not “the peak rate observed once.” The launch criterion might be: at the declared pod limit and representative sink latency, sustain the planned peak plus agreed headroom for 30 minutes; keep successful-request latency below the service objective; bound resident memory; keep the queue below its alert threshold; reject at admission rather than timing out after expensive work; and recover to the normal state after the burst without oscillation. The exact values come from the product SLO and infrastructure budget, not this manuscript.

Little’s Law can cross-check a stable interval: concurrency is approximately throughput multiplied by mean time in system. It does not excuse using means for tail-latency approval, and it does not apply blindly during an accumulating queue. If a measured run violates the relationship dramatically, check instrumentation boundaries, batching, retries, and whether the interval was actually stable.

Preserve benchmark evidence without promoting it beyond its scope

The parser benchmark fixture was run on 2026-07-13 with Rust 1.97.0, Cargo 1.97.0, x86_64-unknown-linux-gnu, Linux 7.0, and 12 reported CPUs. It uses an optimized dependency-free harness, 60 samples, 2,000 iterations per sample, warmup inside the harness, consumed checksums, and zero measured allocations in its counting allocator:

These results prove that the harness produced this distribution on this host. They do not establish production request latency, future compiler performance, cross-host equivalence, or a launch threshold. The release packet must rerun the benchmark on controlled hardware, compare against a source- and environment-matched baseline, retain raw output, and apply a preregistered regression rule. The macro load test must include admission, parsing, authentication, scheduling, sink I/O, telemetry, and deployment limits.

The scaling across these three inputs is plausible for a linear scan and supplies a useful guard against an accidental superlinear parser path. It still needs adversarial families—dense escapes, maximum field counts, malformed suffixes, and rejected oversized requests—because a friendly 64 KiB record does not answer Chapter 91’s algorithmic-denial question.

Explain what profiling changed

The profiling fixture processes 2,000 identical generated events through one bounded teaching pipeline. On the same host, its optimized baseline completed in 682 microseconds with a reported maximum queue age of 682 microseconds. The deliberately regressed classifier completed in 12,666 microseconds with a maximum queue age of 12,660 microseconds. Both accepted 2,000 events, retained bounded labels, reported zero telemetry drops, and produced the identical checksum 5308874708560; only the redundant CPU work differs.

The important result is causal: extra CPU work while the single worker owns classification lengthens service time, which increases queue age. A CPU profile should therefore attribute new samples to that classifier, while the queue-age signal shows the operational consequence. Removing or precomputing that work is justified only after an optimized rerun preserves functional outputs and improves the same workload.

This fixture is not a before/after optimization claim. It is a controlled regression used to verify that the chosen evidence detects the failure. The release review needs a profile of the actual candidate under the capacity workload, including CPU stacks, allocation behavior, syscalls or sink waits as appropriate, lock or task contention, queue depth, and a time-correlated trace. Absence of one suspected hotspot is evidence against that hypothesis, not evidence that the service is fast.

Retain the profiler, version, collection rate, symbolization recipe, workload window, and overhead assessment. Sampling percentages from different-duration runs cannot be compared as absolute time without care. A shorter bar after optimization can hide a slower total run; pair profiles with wall time, throughput, latency distribution, resource use, and correctness checks.

Join errors, cancellation, and observability

Every failure path needs an owner, client outcome, retry policy, telemetry signal, and cleanup result. Parse rejection is a bounded client error. Admission rejection is an overload outcome and increments a low-cardinality counter by reason. Sink timeout preserves the causal error chain, cancels sibling work, and does not acknowledge success. Panic is an internal defect: the process or supervised worker follows the declared containment policy, secrets are redacted, and readiness changes before unsafe traffic continues.

The observability contract answers operational questions rather than mirroring implementation. Required dimensions include request or trace identity, tenant only where access and cardinality policy allow it, endpoint, outcome enum, queue age, service duration, payload-size bucket, sink duration, retry count, and release identity. Never put raw secrets, arbitrary error text, object names, or unbounded tenant values into metric labels. Structured events may carry richer redacted context under access controls.

The candidate’s dashboard and alerts must distinguish:

• traffic admitted, rejected, completed, cancelled, and expired;
• end-to-end latency from queue wait and service time;
• queue occupancy, oldest age, worker saturation, and memory pressure;
• sink errors and commit latency;
• degraded-mode entry, recovery, and operator overrides;
• task leaks or incomplete drain at shutdown;
• panic or crash loops and rollout health;
• telemetry drops, exporter backpressure, and schema/version mismatches.

A green health endpoint is weak evidence. Readiness should turn false when the instance cannot safely accept new work; liveness should not kill a process merely because a downstream dependency is slow when restart would amplify the outage. Alert tests inject a known condition and verify that the intended page or ticket fires with a usable runbook link and release identity.

Carry the security and dependency cases into launch

The security packet names assets, actors, trust boundaries, authorities, interpreters, expensive operations, secrets, unsafe or FFI surfaces, and residual abuse cases. For relay-service, launch requires evidence that authentication is distinct from tenant-scoped authorization; input size and complexity are bounded before expensive work; path-like object names cannot escape the namespace; administrative mutations are audited and idempotent; secret material is redacted and short-lived; and overload controls cannot be bypassed through a privileged endpoint.

The Chapter 91 fixture’s lexical path policy is deliberately not a filesystem race defense. If the candidate touches a filesystem, review canonicalization authority, symlinks, directory handles, replacement races, permissions, and sandbox boundaries on the deployed platform. Likewise, Rust memory safety does not establish protocol authorization, constant-time cryptography, or bounded parser cost.

The dependency inventory links the exact lockfile and build graph to an SBOM for the actual artifact. Review direct and transitive packages, target-specific edges, enabled features, licenses under the organization’s authority, advisories, native code, unsafe code, build scripts, and procedural macros. Provenance binds source, builder, parameters, dependencies, and artifact digest. A signature proves endorsement by a key; the release decision still depends on key custody and verifier policy.

Launch blocks on an unresolved high-consequence advisory affecting a reachable deployed path, unreviewed build-time execution with valuable credentials, an unknown artifact identity, or inability to replace a critical component. A time-bounded exception may address a lower risk only with scope, owner, compensating control, approving authority, expiry, and tested exit.

Make failure injection falsify the design

The campaign is written before execution and runs against the release candidate in a production-shaped environment. Each injection states the hypothesized invariant, observation, stop condition, recovery procedure, and artifact to retain.

At minimum, exercise a full admission queue; expired requests while queued and while active; slow, failed, and partially successful sink operations; worker panic; process termination during commit; telemetry exporter stall; configuration rejection; secret provider loss; dependency DNS or connection failure; rolling upgrade with old and new schemas; shutdown during peak traffic; and rollback after a durable-format change. Include resource pressure without exceeding the safety of the test environment.

Passing means the specified behavior occurred: memory stayed bounded, overload failed early, duplicate effects were prevented, errors retained their causes without secrets, cancellations reached descendants, accepted work either committed or produced a documented retryable outcome, readiness changed appropriately, and recovery converged. “The service came back” is insufficient if it lost acknowledged work or created duplicates.

Fault injection also tests the incident interface. Can the operator see the active release, capacity mode, oldest queued work, dependency health, and safe controls? Does a control require authenticated, authorized, audited action? Can it worsen retry storms? Does the runbook explain when to shed work, pause rollout, drain, rollback, or escalate?

Prove shutdown, upgrade, and rollback as state transitions

Shutdown proceeds through explicit states: stop readiness; reject new admission; signal the task tree; let cancellable work stop; bound the commit region; drain accepted work until the budget; flush essential telemetry within its own smaller budget; persist or expose unresolved identities; and exit with a status the supervisor interprets correctly. The platform termination grace period must exceed the application drain budget plus scheduling and cleanup margin. Test the hard-kill boundary too.

Upgrade is a compatibility exercise. During a rolling deployment, old and new instances may coexist, read shared state, exchange messages, and emit telemetry. Prefer expand–migrate–contract changes: deploy readers tolerant of both forms, introduce new writes, migrate or observe, then remove the old form in a later release. Configuration and feature flags need the same forward/backward review.

Rollback proves more than “the old image starts.” The previous binary must understand current configuration, durable data, queued messages, protocol traffic, and dependency state. If a migration is irreversible, rollback may require a forward repair or traffic cutover instead of binary reversion; rehearse that path and name the decision authority. Backups are not a rollback plan until restoration time and data-loss bounds are measured.

Write the runbook around decisions

The incident runbook begins with symptoms and safe discriminators: overload, sink latency, crash loop, authorization failures, telemetry loss, or release regression. It names dashboards and queries, release and configuration lookup, expected healthy ranges, immediate containment, operator controls, rollback prerequisites, communication channel, data-integrity checks, security escalation, and evidence preservation.

Avoid a brittle list of shell commands without decision context. A good step says what observation justifies it, what authority it requires, what outcome confirms success, and what harm requires stopping. Test commands against least-privilege access. Keep a break-glass route whose use is audited and reviewed.

After stabilization, reconcile accepted requests with durable outcomes, identify duplicates or gaps, rotate exposed credentials if relevant, preserve traces and profiles, and create regression tests. The post-incident action owner and due date enter the risk register; the runbook update is part of completion.

The readiness scorecard

Status is not averaged. A single block remains a block even if eleven rows are green. Scores can help track work, but they must not convert a missing data-integrity proof into a 92% pass.

Final review-board exercise and written decision

Run the review as an adversarial evidence hearing. Assign API, service, platform, data, security, performance, and incident-command roles. Give the board the candidate identity, architecture map, contract tests, capacity plan, raw benchmark output, profiles, dashboards, threat model, dependency inventory, fault results, rollback record, runbook, and risk register. Each role must identify one claim the packet supports, one claim it cannot support, and one failure that should change the decision.

Then write a decision in this form:

Decision: NO-GO pending conversion to GO. The board accepts the bounded-admission architecture, API and cancellation model, parser benchmark method, profiling signal selection, observability schema, security review structure, and dependency-governance policy as suitable foundations. The reviewed fixtures passed 3 benchmark tests, 6 profiling/observability tests, 7 operability/resilience tests, and 5 security/supply-chain tests on Rust 1.97.0 and the declared Rust 1.85.0 MSRV. Those deterministic results are not release-candidate or production evidence. Launch remains blocked until the named candidate completes the production-shaped fault campaign and mixed-version rollback rehearsal, and until all residual risks have accountable owners, expiry dates, and monitored mitigations. The service owner may request a new review with retained raw artifacts. Any material source, toolchain, dependency, schema, durable-format, capacity, or deployment change invalidates the affected evidence.

Production readiness is not confidence accumulated from good code. It is a connected, falsifiable argument about a named artifact under declared conditions, with explicit authority to stop. Remove the allocator, operating system, and dynamic resources next, and the same method asks a sharper question: which contracts survive when the platform supplies far less?

Sources and evidence notes

• The executable evidence lives in examples/rust-engineering-handbook/part-14/benchmark-design-lab, profiling-observability-lab, operability-resilience-lab, and security-supply-chain-lab. Their results are bounded teaching evidence, not a deployed-service certification.
• Revalidate toolchain, environment, workload, target, dependency, and platform claims for the actual release candidate. Rust’s type system cannot prove operational capacity, external authorization policy, filesystem race safety, durable acknowledgement, or rollback viability.