Appendix H — Error and Panic Design Checklist

An operator does not need this message:

Error: Custom { kind: PermissionDenied,
  error: "open /srv/relay/tenants/acme/token-prod-eu.txt: token=s3cr3t" }

It exposes a tenant, a deployment path, and a credential. It says nothing about whether traffic is being rejected, whether retrying is useful, which component owns recovery, or how to correlate the event. Replacing it with “I/O error” stops the leak but destroys the diagnosis.

A useful report is deliberately split:

operator: relay could not load tenant credentials; traffic for tenant_ref=t-184 is paused
action:   verify secret-store access; resume the tenant after correction
retry:    after operator action, within the existing 10-minute recovery budget
trace:    incident=74291 operation=load_credentials error_class=configuration
source:   permission denied                 # protected diagnostic channel

Error design is the journey from detection to action. Review it outside-in: what a caller can match, what an operator can safely read, what diagnostics retain, what retry machinery does, what a process returns, and only then how the leaf error is represented. A polished enum is insufficient if the system retries permanent corruption, loses the source, leaks the input, or catches a panic and continues with broken state.

The failure-journey ledger

For each failure class, fill one row. “Log and return error” is not a complete entry.

The row forces three distinctions that are often collapsed: classification is not presentation, retryability is not transience alone, and panic containment is not recovery proof.

Classify by caller action

A taxonomy should predict what the next layer can do. Useful top-level classes often include:

• Invalid input or protocol: the caller must change data, version, or request.
• Conflict or precondition: the caller may refresh state or choose another operation.
• Transient dependency: another attempt may succeed within a bounded policy.
• Permanent dependency or configuration: an operator or deployment change is required.
• Resource exhaustion: shed load, wait, resize, or reject according to ownership and budgets.
• Cancellation or deadline: preserve whether work stopped, may still complete elsewhere, or can be repeated.
• Bug or invariant violation: continuing may expose corrupted logical state; preserve evidence and apply containment policy.

Do not create one enum variant for every message emitted by every dependency. Conversely, do not collapse all external failures into Unavailable when callers must distinguish malformed input from a timed-out idempotent read. The right granularity is the set of stable decisions the public caller is authorized to make.

The companion fixture uses a small standard-library-only type:

#[non_exhaustive]
pub enum RelayError {
    InvalidFrame(NumericError),
    Read { operation: &'static str, source: io::Error },
    CapacityExhausted { resource: &'static str },
    InternalInvariant { incident_id: u64 },
    PanickedPlugin,
}

It is evidence for mechanics, not a universal taxonomy. A real service may separate library errors from transport responses and operational incidents. The public type should expose only distinctions callers need; internal diagnostic types can be richer.

Treat public errors as compatibility surface

A public error enum is part of the API. Adding a variant can break downstream exhaustive matches unless evolution was designed in. #[non_exhaustive] requires external callers to retain a wildcard arm and prevents constructing the enum freely outside its defining crate. It buys room for variants; it does not make semantic changes harmless.

Review stability at three levels:

1. Type shape: enum, opaque struct, trait object, or generic error parameter.
2. machine contract: variants, accessor methods, stable codes, retry classification, and source availability.
3. human contract: Display text, diagnostic fields, documentation, and localization expectations.

Do not encourage callers to parse Display. Provide a stable variant, code, or accessor for machine decisions. Human text may improve without a major version change if documented as unstable prose. If exact text is a protocol, say so and test it—but a typed code is normally stronger.

Libraries generally return typed errors rather than printing or exiting. Applications translate domain errors into transport status, telemetry, operator messages, and ExitCode at their outer boundaries. Keeping that translation outside the library prevents a reusable parser from deciding that one malformed record should terminate a process.

Add context without flattening the source

Context answers “what was the system trying to do?” The leaf error often answers only “what failed?” Compare:

permission denied

with:

unable to read relay credential bundle
caused by: permission denied

The outer error should name the stable operation, relevant safe identifiers, and perhaps the phase or attempt. It should not merely repeat the source. Implement std::error::Error::source so diagnostic code can traverse structured causes:

impl Error for RelayError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Read { source, .. } => Some(source),
            Self::InvalidFrame(source) => Some(source),
            _ => None,
        }
    }
}

The standard-library guidance is important: a wrapped source should normally be returned by source() or rendered into the outer Display, not both. Rendering it at every layer produces duplicated chains such as “read failed: I/O failed: permission denied: permission denied.” Keep each layer’s Display concise; let the final reporter decide how to walk the chain.

Do not stringify an error early. map_err(|e| format!(...)) loses its concrete class, source traversal, and often useful fields. Wrap it in a type that stores e. At dynamic application boundaries, Box<dyn Error + Send + Sync> can preserve a chain when callers do not need variant matching, but it still needs explicit presentation and redaction policy.

Context also has a cardinality cost. Raw URLs, file paths, user IDs, SQL, request bodies, and generated messages can create unbounded metric labels and high-volume logs. Put bounded class codes in metrics; keep richer safe context in structured logs or traces with retention and access controls.

Redact before formatting

Secrets can appear in input, file paths, URLs, headers, database errors, subprocess arguments, and dependency messages. Redaction is a data-model decision, not a last-minute regular expression.

Separate fields into lanes:

Do not assume Debug is safe. Derived Debug prints fields, including a String that may contain a credential. Store a redacted identifier instead of the secret where possible, wrap sensitive values in types with intentionally redacted Debug, and test that both Display and Debug outputs do not expose sentinel secrets. Panic payloads and assertion messages follow the same rule because panic hooks often send them to logs.

An operator message should answer four things in order: impact, safe identity, action, correlation. The source chain belongs in a separately controlled diagnostic rendering. The fixture’s operator_report intentionally does not render its io::Error source; a production reporter could attach that source to a protected trace.

Reserve panics for violated execution assumptions

Use Result when the caller can reasonably encounter and handle the condition: malformed input, unavailable dependency, permission failure, timeout, cancellation, capacity rejection, or missing configuration. Panic is appropriate when continuing the current operation would mean the program’s own invariant or documented precondition has been violated and ordinary callers are not expected to recover locally.

Even then, record the panic contract:

• Which condition proves a bug rather than hostile input?
• Can partially updated state be observed after unwinding?
• Which destructors run, and do they perform fallible or blocking work?
• Does a lock become poisoned, and who validates it before recovery?
• Is the binary built with unwind or abort strategy on every target?
• Can the panic reach an FFI boundary, thread root, task runtime, or host callback?
• What evidence survives: incident ID, hook output, backtrace policy, minidump, or core dump?

unwrap and expect are panic sites. expect can document an invariant for maintainers, but its message must not reveal secrets. In tests and one-time construction where impossibility is locally evident, either may be reasonable. In a long-lived request path, an external Err is not made impossible by optimism.

Unwinding is a control path, not an exception API

With the unwind panic strategy, Rust unwinds frames and drops live values until a catch or thread boundary. With the abort strategy, the process terminates without unwinding. Not all targets support unwinding, and a double panic during unwinding can abort. Cleanup that must occur after either strategy belongs in process supervision, durable leases, protocol timeouts, or other external recovery—not only in Drop.

catch_unwind catches unwinding Rust panics, not aborts. It is primarily useful at narrow containment boundaries such as invoking an extension callback when the host can prove that failed state is isolated. It should not turn every panic into a routine Err and continue. Panic payload destruction can itself panic, and foreign unwinding has additional restrictions.

UnwindSafe and RefUnwindSafe are advisory “speed bumps” around witnessing broken logical invariants. AssertUnwindSafe is a claim by the author, not a repair. Before using it, list every captured mutable value and prove what state the caller may observe after each panic point. The fixture uses it around a zero-state example callback solely to demonstrate mechanics; a real plugin host needs a transactional or disposable state boundary.

Never permit unwinding across an ABI that does not allow it. Define the FFI policy explicitly: convert ordinary errors to an ABI result, contain permitted Rust unwinds on the Rust side when feasible, and use the correct ABI and toolchain guidance for cross-language unwinding. “Catches panics” is not an FFI safety argument.

Retry is a protocol with an owner

An error variant named Transient does not authorize a retry. A retry policy needs all of these:

• classification: which exact failures may improve without changing the request;
• idempotency: whether repeating can duplicate or corrupt effects;
• owner: caller, client library, worker, proxy, queue, or operator—but not all of them independently;
• budget: maximum attempts or elapsed time, including time already spent upstream;
• schedule: exponential or other backoff, jitter, and server hints;
• deadline/cancellation: whether another attempt can finish within the remaining contract;
• load effect: whether retries amplify an overload;
• observability: one logical operation ID plus attempt count, not unrelated incidents;
• exhaustion result: final caller response and operator signal.

Classify action separately from cause. A connection reset during an idempotent read may be retryable. The same reset after an unacknowledged payment write is an ambiguous outcome, not permission to repeat. Malformed input remains permanent even if it arrived from a usually transient network dependency. Resource exhaustion may require load shedding rather than another immediate attempt.

Exercise the exhaustion path. If five layers each retry three times, the system can produce hundreds of leaf attempts. Trace which layer owns the budget and propagate remaining deadline and attempt metadata.

Map process status at the outermost boundary

Rust’s ExitCode represents normal termination of the current process and supplies portable SUCCESS and FAILURE values. Arbitrary numeric values do not have portable meanings; platforms and supervisors may mask or reinterpret them. If an operational environment defines a small code convention, document that environment, keep the mapping at main, and test it there.

Prefer returning from main so normal stack cleanup occurs. process::exit terminates without running destructors on the current or other thread stacks. Neither return nor exit replaces an orderly shutdown protocol: stop admission, signal workers, drain or abandon according to policy, flush bounded outputs, release leases, then return status.

Services should not exit for every request error. Decide scope:

• reject one request;
• isolate one tenant, peer, partition, job, or plugin;
• restart one worker;
• stop the process for supervisor replacement;
• abort because memory or invariant integrity cannot be trusted.

The exit code reports a final process disposition. It does not carry the full error taxonomy; structured diagnostics and supervisor metadata do that work.

Failure-contract exercises

Classify six failures

For malformed JSON, expired credentials, connection refused, deadline after a possibly committed write, queue capacity exhaustion, and an impossible enum state, fill the failure-journey ledger. For each, name the authorized caller action, retry owner, idempotency evidence, containment scope, and safe operator message. If two failures share a class but require different action, refine the class or expose another stable accessor.

Rewrite a leaking report

Start with:

upload failed for customer@example.com to
https://user:password@storage.internal/acme?token=abc123:
request body invoice-942.pdf rejected

Produce caller, operator, protected-diagnostic, and metric representations. Place a unique sentinel in every sensitive input and add tests proving the sentinel is absent from Display, Debug, operator output, metrics, and panic-hook fields.

Review an unwind boundary

Choose a plugin or callback boundary. Draw the state before invocation, every mutation during invocation, and state visible after a panic. Remove AssertUnwindSafe until the proof explains every captured mutable value. Run the test under unwind strategy, then verify the product’s abort-strategy behavior separately. If the process cannot safely continue, replace catching with process or worker isolation.

Rehearse exhaustion

Inject a retryable dependency failure. Observe attempt count, total elapsed time, jitter, cancellation, emitted metrics, final caller result, and operator alert. Then inject a permanent failure and prove only one attempt occurs. Finally inject an ambiguous write outcome and prove the system reconciles or surfaces uncertainty instead of blindly repeating the effect.

Error and panic review card

• Error classes correspond to stable caller actions rather than incidental leaf messages.
• Public variants, codes, accessors, non-exhaustiveness, and text-stability promises are documented.
• Libraries return errors; application boundaries own printing, transport mapping, and exit status.
• Every wrapper adds safe operation context and preserves a structured source where useful.
• The same source is not rendered redundantly at multiple layers.
• Caller, operator, protected-diagnostic, and metric lanes have explicit redaction and cardinality rules.
• Debug, panic messages, hooks, and backtraces follow the secrecy policy.
• Expected external failures return Result; panic conditions name the violated invariant.
• Unwind and abort behavior are tested or otherwise verified for supported targets and profiles.
• catch_unwind is narrow; post-panic state, payload handling, and UnwindSafe claims are reviewed.
• No unwind crosses an incompatible FFI boundary.
• Retry classification includes idempotency, one owner, budget, backoff, jitter, deadline, load effect, and exhaustion behavior.
• Operator messages state impact, safe identity, action, and correlation.
• Process status is mapped at the outer boundary and does not replace graceful shutdown.

A good failure contract preserves two things at once: the machine-readable decision a caller needs and the safe causal evidence an engineer needs. It does not force human text to serve as an API, retry policy, security filter, and incident report simultaneously.

The same ownership discipline applies to build and release evidence. A Cargo command needs an explicit scope, a failure needs an owner, and registry publication needs a separately authorized boundary. Appendix I makes those operational units visible.

Sources and version notes

• std::error::Error defines Display, Debug, and structured source() behavior, including guidance against rendering a wrapped source twice.
• The Rust Reference panic chapter distinguishes unwind and abort strategies and documents unwinding restrictions.
• std::panic::catch_unwind, UnwindSafe, and AssertUnwindSafe define the standard containment mechanisms and their limits.
• ExitCode documents portable success/failure values and cautions that arbitrary numeric meanings vary across platforms. process::exit documents destructor and interoperation consequences.
• RFC 2008 explains non-exhaustive public types; current compiler and Reference behavior remain authoritative.
• The companion fixture is examples/rust-engineering-handbook/appendices/numeric-error-policy-lab/. It targets Rust 2024 and declares Rust 1.85 as MSRV. Its containment example proves only that an unwind can be converted at a state-free callback boundary; it is not evidence that arbitrary plugins are unwind-safe.