Appendix O — Performance Experiment Template

The candidate is 9.03% faster on the primary workload in the completed record that accompanies this appendix. The predeclared minimum useful improvement is 8%. Every listed guardrail passes. Is the change approved?

Not for production. The recorded decision is adopt for a component trial. The experiment isolates the parser scanner on one supported host class; it does not establish end-to-end capacity, saturation behavior, or results on another target. A number crosses a threshold only inside the question, workload, artifact, environment, and uncertainty that gave the number meaning.

Use this template to create a decision record, not a benchmark scrapbook. Its contract is:

A performance decision is reviewable only when another engineer can identify the compared artifacts, reproduce the protocol, inspect ordered raw observations, test the proposed mechanism, apply predeclared practical and uncertainty thresholds, and know what operational action follows.

The file-backed worked record is in examples/rust-engineering-handbook/appendices/experiment-security-review-pack/experiment-record.json. The benchmark implementation it references is the canonical examples/rust-engineering-handbook/part-14/benchmark-design-lab.

The experiment dossier at a glance

decision
   │ constrains
   ▼
question ── predicts ──> hypothesis and mechanism
   │                         │
   ├── selects ──────────────┴──> workload families
   ├── fixes ───────────────────> baseline and candidate artifacts
   ├── declares ────────────────> primary metric + guardrails
   └── bounds ──────────────────> environment + protocol
                                      │
                                      ▼
ordered raw observations ──> variance and effect analysis
                                      │
                                      ▼
                         result ─> interpretation ─> decision
                                                        │
                                                        ▼
                                         regression and rebaseline policy

Every arrow is an audit question. If the workload does not derive from the question, a precise result answers the wrong question. If the result cannot be traced to artifact digests and raw observations, it is a claim without custody. If the decision is absent, the experiment can continue indefinitely or be reinterpreted after the numbers are known.

Record the decision before the stopwatch

The first page of a dossier should be understandable without opening the harness.

“Optimize parsing” is not a question. “New implementation should be faster” is not a hypothesis. Name the causal mechanism: fewer allocations, fewer copied bytes, a smaller contention window, fewer syscalls, better locality, less dispatch, or reduced algorithmic work. Then select at least one workload that should amplify the mechanism and one that should not. If every workload moves equally, the proposed explanation may be wrong even if the change is useful.

Write the null outcome in operational language. A useful record says, “If improvement is below 8%, retain the baseline because the extra state-machine complexity is not justified,” or “If allocation grows above zero, reject regardless of latency.” That prevents the team from inventing a more convenient rule after seeing 7.6%.

Bind the evidence to executable artifacts

Source revision alone is not an artifact identity. Performance can change with the compiler, target, profile, features, link settings, dependencies, generated code, and environment variables. Record both baseline and candidate with:

• source revision and dirty-state disposition;
• binary or package digest when the experiment measures a built artifact;
• Rust and Cargo versions, edition, target triple, and enabled features;
• Cargo profile, optimization level, LTO, code-generation units, panic strategy, target CPU flags, and linker when relevant;
• benchmark harness and corpus revision;
• exact build and execution command;
• dependency lockfile identity and native-library versions;
• semantic-equivalence test result.

The worked record intentionally uses teaching digests rather than pretending to identify a release. A real dossier obtains digests from the built files and preserves the command output. If a candidate and baseline use different profiles, the experiment is comparing profiles as well as code. That can be the question, but it must be explicit.

Before timing, run correctness checks. The scanner candidates must agree about fields, escapes, checksum, malformed inputs, and limits. An implementation that skips validation may be faster because it implements a weaker contract. Benchmark tooling cannot detect that semantic substitution.

Describe workloads as a population model

A corpus is evidence only when its relationship to expected use is stated. The worked record names three families:

For each family, record size and shape distributions, source, sampling or generation method, seed, weights, protocol maxima, cache and reuse policy, concurrency, and whether setup and destruction are inside the measured boundary. Production traces require privacy review and stable sanitization; generated data requires a reviewed model. “Realistic” is not a reproducible description.

Workload weights answer a product question; unweighted families answer sensitivity questions. Preserve both. A weighted average can conceal a severe regression in a low-frequency but high-consequence request. Report the family-level result and guardrails before any aggregate.

Do not make every experiment micro-scale. Use the smallest boundary that can identify the mechanism, then add component or macro evidence when the decision concerns user latency, capacity, queueing, or operating cost. A nanosecond scanner result cannot by itself justify a worker-count reduction.

Freeze the protocol, not the machine’s humanity

Environment control removes irrelevant variation; it should not remove the production condition being studied. Record host class, CPU model and topology, frequency policy, affinity, memory and NUMA placement, kernel, allocator, container or VM limits, background activity, thermal observation, and clock source as applicable.

The protocol must state:

1. Warmup: which transient is being settled and how the condition is detected. For startup questions, explicitly preserve cold observations.
2. Ordering: alternate or randomize baseline and candidate so temperature, drift, and background load do not map consistently to one artifact.
3. Sample unit: distinguish inner iterations from independent batches, process runs, host runs, or replay windows.
4. Stopping rule: choose sample count or precision criteria before inspection.
5. Exclusion rule: discard a whole paired observation only for a recorded protocol failure, not because it looks unfavorable.
6. Retention: keep ordered raw values, errors, timeouts, censored observations, host records, and harness logs.

The worked example uses twelve process-level paired batch medians in an ABBA order. Twelve is not a universal adequate sample size. It is enough for the verifier to demonstrate custody and arithmetic. A production study should choose replication based on observed variance, effect threshold, and the independent unit at which noise enters.

Warmup and pairing do different work. Warmup controls a predicted transient. Pairing controls some shared host drift. Neither makes an overloaded laptop equivalent to a production worker, and neither turns dependent loop iterations into independent observations.

Declare metrics and guardrails together

Choose one primary metric for the decision. Additional metrics should diagnose the mechanism or protect contracts, not offer more chances to find a green result.

The parser record uses paired median nanoseconds for the escape-heavy family. Its guardrails are:

• zero measured global-allocator calls per record;
• identical malformed-input acceptance and rejection;
• no more than 3% regression on typical 96-byte records.

Other experiments may need p95 or p99 latency, throughput at fixed offered load, CPU-seconds per unit, peak memory, retained bytes, binary size, energy, lock wait, syscall count, error rate, timeout rate, or fairness. Every metric needs unit, aggregation, measurement point, and sampling interval. “CPU improved” does not say whether the record means process time, normalized cores, host utilization, cycles, or cost.

Latency percentiles require enough independent observations and an explicit population. Throughput requires offered load, concurrency, duration, errors, and latency alongside it. Allocation instrumentation must say which allocator and process it observes. Profiler samples explain where time was observed; they are not exact accounting and do not replace end-to-end measurements.

Preserve raw observations and calculate the narrow result

The worked record retains these ordered batch medians:

baseline ns: 830 818 826 822 835 821 829 824 817 832 823 828
candidate ns: 754 746 751 749 759 748 753 750 744 756 747 752

The medians are 825 ns and 750.5 ns. The relative improvement is:

(825 - 750.5) / 825 × 100 = 9.03%

The result statement is deliberately literal:

Under the recorded protocol, the candidate’s paired-sample median for the escape-heavy teaching workload improved by 9.03%; all three declared guardrails passed.

The interpretation is a separate sentence:

The direction and workload sensitivity are consistent with removing a second scan, but this component experiment does not estimate end-to-end capacity or other targets.

The decision follows:

Adopt the candidate for a representative component trial; do not promote it on this record alone.

Keeping result, interpretation, and decision separate prevents a measured fact from silently becoming a causal claim and then a deployment authorization.

Inspect spread and time order before summarizing. Report an interval or other uncertainty analysis appropriate to the independent sample unit. If the effect estimate overlaps the predeclared decision boundary, record inconclusive under this protocol. That is a valid result. Commission more independent runs, reduce an identified noise source, redesign the workload, or decide that more evidence costs more than the decision is worth.

Do not automatically trim outliers. A long observation can be interruption noise, thermal throttling, allocator behavior, lock convoying, page fault, or the failure mode the system will experience. Classify it with corroborating evidence. Never remove only one side of a pair.

Convert the decision into a regression contract

A benchmark result decays when its workload, host, compiler, or product contract changes. Preserve a policy, not a magic historical number.

The example policy investigates a relative median regression of 5% against a colocated locked baseline, rejects any allocation above zero, and rebaselines only when a reviewed trigger occurs:

• workload model or protocol changes;
• supported target or host class changes;
• compiler, linker, dependency graph, or profile changes;
• an intentional product-contract change alters the measured work;
• the measurement implementation is corrected.

A rebaseline record must preserve the old baseline, explain why comparison is no longer like-for-like, identify the approving owner, and establish the new threshold from fresh evidence. Never rebaseline merely because a candidate fails.

Use absolute budgets for stable capacity contracts on controlled infrastructure. Use colocated relative comparisons when hosts vary but candidate and baseline can share a job. Use historical control bands to detect drift, with explicit host strata and maintenance. One policy may combine them: a relative code-change gate plus an absolute release-capacity test.

Performance gates should fail informatively. Preserve the artifact pair, raw data, environment, and failed criterion. A flaky gate that engineers reflexively rerun until green has no decision authority.

Copyable experiment record

The following is the minimum record shape. Copy the completed JSON fixture when machine validation is useful; use this compact form in a design or pull request.

# Performance decision record

## Decision
- Owner and deadline
- Candidate action for pass, fail, and inconclusive outcomes
- Explicit exclusions
- Evidence expiry and the event that forces a fresh decision

## Question and hypothesis
- Falsifiable question with population, primary metric, practical threshold, and guardrails
- Predicted mechanism and sensitivity across workloads

## Compared artifacts
- Baseline and candidate revisions, digests, profile, target, features, lockfile, harness, and corpus
- Semantic-equivalence evidence

## Workloads and boundary
- Families, distributions, source or generator, weights, limits, cache/reuse policy, concurrency
- Setup, timed work, returned observation, and destruction

## Environment and protocol
- Hardware, OS, isolation, allocator, toolchain, command
- Warmup, ordering, independent sample unit, stopping rule, exclusion rule, raw-data location

## Results
- Ordered raw observations
- Primary effect and uncertainty
- Guardrail outcomes, failures, timeouts, and censored data
- Decision-boundary comparison using the predeclared practical threshold

## Interpretation and decision
- Literal result
- Mechanism interpretation and competing explanations
- Decision, follow-up evidence, owner, and expiry

## Regression policy
- Relative or absolute threshold, noise rule, host policy, rebaseline triggers, and steward
- Investigation action, evidence retention, expiry, and approval required after a failed gate

Exercise: review an inconclusive optimization

Create a dossier for a proposed change from Arc<Mutex<HashMap<...>>> to sharded ownership. Use two workload families: ordinary lookups and a hot-key distribution. Require throughput at fixed offered load as the primary metric, with p99 latency, fairness across tenants, allocations, and shutdown time as guardrails.

Then give yourself this result: throughput improves 11%, but the interval for p99 change spans from 2% better to 14% worse, and the hot tenant receives a disproportionate share of service. Complete the record without changing its thresholds. A strong submission:

1. records an inconclusive or rejected decision rather than averaging guardrails away;
2. identifies sharding and scheduler interaction as competing mechanisms;
3. retains per-tenant and time-ordered data;
4. proposes a bounded follow-up with independent process or host replications;
5. states the ordering and shutdown correctness evidence needed separately from performance;
6. defines rebaseline conditions without using the failed candidate as the new normal.

Performance experiment audit card

Before accepting a performance claim, confirm:

• the question names a decision, population, metric, useful effect, guardrails, owner, and deadline;
• the hypothesis predicts a mechanism and workload-sensitive pattern;
• baseline and candidate identify source, artifact, toolchain, target, profile, features, and dependencies;
• workloads have reviewed distributions, limits, provenance, and measurement boundaries;
• semantic equivalence passes before timing;
• warmup, order, independent sample unit, stop rule, exclusions, and retention were predeclared;
• raw ordered data, failures, variance, effect size, and uncertainty remain inspectable;
• result, interpretation, and decision are separate statements;
• an inconclusive result remains an allowed outcome;
• the regression policy names thresholds, host policy, rebaseline triggers, and owner.

If any item is missing, the record may still guide investigation, but it cannot carry a release or architecture decision.

A performance dossier authorizes only its named operational decision. It does not establish that the measured artifact is safe to release: dependency execution, unsafe and FFI surfaces, input limits, secrets, provenance, and advisory response still need their own custody and stop-ship evidence.

Sources and version notes

• std::hint::black_box documents a best-effort optimization barrier and explicitly does not provide correctness or cryptographic guarantees.
• The Cargo profiles reference defines profile settings that can change the measured artifact; record the effective profile rather than assuming release is universal.
• cargo bench documents Cargo’s benchmark command and bench profile behavior. Harness and statistical-tool behavior remain tool-version-sensitive.
• Cargo target documentation for explicit benchmark targets defines the manifest surface used by the canonical Chapter 84 fixture; inspect the checked-in fixture and its README for the actual protocol.

The worked fixture targets Rust 2024 and records Rust 1.97.0. Its numeric observations are intentionally synthetic teaching data verified for internal arithmetic; they are not measurements of the current host and must not be used as performance budgets. The canonical Rust benchmark lab provides executable semantics and harness evidence. Revalidate tool options, target behavior, measurement code, and statistics for the publication toolchain and supported deployment environments.