Appendix F — Benchmark and Experiment Report Template

A result folder contains p99=47ms, a screenshot, and the phrase “three times faster.” The binary that produced it has changed. Nobody can say whether 47 ms includes queue wait, whether failed requests were excluded, or whether the client paused whenever the service paused. The number is precise; the evidence is unusable.

A benchmark report is a custody record for a claim. It binds an observation to the decision it can support, the population it represents, the system boundary and versions that produced it, the method used to reduce it, and the conditions under which it stops transferring. Its job is not to make a result look decisive. Its job is to let another engineer reproduce the result, challenge the inference, and reach the same or a better decision.

Use the copyable report at examples/performance-engineering-system-design-handbook/appendices/benchmark-report/benchmark-experiment-report-template.md. An abbreviated Mercury example and a structural verifier live beside it.

Appendices A–D supply units, statistical language, capacity laws, and dated calibration. Appendix E provides the review that consumes the evidence. Chapters 51 and 53 develop benchmarking and experimental design in depth. This appendix is the compact reporting instrument; it is not a substitute for those methods.

Put the claim before the measurements

Write the claim as a comparison or bound with a population, state, and metric. Then name the decision it changes.

For authorized Mercury base checkouts with 4–48 items, at 4,100 scheduled requests/s in the one-zone-loss topology, bounded optional enrichment keeps base-response p99 at or below 300 ms and valid goodput at or above 99.95%.

This claim can support a rollout gate for the bounded-enrichment design. It cannot establish recommendation value, cross-region behavior, or safety above 4,100 requests/s. Record the null or rival explanation as well: perhaps the apparent improvement comes from silently omitting difficult requests, a colder baseline cache, or a client that generated less work during stalls.

The first block of a report answers:

Choose acceptance rules before examining final results. Post-hoc thresholds turn noise into a story. If exploration changes the hypothesis, label the new result exploratory and require a fresh confirmatory run.

Draw a measurement boundary in words

Name where the clock, count, and responsibility begin and end. “API latency” is ambiguous: client wall time, server handler time, and dependency span duration describe different populations. Define the unit of useful work and what counts as valid completion. Count offered attempts, admitted work, total completions, valid completions, errors, timeouts, cancellations, and late effects separately.

Record the topology as a path table rather than decorative boxes:

Include trust, tenant, host, zone, and region boundaries when they affect interpretation. State which clocks are compared and how synchronization error is bounded. If client and server timestamps cannot be reconciled, do not fabricate an end-to-end decomposition.

Freeze the version manifest

“Same environment” is not a manifest. Retain identifiers sufficient to reconstruct code, configuration, infrastructure, data, and tools:

• source revision, dirty-state digest, build command, compiler/runtime, dependencies, and optimization flags;
• service, schema, feature-flag, protocol, kernel, container, firmware, and benchmark-tool versions where material;
• host or instance shape, CPU allocation, memory, storage, network, accelerator, placement, and power policy;
• topology, replica count, affinity, isolation, autoscaling, quotas, background work, and failure injection;
• sanitized configuration and secrets-independent environment variables;
• dataset generator revision, seed, schema, cardinality, distribution, skew, locality, and snapshot identity; and
• wall-clock date, timezone, run IDs, operator or automation identity, and immutable artifact paths.

Record unknowns. An honest unknown narrows transfer; an invented default corrupts it. Hash retained manifests, configs, raw files, and analysis code so later readers can detect drift.

Describe demand as the system received it

The workload section records operation mix, arrival model, requested and achieved rate, concurrency, payload/data distributions, key or tenant skew, connection behavior, retries, timeouts, think time, locality, and background work. Separate scheduled load from observed arrivals. In an open model, the scheduler attempts arrivals independently of response time. In a closed model, completion releases the next request; throughput can therefore fall when response time rises without the client reporting a growing scheduling lag.

Generator validation is an explicit result, not a setup footnote. Report scheduler lag, client CPU and event-loop delay, socket or port exhaustion, dropped samples, connection establishment, network headroom, and achieved mix per generator. Compare client counts or placements when one client could be the bottleneck. A test is inconclusive if the generator cannot sustain the requested schedule or record all outcomes.

Coordinated omission occurs when a stalled system also prevents the client from scheduling or recording the requests that would have arrived during the stall. Preserve intended start times or use an equivalent correction whose assumptions are stated. Report corrected and uncorrected distributions when the distinction teaches the failure; do not silently “correct” a closed user model into an open arrival model.

Make state transitions inspectable

Warm-up ends because a declared condition becomes true, not because five convenient minutes elapsed. Conditions may include compiled code stability, populated caches, established connections, storage temperature, completed compaction, stable queue age, and a bounded trend in throughput or resource demand. List which state is intentionally cold or warm.

Define:

1. setup and data preparation;
2. calibration and generator qualification;
3. warm-up rule and maximum warm-up time;
4. measurement window and steady-state rule;
5. treatment order, randomization, blocking, pairing, and reset;
6. independent repetitions and stopping rule;
7. failure injection and recovery observation; and
8. cleanup plus evidence finalization.

Do not delete inconvenient transients. Tag warm-up, steady, overload, failure, and recovery intervals in raw data, then explain which interval answers which claim. If steady state never occurs, that is a result about the tested state, not permission to select the flattest window.

Preserve distributions, uncertainty, and correctness

For every metric state its definition, unit, population, collection point, missing-data behavior, and aggregation. Keep raw observations or lossless histograms with enough range and precision for the decision. Never average host percentiles. Merge compatible histograms or return to observations, then compute the population quantile. Report throughput as offered load, completion rate, and valid goodput so retry amplification and invalid results remain visible.

The statistical block records the experimental unit, independent-run count, estimator, interval method, effect size, practical threshold, treatment of outliers, multiple comparisons, and any model assumptions. Pair runs only when the pair shares a meaningful block; randomize order to reduce drift. A narrow confidence interval does not repair a biased workload.

Every performance result needs a correctness companion. Validate response semantics, authoritative state, duplicates, ordering, freshness, durable effects, and abandoned work as applicable. Report the denominator. “Zero errors” without the number and class of checked operations is not evidence.

A result table should keep observation and decision separate:

Plots retain axes, units, population, sample count, run IDs, and uncertainty. Use aligned axes for comparisons. Show the throughput–latency curve when a single operating point could hide the saturation knee.

Threats to validity are decision boundaries

Organize threats by the inference they weaken:

• measurement: clock error, dropped telemetry, histogram range, sampling, or observer overhead;
• internal validity: order effects, cache state, noisy neighbors, background work, configuration drift, or unequal failures;
• construct validity: metric or synthetic operation does not represent the objective or useful work;
• external validity: data, topology, hardware, traffic, geography, or operating state differs from the target; and
• statistical conclusion: too few independent units, high variance, violated assumptions, selective windows, or multiple comparisons.

For each threat, state likely direction, severity, mitigation, and the decision it prevents. “Lab environment differs from production” is too vague. “The lab has no TLS terminator and therefore cannot support the CPU-capacity claim for the production ingress path” is actionable.

Transfer limits finish the argument. State the largest workload envelope, topology, version family, state, and time horizon justified by the evidence. Name the observation that requires rerun: a payload shift, new region, runtime upgrade, data growth, different durability mode, or changed dependency behavior.

Reproduction is a fresh-run contract

Commands should begin from a documented clean environment and produce a new run ID. Include prerequisite access, dataset preparation, deployment or local setup, calibration, execution, correctness checks, analysis, expected artifact tree, and teardown. Never make reproduction overwrite the report being audited.

The companion fixture verifies the report structure and artifact hashes. That verifier cannot prove workload representativeness, independence, causal attribution, or transfer. Those remain engineering judgments exposed for review.

A bounded Mercury report

The companion example demonstrates a deliberately inconclusive outcome: generator qualification passes, but one treatment repetition loses its correctness ledger during the failure interval. Latency summaries remain available, yet the acceptance gate is inconclusive, because the promised valid-goodput denominator cannot be reconstructed. The correct action is to repair evidence capture and repeat the affected block—not to publish the attractive latency result.

This is a useful counterexample to “more samples fix uncertainty.” More latency samples reduce sampling error around the wrong evidence set; they do not restore missing correctness outcomes.

Applied exercises

Repair the headline. A report says “p99 improved 31%” after changing the cache. Add the minimum fields required to evaluate it: request population, measurement boundary, baseline and treatment versions, scheduled and achieved load, cache/data state, independent runs, quantile method, uncertainty, backend work avoided, correctness denominator, raw path, and transfer limit. If these cannot be recovered, relabel the headline as an unverified observation.

Find the omitted population. A closed-loop client reports stable p99 while throughput falls by half during a 12-second pause. Compare intended arrivals with actual starts. If the production claim concerns open arrivals, repeat with a qualified open scheduler or preserve expected intervals; include the requests displaced by the pause. If real users are modeled as closed sessions, keep that model but report response-time and throughput collapse together.

Choose the next experiment. Three paired runs suggest a 4% CPU reduction, with an interval spanning −2% to 10%; the operational threshold is 8%. Do not declare equivalence or victory. Estimate whether additional independent blocks can distinguish an 8% effect, investigate dominant variance, and ask whether the decision is worth that evidence cost. A cheaper decision may be to retain the baseline.

Field review

Before citing a result, confirm that you can answer:

• What exact claim and smallest decision does this report support?
• What is the useful-work denominator and correctness invariant?
• Where do clocks, counts, state, and responsibility begin and end?
• Can code, configuration, topology, data, tools, and dates be reconstructed?
• Did the generator produce and record the intended workload without becoming the constraint?
• Are warm-up, measurement, failure, and recovery states declared rather than selected afterward?
• Are distributions computed over the right population with uncertainty across independent experimental units?
• Are raw observations, manifests, analysis, logs, and hashes retained?
• Which rival explanation remains plausible?
• What target condition lies outside transfer, and what change forces a rerun?

The decision rule is: use a benchmark result only for the named claim and smallest decision whose population, boundary, versions, workload, state, correctness, statistical method, raw evidence, and transfer limits are all inspectable; otherwise call the result exploratory or inconclusive and run the discriminating experiment.

Primary references and transfer notes

• RFC 8239, Data Center Benchmarking Methodology demonstrates objective–method–report organization, explicit iterations, repeatability reporting, traffic patterns, calibration, and isolated benchmark safety. Its numerical rules are network-device specific; transfer the reporting discipline, not its thresholds, to application benchmarks.
• RFC 6815, Applicability Statement for RFC 2544 explicitly confines laboratory benchmarking methods from service activation and production SLA validation. It supports the distinction between controlled characterization and production evidence.
• HdrHistogram coordinated-omission documentation explains expected-interval correction for recorded latency data. The correction assumes an intended arrival interval and does not decide whether an open or closed workload represents the target.
• OpenTelemetry HTTP semantic conventions provide versioned names and units for HTTP telemetry. Record the convention version because parts of the specification have different stability levels.

Before the next test begins, move from evidence custody to precommitment. Appendix G turns the claim, population, and transfer boundary into a load-test plan with declared phases, safety limits, correctness checks, abort authority, and recovery evidence.