Performance Engineering and System Design Handbook
Appendix G — Load-Test Plan Template
Design a safe, decision-bearing load test that validates capacity, degradation, correctness, and recovery before traffic begins.
The load generator is armed for ten times normal traffic. The team has a target request rate but no owner authorized to abort, no invariant check for duplicated writes, and no definition of recovery. Starting the test would create load, not evidence.
A load-test plan is a preflight contract. Before traffic begins, it states the decision, representative demand, state preparation, phase transitions, failure schedule, safety envelope, telemetry, correctness assertions, recovery exit, evidence custody, and authority to stop. The plan separates what the test intends to learn from whatever the system happens to survive.
Use the copyable artifact at examples/performance-engineering-system-design-handbook/appendices/load-test-plan/load-test-plan-template.md. Its abbreviated Mercury plan targets a bounded traffic-event decision. The structural verifier checks the required controls.
Appendix F defines the report this plan must produce. Chapters 2–8 provide workload, objective, queue, bottleneck, and design-loop foundations. Chapter 52 develops the test taxonomy. This appendix does not authorize testing production, prescribe a load tool, or replace change, security, privacy, incident, and service-owner controls.
Decide what the test may authorize
“See how much traffic it handles” is not an objective. Name the smallest decision and the evidence required for it:
Decide whether Mercury can admit a forecast traffic event capped at 4,100 scheduled checkout requests/s while one zone is unavailable, preserving base-response p99 at 300 ms, at least 99.95% valid goodput, bounded optional-enrichment queues, and stable recovery within 15 minutes.
Write acceptance criteria before choosing phases. Each criterion needs a population, boundary, statistic, threshold, window, and state. Pair performance with correctness and recovery:
| Claim | Acceptance criterion | Evidence | Owner |
|---|---|---|---|
| temporal | base checkout p99 ≤ 300 ms during peak hold and zone loss | client histogram plus trace sample | performance lead |
| useful work | valid goodput ≥ 99.95% of admitted authorized checkouts | independent correctness ledger | service owner |
| overload | enrichment queue oldest age ≤ 20 ms; new optional work sheds above trigger | pool metrics and decision log | runtime owner |
| recovery | queue age, error rate, capacity, and latency remain inside recovery bands for 10 min within 15 min | aligned timeline | incident commander |
Define fail, abort, and inconclusive separately. A missed acceptance threshold can be a valid failed result. An abort protects people or systems. Inconclusive means the method or evidence cannot answer the claim—for example, the generator missed its schedule or correctness records were lost.
Model clients, not just requests
Describe units of work, open/closed/mixed arrivals, operation and dependency mix, payload/data distributions, tenant/key skew, geography, locality, connection reuse, session think time, retries, timeouts, cancellation, and growth. Include nominal, peak, burst, failure, and recovery demand. Record scheduled rate separately from admission, attempts, completions, and valid goodput.
A client is part of the experiment. Specify:
- scheduler behavior and acceptable scheduling lag;
- connection pools, protocol versions, authentication, DNS, TLS, and session reuse;
- timeout, retry, hedge, backoff, jitter, and cancellation rules;
- source-address, port, CPU, memory, network, and telemetry capacity;
- placement relative to the system and failure domains;
- deterministic seeds where useful and the allowed run-to-run variation; and
- calibration steps that prove generators can exceed the test envelope without saturation.
Open arrivals are useful when demand should continue independently of response time. Closed clients are useful for interactive sessions whose users wait. Neither is universally correct. State the production behavior represented and the transfer limit. A closed client that slows with the service must not support an open-arrival capacity claim.
Prepare data and state deliberately
Record dataset schema, cardinality, size, age, distributions, skew, compression, locality, account/tenant mix, and generator revision. Identify authoritative, replicated, cached, derived, and ephemeral state. Define privacy classification, transformation, access, retention, deletion, and prohibited production data.
Preparation includes more than loading rows:
- establish baseline topology and versions;
- create or restore the named dataset snapshot;
- verify invariants and referential integrity;
- populate or deliberately empty caches and indexes;
- settle or intentionally retain compaction, replication, repair, and background jobs;
- set quotas, admission, autoscaling, alerts, and feature flags;
- capture pre-test capacity and health; and
- prove cleanup, rollback, and test-data deletion.
State “warm” only with an observable rule. If the event will begin after a quiet night, a fully warmed dataset may overstate readiness. If the test starts cold, explain whether cold behavior belongs to the claim.
Build phases around hypotheses
A useful phase table is a state machine with entry, load, perturbation, observation, exit, and abort rules:
| Phase | Purpose | Offered behavior | Entry and exit | Required observations |
|---|---|---|---|---|
| qualification | prove environment and generators | low load plus calibration burst | healthy baseline; generator headroom passes | schedule lag, telemetry, correctness |
| warm-up | reach declared state | representative mix below knee | warm-state conditions or max-time failure | cache/index/runtime/background state |
| baseline hold | establish comparison | nominal load | fixed duration and stable bands | full distributions and resource demand |
| ramp | locate nonlinear change | stepped or shaped increase | each hold completes or guard trips | offered/admitted/goodput, queues, saturation |
| peak hold | test event claim | peak mix, skew, and bursts | declared duration | acceptance populations |
| adverse hold | test lost capacity or slow dependency | peak plus injected failure | failure confirmed; safety remains | degradation, amplification, correctness |
| restoration | remove perturbation | peak or reduced load as planned | dependency/zone restored | retained work and control response |
| recovery | prove stable service | new arrivals plus backlog drain | stable exit bands or timeout | net drain, SLO, repairs, next constraint |
| cooldown | protect and preserve | controlled reduction to zero | cleanup and evidence custody complete | late effects, leaked work, final hashes |
Do not ramp continuously when the decision needs a steady operating point. Do not hold merely for a round duration; relate duration to cache cycles, autoscaling delay, compaction, leak horizon, or another mechanism. For a soak test, name the drift or leak that the duration can reveal.
Inject failures with a recovery claim
Failure scenarios must identify mechanism, scope, timing, confirmation signal, expected control response, protected invariant, restoration action, and recovery criterion. Possible scenarios include dependency latency or errors, packet loss, node/zone loss, leader movement, storage pressure, quota exhaustion, expired credentials, and telemetry degradation. Select only failures that challenge the decision.
Inject one discriminating perturbation at a time unless the compound state is itself the claim. A zone loss plus peak load may be required for capacity. Adding packet loss, deploy churn, and a cache flush simultaneously makes causal diagnosis harder unless that combination represents the operating envelope.
Recovery is not “the graph came down.” Measure:
- time from restoration to bounded admission and valid goodput;
- oldest queue age and net backlog drain after new arrivals;
- retries, hedges, duplicates, abandoned work, and late effects;
- repair, rebalance, cache refill, compilation, and autoscaling demand;
- latency and error distributions by population;
- state convergence and correctness reconciliation; and
- a stability dwell that prevents a brief crossing from counting as recovery.
If backlog is B useful units, arrival rate is λ, and recovery completion capacity is X_r, a first drain estimate is T = B/(X_r - λ), valid only when X_r > λ and service demand remains comparable. The plan must stop recovery work that reduces foreground goodput enough to violate the protected objective.
Put safety above test completion
State whether the environment is isolated, pre-production, or production-like. A production test requires explicit service-owner and operational authority; this template does not grant it. Inventory blast radius across users, tenants, dependencies, data, quotas, cost, on-call load, and shared infrastructure.
Every hazard has a measurable limit, signal, action, and authorized owner:
| Hazard | Abort signal | Automatic action | Human authority |
|---|---|---|---|
| user impact | protected-class SLO/error burn crosses approved test budget | stop new test arrivals; preserve existing safety traffic | incident commander |
| correctness | duplicate or invalid authoritative effect > 0 where invariant forbids it | stop writes and failure injection | data/service owner |
| queue growth | oldest age or backlog crosses recoverable bound | shed optional/test work; enter cooldown | service owner |
| dependency harm | downstream rate, saturation, or quota exceeds agreement | cap client and remove perturbation | dependency owner |
| evidence loss | generator lag, missing outcomes, or telemetry gap exceeds bound | mark phase inconclusive; stop escalation | test lead |
| spend or resource | approved cost/capacity ceiling reached | stop ramp and autoscaling expansion | budget owner |
Pre-stage kill switches, credentials, communication channel, incident handoff, rollback, cleanup, and escalation contacts. Test the abort path below meaningful load. Never rely on the load generator’s dashboard as the only stop signal; it may fail with the experiment.
Observe mechanism, outcome, and test validity
The telemetry checklist has three lanes.
Outcome: offered, admitted, completed, valid-goodput, error, latency, freshness, and business/invariant populations. Mechanism: queue age/depth, concurrency, pool occupancy, CPU, throttling, memory, pauses, storage, network, locks, dependencies, retries, autoscaling, replication, repair, and cost. Method validity: intended versus actual starts, generator resources, missing samples, time synchronization, configuration changes, failure confirmation, and observer health.
Preflight dashboards, queries, trace sampling, log retention, histogram range, cardinality limits, and timestamps. Record semantic-convention and collector versions. A dashboard screenshot is navigation, not raw evidence. Retain machine-readable observations and the queries or analysis that produced conclusions.
Use an independent correctness oracle where feasible. Assertions may cover response schema and authorization, state-machine invariants, uniqueness, idempotency, ordering, balances, freshness, durable acknowledgment, reconciliation, and absence of late effects after cancellation. Sampled correctness needs a justified population and rate; a forbidden duplicate often requires complete accounting.
Assign authority and evidence custody
Name the decision owner, test lead, generator owner, service owner, dependency owners, data/correctness owner, observability owner, safety or incident commander, and report author. One person may fill several roles, but no critical condition should be ownerless.
The run log records planned and actual phase times, configuration changes, failure injection/restoration, alerts, abort decisions, anomalies, and excluded intervals without erasing them. The report owner packages the plan revision, version manifest, workload config, dataset manifest, raw outputs, correctness ledger, event log, analysis, and hashes using Appendix F. The decision owner issues pass, fail, inconclusive, or follow-up; the test operator does not promote a design by declaring the run “successful.”
Mercury pre-launch plan
The companion plan tests a forecast checkout event in an isolated production-like region. It uses scheduled open arrivals, a representative item-count distribution, 12% top-tenant skew, optional enrichment, and a one-zone-loss phase. The plan protects authoritative checkout semantics and base goodput while allowing recommendation omission.
Its counterexample is a throughput-only plan that ramps to collapse. That plan finds a number but cannot authorize the event: it does not represent the forecast mix, bound dependency harm, verify valid effects, or show recovery. The repaired plan may stop below collapse if queue age reaches the recoverable bound. This is not lost ambition; it preserves the decision’s safety and evidence.
Applied exercises
Prepare a tenfold event. Start with a 10× forecast and decompose it into operation mix, burst shape, tenant/key skew, payloads, geography, client behavior, and background work. Define nominal, peak, lost-capacity, and recovery modes. Add acceptance criteria for valid goodput, tails, queues, correctness, and recovery. If the environment cannot safely represent 10×, test components of the claim and record the remaining model rather than relabeling 4× as 10× evidence.
Repair throughput-only testing. A plan measures maximum completed requests/s for ten minutes. Add scheduled versus achieved arrivals, latency distributions, useful-work validity, resource and queue signals, a ramp with holds, a failure at a decision-relevant point, recovery exit bands, generator qualification, raw retention, and an abort owner. The maximum may still be reported, but it is no longer the only result.
Handle a failed abort drill. During preflight, the kill command stops one client but two continue for 40 seconds. Do not begin the main run. Bound retained load, repair fan-out control, repeat the abort drill, and require independent server-side admission. The drill has already produced the most valuable finding: the planned safety boundary was fictional.
Preflight card
Do not start until:
- the decision, claims, exclusions, acceptance, abort, and inconclusive rules are signed by their owners;
- workload and client behavior represent the named target and generators have headroom;
- data, state, topology, versions, privacy controls, and cleanup are manifest;
- each phase has entry, exit, observation, and maximum duration;
- failure confirmation, restoration, recovery, and stable-exit criteria are rehearsed;
- safety signals, automatic actions, kill paths, escalation, and incident handoff work;
- outcome, mechanism, method-validity, and correctness evidence are queryable;
- raw retention, clocks, run IDs, event logging, artifact paths, and hashes are ready; and
- the report and decision owners accept custody after the run.
The decision rule is: start a load test only when its representative workload, phase transitions, failure and recovery claims, correctness oracle, safety bounds, abort authority, telemetry, and evidence custody are explicit; stop escalation when safety or method validity fails, and authorize only the smallest decision supported by the resulting Appendix F report.
Primary references and transfer notes
- RFC 8239 requires controlled benchmark setup, calibration, repeatability, reporting, and isolation from production forwarding. Its device-specific traffic rules do not define an application load profile.
- RFC 6815 warns that laboratory benchmark methods are not service activation or production SLA validation. Use it to keep characterization and operational proof distinct.
- RFC 9411 provides current examples of initialization, qualification, load profiles, measurements, and validation criteria for network-security devices. Transfer the explicit procedure structure, not appliance-specific metrics.
- OpenTelemetry HTTP metrics conventions define versioned HTTP metric names, attributes, and units and warn about opt-in cardinality. A load plan must still define its own useful-work and correctness populations.
A plan governs a controlled run; production can still leave its tested envelope or combine mechanisms the plan isolated. Appendix H is the command instrument for that moment: protect valuable correct work, break confirmed feedback, preserve expiring evidence, and prove recovery rather than a temporary graph crossing.