Appendix E — Performance Design Review Template

The packet is polished: a twelve-box architecture diagram, a dashboard screenshot, and a benchmark headline. Yet the review chair cannot answer three basic questions. What decision must be made today? Which invariant survives when the dependency slows? What evidence would reverse the proposed choice?

Stop the meeting. More diagram narration will not make an undecidable packet decidable.

A performance design review is a decision gate, not an architecture presentation and not approval by attendance. Its artifact must expose the workload, temporal and functional promises, critical path, resource demand, state ownership, failure behavior, evidence, alternatives, and ownership needed to choose or refuse a design. The disposition must be conditional enough to revisit when its assumptions change.

Use this appendix when commissioning a consequential design, preparing its review packet, chairing the meeting, recording a decision, or reopening an old decision after workload, topology, SLO, cost, security, or evidence changes. The copyable artifact is in examples/performance-engineering-system-design-handbook/appendices/performance-design-review/performance-design-review-template.md.

After using the template, you should be able to:

• bound one review around a named decision and useful-work population;
• connect functional invariants to SLOs, queues, failure semantics, and migration;
• distinguish assumed, modeled, simulated, observed, and inferred evidence;
• stop a review when missing evidence makes alternatives indistinguishable;
• issue an accept, conditional accept, reject, or new-evidence disposition with owners; and
• define the observation or threshold that reopens the decision.

Appendices A–D provide units, statistical discipline, capacity laws, and provisional calibration. Chapters 1–8 establish the performance contract and design loop. This template consumes those foundations; it does not repeat their derivations.

Commission one decision, not one system

“Review the new checkout architecture” is too broad. It encourages a tour of components and lets unresolved choices hide inside the tour. Write the decision as a verb, boundary, objective, and operating envelope:

Decide whether optional recommendation enrichment remains synchronous inside Mercury checkout’s 300 ms p99 boundary at 4,100 request/s peak, including dependency slowdown and one-zone capacity loss.

That sentence tells the chair what can be decided. It excludes payment-write semantics and recommendation quality except where they constrain the choice. Another review can own those decisions.

Every packet starts with six controls:

An exclusion is not permission to ignore an interaction. If payment commit is outside the read-path decision but checkout retries can duplicate payment, the invariant still crosses the boundary and must be recorded. Scope controls depth; it does not erase correctness.

Roles and authority

A useful review separates advocacy, evidence, risk ownership, and disposition.

• The decision owner accepts the result and residual risk. The owner can fund or stop the work.
• The design owner prepares the model, alternatives, and migration plan. This person may advocate but may not declare evidence sufficient alone.
• The performance-methods reviewer challenges workload, measurement, statistics, benchmark transfer, and causal claims.
• The reliability reviewer challenges overload, partial failure, feedback, recovery, and operational controls.
• The security/privacy reviewer challenges trust, abuse, sensitive evidence, retention, and unsafe shortcuts.
• Domain owners validate dependencies, data semantics, runtimes, storage, networks, or products inside the boundary.
• The chair protects the decision question, time, stop conditions, and disposition record.
• The recorder captures changed assumptions, open evidence, owners, dates, and dissent without rewriting the meeting as unanimous.

One person can fill several roles in a small team, but conflicts remain visible. A design owner who is also decision owner still needs independent challenge for consequential claims. Attendance from a dependency team is not acceptance of that team’s operational obligation; name the accepting owner.

Entry gate: is the packet reviewable?

Perform the entry check before scheduling the decision meeting. A packet is reviewable when:

1. the decision and deadline are explicit;
2. the useful-work boundary and functional invariants are named;
3. the workload envelope includes mix, skew, bursts, growth, failure, and recovery demand;
4. the current and proposed critical paths are traceable;
5. at least two viable alternatives include a conservative baseline;
6. consequential numerical claims have scoped evidence or are labeled assumptions;
7. overload, rollback, and evidence gaps have owners; and
8. reviewers can inspect the model or raw evidence before the meeting.

Missing detail does not always block entry. A review may exist to choose the experiment that reduces uncertainty. The packet must then say that the requested decision is an experiment or prototype, not production adoption. What blocks entry is ambiguity about which uncertainty the work will resolve.

Use this evidence sufficiency ladder:

Evidence need not eliminate uncertainty. It must distinguish the alternatives enough for the requested commitment.

The review record, section by section

The artifact uses fifteen sections. They are not an agenda that must receive equal meeting time. Complete them before the review, then spend meeting time on the disputed causal link, trade-off, evidence gap, or risk transfer.

Functional scope and invariants

List user journeys and operations, their required effects, invariants, failure semantics, and owners. “Checkout succeeds” is not an invariant. “Recommendation enrichment cannot change authoritative price, currency, cart version, or authorization scope” is inspectable.

For each effect, ask:

• What must be true even after timeout, cancellation, retry, duplicate delivery, partial result, or rollback?
• Which response can be stale, omitted, approximated, or delayed?
• Where is externally visible completion, and can abandoned work continue past it?
• Who owns reconciliation when observed state diverges?

Performance choices frequently change correctness boundaries. Moving work asynchronous may reduce request latency while introducing freshness, ordering, and repair obligations. The review cannot record the latency benefit without the new invariant.

Workload envelope

Describe demand before capacity. Segment by operation or cost class; record open, closed, or mixed arrivals; nominal and peak rates; burst duration; payload or compute distribution; skew and locality; concurrency; geography; background work; retries; growth; and failure/recovery load.

Use ranges where inputs are uncertain. A single forecast point makes headroom look precise. State the horizon and the observation that updates the forecast. If one tenant can supply 20% of work or one key can receive 5% of requests, fleet average is not a partition model.

The template asks separately for repair, migration, and retry demand because this work often appears only after the system has lost capacity. A design that serves peak traffic with all replicas healthy may still fail its recovery objective.

SLOs and budgets

Every service-level objective (SLO) needs an indicator population, measurement boundary, statistic, target, and window. Define success and failure inclusion. Allocate end-to-end latency, resource, freshness, error, and cost budgets without pretending the allocations are independent.

Record behavior after deadline expiry. Does work cancel, complete for correctness, continue for cache value, or become harmful abandoned load? Record reserve rather than allocating 100% of the objective to expected component times. A budget is an architecture constraint and an instrumentation contract, not a decorative table.

Architecture and critical path

Trace the normal path in order. Mark serial and parallel work, optional branches, synchronous fan-out, queues, locks, pools, schedulers, copies, state access, and failure-domain boundaries. Attach latency or service-demand budgets only where provenance is known.

Then trace a slow path and at least one failed or recovering path. A diagram that shows only steady-state components cannot support an overload or recovery claim. The critical path artifact may be a diagram, trace, or annotated sequence, but it must reveal where work waits and which state makes completion valid.

State ownership and data lifecycle

For each state set, name the authority, replicas, caches, derived forms, freshness rule, retention/deletion behavior, and repair mechanism. Identify commit and visibility points. Record duplicate, idempotency, ordering, and invalidation semantics.

This section stops latency optimization from moving state obligations out of frame. A materialized view may make a read fast while adding write amplification, repair bandwidth, staleness, and deletion lag. Those are properties of the proposed design, not later implementation details.

Partitioning, replication, and consistency

State the partition key, expected skew, hot-partition policy, rebalance mechanism, replica topology, acknowledgment path, read/write consistency by operation, and behavior during replica, zone, or region loss. Quantify repair traffic and its interference with foreground work.

Do not accept “eventual consistency” as a complete guarantee. Record which effect may lag, by what observable bound or convergence condition, and what reads or writes can violate a user expectation during failure and repair.

Queueing, admission, overload, and recovery

Every waiting point needs an owner, capacity or concurrency bound, age/deadline policy, admission action, and telemetry. Record whether overload rejects, degrades, expires, spills, persists, blocks upstream, or steals capacity from another class.

For retries and hedges, calculate attempt amplification and retained work after callers stop waiting. For backlog recovery, use net useful drain after new arrivals and recovery-specific service demand. Reserve foreground capacity. Define a stop condition when replay, rebuild, or retry makes goodput worse.

A credible row might read:

The trigger, action, protected result, and exit condition make the policy operable.

Capacity and cost model

Normalize resource demand to useful completions, not attempts. Record CPU, memory, storage, network, connections, accelerators, licenses, and external quotas as applicable. Model nominal, peak, failover, deployment, migration, and recovery states. Identify the current bottleneck and the likely next bottleneck after the proposed change.

Use Appendix C’s service-demand and bottleneck laws with their conditions. Carry Appendix D ranges until sensitivity crosses a decision boundary, then replace them with representative observations. Include lead time and procurement or scaling triggers. Report cost per successful objective-compliant outcome so retries, errors, and late results do not look efficient.

Security and privacy

Mark trust and tenant boundaries. Include authentication, authorization, encryption, key access, audit, rate limits, and sensitive-data handling in performance paths. Identify adversarial workload shapes and abuse limits.

Performance evidence itself can be sensitive: traces may contain identifiers, profiles may reveal code or data shapes, and production replays may copy regulated data. Record collection, minimization, access, retention, locality, and deletion. A design fails this section if its fastest path bypasses a required control or if its experiment creates an unmanaged data copy.

Observability and validation

Turn each consequential claim into a decision consequence, evidence type, population, method, acceptance rule, owner, and raw-evidence path. State generator validation, warm-up, steady state, correctness, failure inclusion, missing data, aggregation, independent runs, uncertainty, and transfer limits.

An acceptance rule must be testable before results arrive. “Latency improves” invites selective interpretation. “For authorized checkout reads at 4,100 scheduled request/s, base-response p99 remains at or below 300 ms, valid goodput remains at least 99.95%, and cancellation releases dependency work within 5 ms p99” specifies the decision population and correctness companion.

Include load, stress, soak, failure, recovery, and migration scenarios only where they challenge a material claim. A long checklist of unowned tests is not a validation plan.

Migration and rollback

Record compatibility windows, mixed-version behavior, expand/migrate/validate/cutover/clean-up stages, background-work capacity, authority transitions, divergence detection, rollback-safe points, and points of no return. Name abort signals and the person authorized to act.

Rollback is a performance mode. It may double writes, invalidate caches, replay events, or compete with foreground demand. “Feature flag available” is insufficient if old and new state cannot be reconciled after the flag changes.

Assumptions, alternatives, risks, and revisit triggers

Compare mechanisms or boundaries, not only products. Always include the conservative baseline: keep the current design, remove the optional feature, reduce the operating envelope, or buy temporary capacity. It prevents the proposed complex design from competing only with obviously weak variants.

For each assumption or risk, record consequence if wrong, decisive observation, owner, and date or trigger. Useful revisit triggers include traffic or skew thresholds, a new geography, an SLO change, a cost boundary, an operational burden, a security requirement, or evidence that invalidates the model.

A risk with no owner is a meeting note. A trigger with no observation path cannot reopen the decision. A residual risk with no accepting authority is not accepted.

Worked review slice: optional Mercury enrichment

The companion example asks whether recommendation enrichment remains synchronous within Mercury checkout. It is intentionally a bounded slice rather than a complete system record.

The workload is an open, bursty checkout class at 2,400 request/s nominal and 4,100 request/s peak; the top tenant contributes 12%, payloads contain 4–48 items, and one-zone loss leaves 72% of nominal capacity. Authorized checkout reads have a 300 ms p99 objective over a five-minute regional population. Recommendations are optional and cannot alter authoritative price or authorization.

Three alternatives remain credible:

1. remove synchronous enrichment and preserve the simplest base path;
2. keep enrichment synchronous but bound it with a 35 ms deadline, concurrency limit, and queue-age degradation; or
3. fetch enrichment asynchronously after the base checkout response.

The proposed bounded path is not accepted because it looks balanced. It is conditionally accepted only if an open-loop replay shows base p99 at or below 300 ms with at least 99.95% valid responses under doubled dependency latency and one-zone capacity loss, and a trace proves cancelled work releases its worker and connection within 5 ms p99. Queue age above 20 ms causes new enrichment to be omitted; age below 8 ms for ten minutes exits degradation.

This slice exposes the mechanism: dependency slowdown retains scarce concurrency, waiting consumes the enrichment deadline, and uncancelled work can survive caller abandonment. The policy protects base goodput by shedding optional work before the dependency pool becomes the checkout bottleneck.

The decision reverses if the base objective fails, cancellation does not release work, peak exceeds 4,100 request/s, tenant skew exceeds 20%, or the measured business value of enrichment changes materially. The design has not become “approved forever.” It has earned a bounded operating envelope and an evidence-backed revisit path.

Run the meeting around disputes and gates

A 60-minute review can use this sequence:

1. Five minutes — decision and disposition sought. The owner reads the boundary, modes, and exclusions.
2. Ten minutes — invariants and workload challenge. Reviewers correct populations, skew, failure demand, and success semantics.
3. Ten minutes — critical-path and state trace. Follow one normal and one adverse request or unit of work.
4. Fifteen minutes — alternatives and causal evidence. Test the decisive claims and transfer limits.
5. Ten minutes — overload, recovery, security, and migration. Identify transferred risk and operational control.
6. Ten minutes — disposition. Record conditions, owners, dates, residual-risk acceptance, reversal evidence, and triggers.

Do not spend meeting time reading fields aloud. Reviewers should comment before the meeting. The chair should maintain a decision log: changed assumption, evidence accepted or rejected, alternative reopened, and risk owner. When a dispute is factual and a discriminating test exists, assign the test instead of deciding by seniority.

Stop conditions

Stop and request new evidence when any of these holds:

• the useful-work or correctness boundary changes during the meeting;
• the workload population cannot be reconciled with the model or benchmark;
• alternatives differ on a consequential claim that no evidence distinguishes;
• failure or migration changes state authority without a defined reconciliation path;
• capacity arithmetic omits the likely constraint or failure/recovery demand;
• a required security or privacy control is outside the measured path;
• the proposed condition has no owner, deadline, acceptance rule, or rollback; or
• the residual risk exceeds the decision owner’s authority.

Stopping is not rejection. It preserves the integrity of the decision. Record precisely what evidence or authority resumes the review.

Dispositions are conditional engineering records

Use four outcomes:

Avoid “approved with concerns.” It hides whether work may proceed and who owns the concern. Avoid voting counts when the issue is a correctness invariant or evidence gap. Record dissent when a reviewer believes the accepted residual risk is understated; do not smooth it into consensus language.

Acceptance at one gate does not imply permission for every rollout stage. A prototype decision may authorize a bounded experiment. A canary decision may authorize 1% traffic under a rollback guard. Expansion requires the production evidence and operating state named in the prior record.

Review failure patterns

Applied exercises with answer guidance

Field — entry-gate review. A packet proposes a new cache. It includes a 40% microbenchmark speedup and a topology diagram. It lacks request mix, authoritative-state ownership, invalidation behavior, backend work avoided, failure capacity, and rollout evidence. Do not schedule an adoption decision. A valid next decision could authorize a prototype if the owner supplies the target population, cache key and consistency contract, hit/miss service demand, stampede behavior, correctness checks, and an experiment that measures end-to-end goodput and tails. The repair is not “more cache detail”; it is a reviewable decision boundary.

Principal — red-team overload. A service runs at 55% fleet-average CPU during peak and therefore claims ample failover headroom. One zone loss removes 35% of capacity, the hottest tenant supplies 18% of demand, retries allow three attempts, and recovery rebuild consumes 20% of remaining storage bandwidth. Challenge the average. Require per-zone and per-partition service demand, useful goodput versus attempts, failover placement, retry suppression, repair reservation, queue age, and a model for the combined state. Multiple designs can pass: reserve more capacity, reduce the failover envelope, shed optional work, isolate repair, or change ownership. Each must state the protected invariant and evidence.

Principal — facilitate disagreement. The security reviewer requires payload encryption in a trace-replay dataset; the performance reviewer argues encryption changes CPU demand. The chair should not choose one concern. Preserve the privacy invariant, measure the replay pipeline with the required control, and separately determine whether representative production encryption is already inside the application path. If synthetic or transformed data can preserve size and skew without sensitive content, record its transfer limit. The decision resumes when the experiment represents both the required control and the workload property under dispute.

Field checklist for the chair

Before issuing a disposition, ask:

• Can every reviewer state the same decision, boundary, useful-work unit, and operating modes?
• Are functional invariants and temporal objectives both explicit?
• Does the workload include mix, skew, bursts, growth, failure, recovery, and background work?
• Can we trace normal, slow, overloaded, failed, recovering, and migrating behavior where material?
• Are state authority, visibility, duplicates, freshness, repair, retention, and deletion owned?
• Are queues, concurrency, retries, admission, degradation, and stop conditions bounded?
• Does capacity normalize demand to useful completion and include the likely constraint?
• Are security, privacy, and adversarial workloads inside the design and evidence boundaries?
• Does each consequential claim have an evidence type, population, method, acceptance rule, raw record, and transfer limit?
• Does at least one conservative alternative remain honestly represented?
• Can rollback preserve invariants through mixed versions and state transitions?
• Does every condition and residual risk have an owner with sufficient authority?
• What observation would reverse the decision, and who watches it?
• Which workload, SLO, topology, cost, operational, or policy threshold reopens review?

The decision rule is: accept only the smallest commitment supported by evidence inside the named operating envelope; condition it when remaining proof has a clear gate; request evidence when alternatives are still indistinguishable; reject when an option cannot preserve an invariant or objective without moving unowned risk outside the boundary.

When the disposition requests new evidence, turn the unresolved claim and gate into a benchmark or experiment report. Its population, boundary, versions, workload, correctness checks, uncertainty, and transfer limits should be inspectable before the review reconvenes.

Artifact and verification scope

The full Markdown template and the Mercury slice live in examples/performance-engineering-system-design-handbook/appendices/performance-design-review/. verify-template.mjs checks that the artifact contains all fifteen required sections, boundary and useful-work controls, failure semantics, reversal evidence, revisit triggers, and the example’s key workload, budget, evidence, risk, and disposition markers.

That check proves structural coverage only. It cannot determine whether a workload is representative, an invariant is complete, a model is dimensionally correct, a benchmark transfers, a security control is sufficient, an owner has authority, or a residual risk is acceptable. Those are the independent review’s intellectual work.