Performance Design Reviews and Decision Records

You are asked to review this proposal:

Use the faster distributed cache. It benchmarks well, the team already knows it, and the principal engineer used it at their last company.

There is no workload distribution, service objective, consistency boundary, cell-loss result, cost model, privacy review, migration plan, or rollback condition. The proposal may be right. It is not yet decidable.

A performance design review does not transfer design authority to a committee. It makes the author’s important claims inspectable before they become expensive production facts. The reviewer’s job is to identify the few assumptions that control the outcome, demand proportionate evidence, ensure risks and tests have owners, and preserve why one option was chosen over another.

The decision rule is: approve a design when its important assumptions are visible, risks have owners, limits have tests, and alternatives were rejected for explicit reasons.

Decide whether a review is warranted

Review cost should follow consequence and reversibility. Require a focused performance review when a change does one or more of the following:

• creates or materially changes an SLO, externally visible limit, or capacity commitment;
• adds a synchronous dependency, coordination boundary, queue, cache, replica, partition, or cross-region path;
• changes authoritative state, consistency, isolation, security, privacy, or recovery semantics;
• commits substantial spend or a hard-to-reverse vendor, protocol, schema, or data-placement choice;
• raises expected utilization, removes failure reserve, changes admission or degradation, or enlarges blast radius;
• introduces a new workload class, untrusted work, high skew, deadline-sensitive fan-out, or paid side effect;
• migrates live state, dual-writes, backfills, or requires rollback through incompatible versions; or
• makes a consequential performance claim that has not transferred from the test environment.

A local refactor inside an established budget may need only code review and regression tests. A time-boxed experiment may need an experiment plan rather than an architecture decision. Document why a review was waived when the boundary is ambiguous. The waiver should expire if demand, state, reversibility, or exposure changes.

Review early enough to change the design. A launch-readiness meeting held after contracts, capacity, and migration are fixed is evidence theater.

The minimum review packet

The author owns one coherent packet. Links are useful only if the reviewer can trace the claim across them.

Workload and objectives

Name the unit of demand and the valuable outcome. Include arrival rate and concurrency, distributions, mix, object sizes, skew, locality, deadlines, retries, seasonality, growth, and adversarial work. State SLO population, percentile or other statistic, window, exclusions, correctness and freshness rules, and error-budget policy.

Critical path, budgets, and state

Draw normal, slow, failure, and recovery paths. Label synchronous edges, queues, fan-out, serialization, resource pools, authentication and authorization, and external quotas. Allocate end-to-end latency and capacity budgets without pretending every component percentile adds. Identify authoritative, derived, cached, replicated, and in-flight state; include ownership, versioning, retention, and rebuild.

Failure, overload, and recovery

Show the first saturated resource for normal, skewed, failed, and recovering states. Provide admission, queue, retry, load-shedding, degradation, isolation, failover, and backlog-drain behavior. State which invariants survive and who may override automation. “Autoscaling enabled” is not capacity evidence.

Cost, security, privacy, and operability

Price the full outcome, including reserve, data movement, labor, recovery, and transferred cost. Trace security controls and adversarial work through the path. Define telemetry access and retention. Name dashboards, alerts, runbooks, control-plane dependencies, on-call ownership, and audit requirements.

Validation, migration, and rollback

Every important limit needs a test or observation plan: benchmark, trace replay, load/failure/recovery test, simulation, canary, production comparison, or explicit uncertainty. Include data conversion, compatibility, dual-run or shadow behavior, rollout steps, abort thresholds, rollback feasibility, and recovery from a partially completed rollback.

Required input is not the same as required certainty. Early design estimates can be rough if labeled and if the decision is reversible. A hard-to-reverse state boundary with a high-impact unknown requires stronger evidence or a staged commitment.

Use roles to accelerate judgment

Assign roles before the review:

Invite a specialist only for a decision they can materially test. Circulate the packet asynchronously, collect blocking questions, and use the meeting to decide unresolved trade-offs. Ten people discovering missing workload data together is not collaboration.

Consensus is useful but not required. Separate factual disputes, risk acceptance, and preference. Facts need evidence. Risk acceptance needs named authority. If multiple options genuinely satisfy the constraints, the decision owner may choose among them and record why. A reviewer should not block a viable design because they prefer a familiar product.

Maintain an assumption register

An architecture proposal is a portfolio of assumptions. Make each one falsifiable:

ID / claim / scope / evidence type and source / confidence
impact if false / owner / test and due date / reversal trigger

“Traffic will grow” is not useful. “Regional peak customer reads will remain below 8,000 requests/s through 2027 Q2, with the current 65:35 endpoint mix and no tenant above 9%” can be compared with a forecast and telemetry.

Rank uncertainty by exposure rather than embarrassment. A simple teaching priority is:

Exposure_i = (1 − Confidence_i) × Impact_i

This is not expected financial loss unless its scales support that interpretation. It orders investigation. In Mercury’s packet, the assumption that one cache cell can fail without exceeding origin reserve has confidence 0.4 and impact 5, producing exposure 3.0—the first test target. A high-confidence assumption can still need a monitor if its impact is catastrophic.

Assumption-to-evidence matrix

An empty evidence cell is not automatically a rejection. The question is whether the unknown is acceptable at this commitment stage and whether a bounded next step can resolve it.

Compare alternatives before selecting a product

Frame alternatives as mechanisms that meet the same decision criteria:

1. keep the current cache and partition it into tenant cells;
2. adopt the proposed distributed cache with explicit origin protection;
3. cache only immutable objects at the edge and serve mutable records from regional stores; or
4. remove the cache by reducing origin service demand and adding capacity.

For each, compare correctness, latency distribution, throughput, state and invalidation, failure/recovery, security/privacy, operability, cost, migration, and reversibility. Include “do nothing for now” when delay is a real option. Reject it with the cost of waiting, not with rhetoric.

A rejection rationale should be strong enough for a future engineer to avoid reopening the same debate without new evidence:

Alternative: tenant-cell current cache
Reason rejected: meets isolation and rollback needs, but modeled cell loss
requires 9,100 origin rps against 7,600 verified safe capacity during the
2027 peak mix. Adding the required origin reserve costs more than Option 2.
Revisit if origin safe capacity exceeds 9,500 rps or cell working set falls 25%.

“Too slow,” “doesn’t scale,” and “team dislikes it” are not rationales. Neither is a benchmark that tests a different object size, hit rate, consistency mode, or hardware boundary.

Score the evidence without outsourcing judgment

Mercury uses a 0–3 evidence scale across eight dimensions. Weights sum to 100. Zero means absent; one means asserted or weak; two means modeled with bounded evidence and a validation owner; three means reproduced at the relevant boundary, including adverse state where required.

The proposal scores 58/100. More importantly, failure/overload, validation, and migration/rollback score below Mercury’s required floor of 2. It is not approved. Raising the weighted score with excellent documentation cannot compensate for an unowned rollback or an untested overload boundary.

The threshold is local policy, not a universal scientific constant. The scorecard makes missing evidence visible and keeps review depth proportional. It cannot price every interaction or accept residual risk. Record the accountable judgment beside the score.

Capacity and overload evidence must be executable

Capacity claims need a workload version, environment, resource boundary, failure state, and uncertainty. Ask:

• What resource saturates first at peak mix and at worst legal skew?
• What is safe capacity after losing the declared failure domain?
• Does admission occur before the constrained queue?
• Are retries charged as additional attempts and work?
• Which quality rung or work class is shed, and which invariant remains?
• Can failover capacity absorb traffic and cache warm-up simultaneously?
• At what replay rate does backlog drain without violating foreground SLOs?
• How does an operator distinguish saturated, degraded, recovering, and unsafe states?

A smooth throughput chart from a five-minute normal-load test does not answer these questions. Attach the load profile, correctness rules, raw result, analyzer, and failure/recovery timeline. If production is the only valid environment, stage exposure with a canary, explicit aborts, and a reversible capacity step.

Red-team the system, then run a pre-mortem

Red-team prompts challenge an intelligent adversary or extreme input:

• Which legal input maximizes CPU, allocation, fan-out, state, paid calls, or response bytes?
• Can one tenant occupy every worker, queue slot, key, or connection?
• What observable timing, cache, or error difference reveals protected state?
• Can retries, failover, or 0-RTT duplicate a side effect?
• What control fails open when policy, telemetry, identity, or time is unavailable?

A pre-mortem assumes the launch failed and reconstructs ordinary organizational causes:

• Six weeks after launch, p99 doubles only for the largest tenant. What skew did the review average away?
• A cell fails during peak and the origin collapses. Which reserve or warm-up assumption was false?
• Rollback restores binaries but not state authority. At what migration phase did rollback cease to be symmetric?
• The bill triples without higher request volume. Which fan-out, retry, egress, or paid-side-effect term was omitted?
• On-call cannot determine whether degradation is safe. Which invariant or control signal is missing?
• A forensic investigation cannot reconstruct access. Which privacy decision removed necessary evidence, and was escalation designed?

Use the prompts as a deck: pick the four that target the design’s largest exposures, assign a scribe, and turn plausible failure stories into owners, tests, monitors, or explicit accepted risks. A generic checklist read aloud produces compliance, not insight.

Write a decision record that can expire

The review output is a short architecture decision record, not meeting minutes:

Title / status / owner / date / decision deadline
Context: outcome, workload, objectives, constraints, state, threat boundary
Decision: mechanism and scope
Alternatives: evidence and explicit rejection rationale
Consequences: benefits, costs, risks, operational and migration effects
Assumptions: IDs linked to evidence, confidence, tests, and owners
Validation: pre-launch, rollout, production, correctness, failure, recovery
Migration and rollback: phases, authority, abort thresholds, irreversible points
Security/privacy/operability: controls, telemetry, retention, incident ownership
Expiry and revisit triggers: date, demand, SLO, cost, incident, version, evidence
Post-launch comparison: predicted versus observed, disposition, superseding record

Use lifecycle states that match action: proposed → evidence pending → accepted → implementing → observed → superseded or retired. Rejected records remain useful when they preserve why. An accepted record should be treated as history; new evidence can append an observation, while a materially changed choice should supersede it so the original context remains visible.

Expiry does not mean the design automatically becomes invalid. It means an owner must compare current conditions with the decision boundary. Revisit triggers should be machine-observable where possible: demand mix changes 20%, cache hit rate falls below 92%, a dependency version changes its concurrency model, cost/outcome rises 15%, a relevant incident occurs, regulation changes, or rollback rehearsal exceeds ten minutes.

Applied rewrite: from opinion to ADR

The original claim—“use the faster distributed cache”—can become:

Accept Option 2 for customer-read objects in two regional cells, conditional on a cell-loss test proving origin demand stays at or below 7,600 requests/s with 10% uncertainty reserve, a revocation fault test proving 30-second propagation, and rollback rehearsal from every migration phase within ten minutes. The decision targets the 2027 Q2 peak mix and 750 ms customer-read objective. Option 1 is rejected because its modeled cell-loss origin demand is 9,100 requests/s; Option 3 misses freshness for mutable records; Option 4 requires an origin-capacity purchase with worse fully loaded cost. The decision expires 2027-06-30 and is revisited if hit rate falls below 92%, origin safe capacity or workload mix changes 15%, rollback exceeds ten minutes, or a security/performance incident crosses this path.

That paragraph is evidence-backed only if its linked artifacts exist. It distinguishes facts, conditions, and recommendation. It also states what evidence can reverse the decision.

Post-launch: compare the forecast with reality

Approval starts an observation contract. At a declared interval, compare:

• forecast and observed workload distribution, not only total volume;
• predicted and observed latency, throughput, correctness, freshness, and cost;
• expected and actual resource demand and first saturation;
• cache, queue, retry, failure, degradation, and recovery behavior;
• security/privacy control performance and audit completeness;
• migration duration, rollback feasibility, operational toil, and incidents; and
• every high-exposure assumption.

The teaching packet observes a 91.5% hit rate against the 94% assumption, a 300 requests/s origin-capacity deficit, and a 13-minute rollback against a ten-minute claim. All three trigger review. This does not prove the architecture must be removed; it proves the accepted decision no longer describes reality. The owner can add capacity, adjust admission, improve rollback, narrow the workload, accept changed risk with authority, or supersede the record.

Do not grade the author on whether every prediction was correct. Grade the process on whether important uncertainty was visible, exposure was staged, production evidence arrived, and the organization reacted before harm compounded.

Review anti-patterns

Benchmark theater: a precise graph without claim scope, correctness, raw data, adverse state, or transfer limit. Repair it by writing the claim first and selecting the smallest valid experiment.

Authority by anecdote: “I used this at scale.” Experience should generate hypotheses and failure questions. It does not replace evidence for this workload, team, state, and objective.

Premature product choice: comparing vendor features before deciding state, consistency, path, failure, and operating requirements. Compare mechanisms against criteria first.

Committee design: everyone edits the architecture and no one owns the final trade-off. Keep one decision owner and one accountable approver; assign reviewers to explicit risks.

Traffic-light optimism: turning nuanced uncertainty into a green cell. Preserve assumption, evidence, confidence, owner, and trigger behind the color.

Action-item acceptance: approving while every proof remains future work. Separate evidence required before commitment, before rollout, before full exposure, and after launch. A missing correctness or rollback proof may block earlier than an optimization experiment.

Permanent ADR: a record without expiry, monitors, or triggers becomes folklore. Make the decision’s validity boundary explicit.

Applied review exercise

Run the deterministic packet:

cd examples/performance-engineering-system-design-handbook/part-07/performance-design-review
node analyze.mjs
node verify.mjs

Review the proposal in four passes.

1. Reproduce the 58/100 score and identify the three required dimensions below the evidence floor. Do not “fix” the result by changing weights after seeing it.
2. Rank assumptions by exposure. For A2, design a cell-loss test with arrival mix, cache state, retries, correctness, origin capacity, duration, abort threshold, recovery, and raw evidence.
3. Rewrite the cache opinion as an ADR. Preserve all four alternatives, the selected boundary, conditions, consequences, rejection and reversal evidence, migration authority, expiry, and triggers.
4. Reconcile launch evidence. Explain the 2.5-point hit-rate miss, 300 requests/s capacity deficit, and three-minute rollback miss. Decide whether to mitigate, restrict, supersede, or accept each deviation and name the authority.

A strong review is short because the decision is well framed, not because unknowns were omitted. It produces a clearer design, a bounded evidence plan, and a record a future engineer can challenge with new facts.

Evidence and limits

• AWS Prescriptive Guidance on ADRs describes records containing context, decision, consequences, ownership, and lifecycle. Mercury adds expiry, assumption evidence, performance gates, and post-launch comparison as local practice.
• Google Engineering Practices’ code-review standard states that technical facts and data overrule opinion and preference. Architecture review has different scope and authority, but the evidence-over-taste principle transfers.
• The score weights, floor, exposure scale, assumptions, alternatives, thresholds, and observations are modeled teaching policy reproduced by examples/performance-engineering-system-design-handbook/part-07/performance-design-review/. They are not an industry standard or proof that a design is safe.

A review system can still decay if nobody owns the service objectives, tools, regression policy, and exceptions between major decisions. Chapter 65 moves from one defensible review to the organization that sustains performance across teams and years.