Appendix M — Review Question Bank and Design Drills

“Provision ten cores,” the reviewer says. The arithmetic appears clean: 3,200 operations/s × 1.6 ms of CPU per operation = 5.12 cores, doubled for safety. The facilitator pauses the review. The requested failure reserve is 25%, the safe utilization is 65%, and the unit is valid checkout, not every retrying attempt. Ten cores may be close, but the reasoning has not yet protected the denominator, failure state, or objective.

A useful drill rewards the moment when a team stops defending an answer and improves the question. This bank practices the handbook’s method: establish boundary and invariant, quantify work and state, trace mechanisms through normal and failed paths, demand discriminating evidence, and make a conditional decision. It is for individual study, team reviews, incident training, and architecture workshops—not candidate assessment, trivia, or vendor recall.

The machine-readable index at examples/performance-engineering-system-design-handbook/appendices/review-question-bank/bank.json records packet types, timing, obligations, scoring dimensions, and checked calculation answers. verify-bank.mjs validates its structure and arithmetic. The packets here contain the context and rubrics that make those fields teachable.

How to run a drill without turning it into theater

Choose one decision and timebox the work. Give participants the prompt first; reveal evidence in stages only when they state what it could discriminate. Permit clarifying questions, documented assumptions, and “insufficient evidence.” Do not award hidden-product knowledge.

Every response should leave seven inspectable traces:

1. Boundary: operation, useful unit, population, state, interval, and objective.
2. Model: a causal path from workload through resources, waiting, coordination, and failure.
3. Arithmetic: equations with units, conservation rules, and sensitivity.
4. Correctness: invariants, authority, completion semantics, and allowed loss or duplication.
5. Evidence: existing signals, competing hypotheses, least risky discriminator, and validity limits.
6. Failure and recovery: overload, skew, partial failure, degraded operation, and restoration demand.
7. Decision: a conditional choice, rejected alternatives, reversible next step, and stop rule.

A polished architecture that skips an invariant should score below an incomplete design that identifies the missing decision. A calculation that exposes uncertainty should score above fake precision.

Short calculation drills

Core demand under failure reserve

Mercury admits 3,200 valid checkouts/s. Matched profiles estimate 1.6 CPU-ms per valid checkout for the checkout workers. In the modeled zone-loss state, service demand rises 25% because of routing and cache loss. The capacity policy permits 65% sustained CPU utilization so bursts and scheduler variation do not consume all headroom.

Calculate nominal busy cores, failure-state busy cores, and provisioned cores. Then name three reasons the model could understate demand.

Answer. Nominal demand is:

3,200 checkout/s × 1.6 CPU-ms/checkout × 1 s/1,000 ms = 5.12 CPU-s/s = 5.12 busy cores.

Failure demand is 5.12 × 1.25 = 6.40 busy cores. Provisioned capacity is 6.40 / 0.65 = 9.85, so ten equivalent schedulable cores is the arithmetic minimum. A review must still account for attempts excluded from valid goodput, per-core heterogeneity, quotas/affinity, tenant or key skew, correlated dependency work, deployment overlap, and uncertainty in the 1.6 ms estimate. “Ten” is not a host count until placement and failure domains are defined.

Drain a queue instead of celebrating recovery

After an 80-second dependency slowdown, a durable queue contains 18,000 valid events. New arrivals continue at 1,200/s. Restored workers complete 1,500 valid events/s, with no retries or new failures in this simplified model.

How long until the queue drains? What service rate would drain it in 30 seconds? What observation would invalidate this calculation?

Answer. Net drain is 1,500 − 1,200 = 300 events/s; drain time is 18,000 / 300 = 60 s. A 30-second target requires 1,200 + 18,000/30 = 1,800 valid events/s. The answer is invalid if arrival or completion rates are not stable and conserved, if “complete” includes duplicates/invalid work, if backlog age changes scheduling, or if recovery traffic reduces service demand elsewhere. Report oldest-event age and valid goodput, not queue length alone.

Size repair bandwidth with foreground protection

A store must rebuild 8 TiB. The incident objective allows 10 hours. The path provides 400 MiB/s physical throughput, but foreground traffic needs 120 MiB/s and policy reserves 15% of physical throughput for uncertainty and control traffic. Encoding amplification is 1.25 transferred bytes per usable byte.

Can the objective be met without changing the constraints?

Answer. Reserving 15% leaves 400 × 0.85 = 340 MiB/s; foreground leaves 220 MiB/s for repair. The transfer is 8 TiB × 1.25 = 10 TiB = 10,485,760 MiB. At 220 MiB/s, modeled repair takes 47,663 seconds, or 13.24 hours. It misses. A defensible change might add an independent path, reduce amplification, temporarily alter the foreground objective with authorization, or revise the recovery objective. Sending at 291.3 MiB/s would meet ten hours, but the current safe repair budget is only 220 MiB/s.

Percentile sample intuition

A window contains 8,000 eligible requests and no losses. Roughly how many observations occupy the slowest 1%? If four hosts each report a p99, may the four values be averaged to obtain fleet p99?

Answer. About 80 observations occupy the upper 1%, subject to the quantile convention. Four p99 values cannot be averaged into the fleet p99 because each discards distribution and population weight. Merge compatible raw observations or histogram counts, then compute the quantile; preserve loss, bucket, interval, and approximation rules.

Diagram critique exercises

The symmetric-store failover map

A proposed diagram contains gateway → orders → {db-a, db-b, db-c}. All arrows are solid. The three stores sit in two dashed zone boxes. A green note says “HA; fail over automatically.” No queue, authority mark, acknowledgement rule, quantity, or recovery path appears.

Critique it using Appendix J, then write the minimum diagram ledger needed before approving failover.

Strong response. The reviewer first writes the scenario header: zone-b loss during peak retail checkout; one durable order per payment ID within 250 ms; modeled topology revision and date. They mark one authority for the mode, classify replicas and derived state, state whether acknowledgement waits for replication, and define promotion eligibility plus fencing. Synchronous calls, asynchronous replication, and bulk repair receive different edges. The design shows the limiter and bounded queue, the checkout critical path, the recovery path, budgets, offered/admitted/goodput rates, data volume, and separately evidenced failure boundaries. Green proximity is discarded as evidence.

Rubric. Full credit requires completion semantics and authority, not a prettier drawing. Deduct when a response assumes zone independence, treats replicas as interchangeable, omits degraded capacity, or makes recovery invisible. An excellent response asks which claim belongs in a second diagram instead of crowding all modes together.

The missing asynchronous lifetime

A trace-derived architecture map ends at API → topic. The arrow is labeled “checkout complete, p99 90 ms.” Consumers debit a ledger and notify fulfillment. The API returns after broker acknowledgement.

Identify the false completion claim. Redraw the logical paths for accepted input, durable business completion, and notification. State the clocks and populations required for each latency distribution.

Answer guide. Broker acknowledgement may complete accepted enqueue, not ledger debit or notification. The diagram needs an asynchronous message edge, queue ownership, consumer scheduling, ledger authority, duplicate/replay behavior, and causal linkage. Measure API acceptance from ingress to durable enqueue; business completion from the same logical origin to authoritative debit; notification from debit or another named event to acknowledged delivery. Censored, failed, expired, and duplicated units require explicit policy.

Incident diagnosis packets

Packet A: low average CPU, rising checkout tail

At 09:10 UTC, Mercury checkout response p99 rises from 145 ms to 410 ms. Valid goodput falls 8%; attempt rate rises 22%; fleet CPU average is 47%; one shard’s oldest queue age climbs from 20 ms to 6.2 s. A deployment completed at 09:04. Region and host aggregates are available; tenant/key breakdown is not. No data loss is observed.

Stage 1 — stabilize. What action limits feedback while preserving evidence? A strong response considers admission or retry reduction at the earliest safe boundary, protects authoritative checkout, degrades optional enrichment, records the change, and avoids an unbounded profiler capture.

Stage 2 — hypotheses. Produce at least three predictions: hot-key/tenant skew at one shard; lowered effective quota or runnable delay after deployment; dependency wait or connection-pool starvation; retry amplification. “CPU issue” is not discriminating.

Stage 3 — evidence choice. Ask for per-shard offered/admitted/goodput, queue age, retry reason, quotas/throttling, runnable delay, dependency/pool wait, and deployment/config diff aligned on one clock. Choose the least intrusive capture only after existing signals. Low fleet average does not exonerate a shard, quota, core, or wait state.

Stage 4 — decision. Rollback is justified when the configuration change predicts and aligns with the onset and the rollback risk is bounded; skew mitigation is justified when concentration and per-shard saturation discriminate it. Preserve an “inconclusive” branch.

Rubric. Boundary and stabilization are mandatory. Full credit requires competing hypotheses with predicted observables, feedback control, correctness protection, a stop rule, and recovery validation through queue drain—not just latency returning.

Packet B: recovery that will not finish

Vaultspan loses one storage domain. Foreground reads remain within objective, but repair ETA grows from 9 to 27 hours. Network utilization is 58% fleet-wide, device utilization 61%, and repair workers are at their configured concurrency. Retry bytes rise 3×; one destination rack reports intermittent checksum failures.

Diagnose without equating utilization with capacity. Trace useful repair bytes, attempted bytes, checksum/retry paths, per-rack queues, device service, network loss/retransmission, and the scheduler priority shared with foreground work. Stabilization may isolate the destination and cap retries before raising concurrency. The answer must protect foreground goodput and data correctness, define a repair stop condition, and recalculate the recovery objective after excluding invalid destinations.

Architecture trade-off prompts

Admit before or after durable enqueue

A messaging service can reject at the edge before durable enqueue, or accept every authenticated message into a regional log and shed later work. Compare the choices for user semantics, burst absorption, cost, fairness, overload transfer, retry amplification, durability, and regional recovery.

There is no universal winner. Edge admission protects scarce downstream state and returns immediate refusal, but requires fair distributed limits and may reject work the log could absorb. Durable-first admission preserves accepted work and decouples producers, but converts overload into backlog, storage, expiry, and recovery debt; “accepted” needs a deadline contract. A strong proposal distinguishes priority classes, caps queue age/bytes, makes producer retry behavior explicit, reserves recovery throughput, and stages migration with shadow decisions and invariant checks.

Cache the result or the authority

Canopy Analytics proposes a shared result cache for multi-tenant queries. Compare exact-key immutable results, bounded-staleness derived results, and cached authoritative metadata. Address invalidation, tenant isolation, skew, memory amplification, stampede control, hit-ratio denominator, p99 behavior, correctness, and cold recovery.

Reject “higher hit rate” as the decision. Require saved service demand per valid query, miss cost, eviction concentration, stale-result policy, and failure behavior. A cached authority is usually a category error unless authority and consistency actually move.

Replicate synchronously or acknowledge locally

For an order ledger, compare local durable acknowledgement plus asynchronous cross-domain replication with synchronous acknowledgement across domains. State the failure classes each survives, latency distribution, write availability, fencing, recovery point, bandwidth, and operator complexity. The response must name the business outcome that can tolerate or cannot tolerate acknowledged loss; geography alone does not choose the policy.

Benchmark validity reviews

The cache benchmark with a changed denominator

Team A reports: “Cache v2 improves throughput 42% and p99 31%.” Baseline ran 10 minutes after cold start at 800 attempted queries/s with 60% small queries. Treatment ran 30 minutes after a warm-up at 1,200 attempted queries/s with 82% small queries. Baseline failures were 4.1%; treatment failures 0.3%. Each condition ran once on different host pools. Only client summaries remain.

Review job. Decide what, if anything, the benchmark supports and design a valid next experiment.

Answer guide. It does not isolate cache v2. Rate, mix, warm-up, duration, host pool, error population, and likely state differ; attempted throughput is not valid goodput. Preserve the result as exploratory, not comparative evidence. Predeclare the claim and operational threshold; use the same eligible-query correctness rule; block or randomize host pool and run order; prepare equivalent cache/data state; match operation, size, tenant/key skew, and concurrency; run independent repetitions through warm-up and steady state; retain raw per-operation observations, manifests, errors, resource counters, and cache events. Report effect and uncertainty, valid goodput, latency distribution with loss policy, service demand, hit/miss cost, and transfer limits.

Rubric. Full credit identifies confounders and supplies controls tied to a decision. Merely asking for “more runs” is insufficient. An excellent response tests both cold-recovery and steady-state claims rather than averaging them.

The profiler optimization

An engineer removes the widest flame-graph frame. Its share falls from 18% to 11%; total CPU samples rise from 30 to 44 CPU-seconds while goodput rises from 800 to 1,200 valid operations/s. Calculate CPU service per valid operation and judge the claim.

Baseline is 30/800 = 37.5 CPU-ms/op; treatment is 44/1,200 = 36.7 CPU-ms/op, about a 2.2% reduction. The narrower percentage does not establish an 7-point absolute saving because the total changed. Whether 2.2% matters depends on run variation, matched workload, correctness, tail, capacity threshold, and cost. Inspect absolute stack deltas and uncertainty.

Staff-level design-review scenarios

Scenario 1: Mercury regional checkout recovery

Mercury serves 3,200 valid checkouts/s in region A at peak, with a 250 ms p99 objective and no duplicate ledger debit per payment ID. Seventy percent of traffic originates near A; region B normally serves 25% of global traffic and retains 35% spare safe capacity. The order ledger is authoritative in A and asynchronously replicated to B with a 30-second freshness objective. A contains two zones; current peak continues after one-zone loss. Product asks for automatic regional failover with “no lost orders” and a ten-minute recovery objective.

Design the review, not merely the target architecture. Clarify whether ten minutes means traffic restoration, ledger authority, or full backlog/repair completion. Quantify B’s simultaneous native and failed-over demand; 35% spare capacity cannot absorb all of A. Define admission and priority policy, payment identity, acknowledgement, replication lag, promotion/fencing, ambiguous client outcomes, DNS/routing, and the exact no-loss boundary. Separate RTO from RPO. Offer at least two options: invest in active capacity and synchronous or bounded-loss authority transfer, or accept a constrained recovery mode that admits priority traffic and reconciles uncertain orders.

Evidence should include failure-state load tests, lag and promotion trials, duplicate/loss invariants, routing convergence, payment-provider behavior, queue drain, repair bandwidth, and cost. The reversible next step is a shadowed failover/control-plane exercise with no authority transfer, followed by a bounded cell. Stop on invariant violation, uncontrolled lag, foreground objective harm, or ambiguous fencing.

Answer rubric. A strong response refuses the impossible combination until objectives and authority are explicit, does rough capacity arithmetic, connects normal/failure/recovery paths, and gives product a choice with cost and residual risk. It does not hide behind “multi-region.”

Scenario 2: Canopy interactive and batch scheduling

Canopy runs interactive tenant queries with a 2-second p99 objective and overnight batch jobs with a 06:00 completion objective. Both use the same 400-worker pool and object-store egress budget. At 10:00, 6% of tenants produce 58% of scanned bytes. At 02:00, batch uses 85% of workers; interactive p99 rises to 7 seconds although average CPU is 54%. The proposal adds 200 workers and one global FIFO queue.

Review three alternatives: separate pools, a shared pool with class/tenant scheduling, and admission with bounded queues plus elastic workers. Model query cost by scanned byte and CPU/service demand, not request count alone. Expose object-store egress, per-tenant skew, head-of-line blocking, worker memory, and startup time. Define interactive goodput, batch remaining work, starvation bounds, cancellation, and failure/recovery capacity. More workers may amplify the egress bottleneck and cost while FIFO preserves unfairness.

A credible design might reserve a base interactive share, use weighted fair scheduling with per-tenant cost estimates, cap batch concurrency, admit by queue-age budgets, and borrow idle capacity with reclaim rules. Test adversarial cost-estimation errors and one-tenant skew. Roll out scheduling decisions in observe-only mode, then one tenant class, with rollback on starvation, cost, correctness, or p99 harm.

Answer rubric. Full credit connects scheduling to a constrained resource, handles unknown job cost and fairness, protects both objectives during failure, and accounts for economics. “Use priority queues” without ownership, weights, aging, admission, and evidence is incomplete.

Facilitation notes

Individual practice

Spend one-third of the time clarifying and modeling, one-third calculating or tracing, and one-third deciding and attacking the answer. Write assumptions before consulting the guide. On a second pass, change one variable—skew, failure domain, loss policy, or recovery objective—and observe whether the decision survives.

Team workshop

Assign rotating roles: boundary keeper, invariant keeper, modeler, evidence skeptic, and operator. Roles are lenses, not status. Give the prompt to everyone; reveal stage evidence only after the group predicts what it would show under at least two hypotheses. End with one decision record, one unresolved question, one reversible experiment, and one stop rule.

Incident simulation

Use a facilitator who controls the clock and evidence cards. Participants must state a stabilization goal before requesting invasive evidence. Charge fictional time or objective impact for broad captures, restarts, and uncontrolled load changes. Inject one misleading aggregate and one clock or population mismatch. Debrief feedback loops, evidence preservation, and recovery validation rather than praising fast root-cause guesses.

Design review

Do not let the most senior voice reveal a preferred architecture first. Begin with reader questions on workload, success, authority, and failure. Record rejected alternatives with the conditions that would reverse the rejection. When the team uses an anti-glossary phrase, pause only long enough to qualify it; the purpose is sharper thought, not vocabulary policing.

Psychological and operational safety

Use fictional systems and sanitized evidence. No drill requires production credentials, personal data, heap dumps, packet capture, destructive failure injection, or a live command. Make it safe to say “I do not know.” Score the quality of assumptions and next evidence, not familiarity with a participant’s employer.

Answer rubric and debrief

Score each dimension from 0 to 3:

A total score is less useful than the shape. A response scoring 3 on arithmetic and 0 on correctness must return to the invariant. A response with a strong boundary and weak evidence needs a discriminating experiment, not a different architecture. Facilitators should record one observed strength, one consequential gap, and the next practice packet.

The answer key is not a hidden target design. Alternative answers pass when their assumptions are explicit, calculations cohere, invariants survive, evidence can falsify the model, and the decision is appropriate to the stated constraints. Reject answers that solve a different workload while sounding sophisticated.

Build a new packet

A reusable packet contains:

• one decision and why it matters;
• a bounded workload, state, topology, and objective;
• enough numbers to calculate, with deliberate uncertainty;
• a correctness invariant and at least one failure/recovery condition;
• staged evidence whose meaning depends on population or clock;
• two credible alternatives and a condition that changes the choice;
• facilitator reveal order, expected misconceptions, and safety limits; and
• a rubric that rewards reasoning traces rather than exact wording.

Test the packet on one reader. If they can answer from a memorized slogan, add a conflicting constraint. If they cannot begin without private domain knowledge, supply the missing mechanism. If every reasonable assumption produces one architecture, check whether the prompt is merely a disguised quiz.

Field card

• Clarify decision, boundary, unit, population, state, interval, objective, and invariant.
• Draw normal, overload, failure, and recovery paths with authority and queues.
• Calculate with units; separate attempts, admitted work, goodput, and amplification.
• State uncertainty and test sensitivity at the binding constraint.
• Offer competing hypotheses or alternatives with predicted consequences.
• Ask for the least risky evidence that can change the decision.
• Preserve correctness, feedback control, recovery demand, and economics.
• Make the decision conditional; name the reversible next step and stop rule.
• Debrief the reasoning gap, not the person.

The decision rule is: accept a drill response when its explicit boundary, causal model, unit-correct arithmetic, invariant, failure/recovery behavior, and falsifiable evidence support a conditional action; do not require the facilitator’s architecture when another answer survives the same constraints and rubric.

Appendix N organizes the primary sources behind these mechanisms. Use it after a drill exposes a knowledge gap; do not replace the missing reasoning with a bibliography before the decision has been framed.