The Performance-Aware System Design Loop

The consequential design decision is rarely “cache or no cache?” It is whether the team can explain what the system promises, what work arrives, where that work waits, which state matters, what breaks first, and what evidence will cause the design to change.

A nominal architecture can answer the functional request and still be incomplete. Boxes have enough aggregate capacity, a load balancer distributes traffic, and a replica protects stored data. Yet the diagram says nothing about a hot tenant, a full queue, a stale derived view, a dependency timeout, replay after failure, or the operator who must distinguish them. Capacity is necessary; it is not a behavior specification.

The performance-aware design loop turns the concepts of Part I into an inspectable argument. It is iterative, because measurements revise workloads and failures revise assumptions. It is also finite enough for a design review: scope, model, estimate, draw, choose, budget, protect, prove, and record. A design is not ready when it merely works at nominal load; its assumptions, limits, observability, failure behavior, and evolution path must be explicit.

Ten steps, with assumptions at the center

1. Scope the function, invariants, and exclusions

Name the unit of useful work and the terminal outcome. For Mercury search, the unit is one admitted search request that returns a policy-valid result or an explicitly allowed degraded result. “An HTTP response” is too weak: a timeout page and a correct response would count alike.

Separate functional requirements from invariants. A functional requirement says users can search recent records. An invariant says authorization filters are applied to every returned record and a result never claims a newer index position than the authoritative store has acknowledged. Then write exclusions: bulk export, global ordering, and arbitrary historical analytics are outside this service boundary. Exclusions prevent accidental architecture by aspiration.

2. Build the workload envelope and growth states

A single peak rate is not an envelope. Record arrival shape, request classes, payloads, skew, locality, fan-out, read/write mix, background work, and retention. Mercury’s teaching model uses 1,800 requests/s steady and 3,000 requests/s peak, 86% reads, 14% writes, 2.4 kB requests, and 12 kB responses. Those are modeled inputs, not observed facts.

Model at least five states: nominal, skewed, overloaded, failed, and recovering. A regional failure can concentrate traffic while reducing capacity. Recovery adds cache fill, log replay, repair, and validation work. Growth scenarios should name both time horizon and trigger: “3,000 requests/s within twelve months” is auditable; “internet scale” is not.

3. Define objectives and failure semantics

Bind indicators to populations and boundaries. Mercury’s illustrative latency objective applies from edge acceptance to a correct or policy-valid degraded response. Rejections before admission are reported separately. Define availability, freshness, durability, and correctness alongside latency; otherwise a fast stale result can improve the wrong indicator.

Failure semantics are product decisions expressed as system behavior. May a search omit enrichment? How old may cached data be? Is a duplicate background update harmless? When a deadline expires, is work cancelled or allowed to consume capacity invisibly? State which outcomes are retryable and who owns the retry budget.

4. Estimate scale and resource demand

Estimate before choosing a product. Use ranges where uncertainty dominates precision. At peak, the modeled response egress is:

3,000 requests/s × 12,000 bytes/request = 36,000,000 bytes/s = 36 MB/s

Weighted CPU demand is 0.86 × 2.4 + 0.14 × 5.8 = 2.876 ms/request, or 8.628 core-seconds/s at peak before headroom, background work, kernel time, or imbalance. Thirty days of raw write payload at the steady rate is about 1.568 TB before indexes, replication, metadata, compaction, and compression. Each result must retain its unit and amplification boundary.

Connections and service demand matter as much as bytes. A 9 ms store visit at 206.4 misses/s consumes about 1.86 concurrent store operations by Little’s Law under its assumptions. Average concurrency does not size a tail-sensitive pool; it only catches impossible diagrams and creates a hypothesis for a distribution-aware test.

5. Draw flow, ownership, queues, and failure domains

An architecture drawing should answer where the request goes, where authoritative state lives, what can be rebuilt, where work waits, and which failures are shared. Draw the fast path, slow path, failure path, and recovery path. Mark resource ownership intervals, not only calls.

The cache is derived; the primary store is authoritative; the event log supports replay. That classification determines recovery and correctness. The admission queue and worker queue are bounded because waiting is a policy, not free storage. Zone outlines expose a deliberate limitation: the pictured placement does not survive loss of Zone B without another authority or failover design.

6. Choose state and interface contracts

Only now choose partitioning, replication, consistency, and APIs. Begin with ownership: who may accept a mutation, what constitutes commit, and how a reader learns which version it saw. Then choose a partition key against access patterns and skew, not just even hash distribution. State whether caches are invalidated, refreshed, or version-checked; whether log consumers are idempotent; and how schemas evolve across mixed versions.

Interfaces must carry the semantics the objectives need: deadlines, idempotency keys, pagination limits, result version, degradation markers, and explicit error classes. Hiding these behind a generic success/failure API forces callers to infer behavior and makes retries unsafe.

7. Allocate latency and resource budgets

The 100 ms illustrative budget allocates 15 ms to edge work, 20 ms to admission and API work, and 65 ms to the data path. The sum is a constraint, not evidence that component p99 values can be added. Serial and parallel composition, covariance, retries, and queueing require trace or distribution-level validation.

Allocate resource budgets per correct result: CPU-ms, store operations, bytes, connection-ms, and background amplification. Reserve capacity for failure and recovery work. If foreground traffic uses every nominal worker, replay either never catches up or steals the objective. Budget uncertainty explicitly, for example by carrying low/base/high demands and identifying which decision changes across them.

8. Protect useful work under overload and recovery

At 3,000 arrivals/s, a 200 ms maximum admission age implies a 600-request queue bound under the simple L = λW relationship. The equivalence does not make 600 automatically safe; memory per queued request, class fairness, bursts, cancellations, and service variability still matter.

Define admission before saturation, shed or degrade by business value and correctness, suppress retries when downstream capacity is unavailable, and preserve control-plane capacity. Decide what happens to in-flight and queued work during deploy and shutdown. Recovery needs a workload budget: log replay, cache warming, repair, and client retries must not recreate overload.

Now review the common “sufficient capacity” design. Suppose workers can complete 4,000 jobs/s at 12 ms service with concurrency 48, above the 3,000/s modeled peak. If the input queue is unbounded and clients retry after 250 ms, a dependency slowdown can retain old work, accept new work, and multiply attempts. The capacity comparison was true only in the nominal state. The design has no bounded wait, admission rule, retry ownership, degraded result, or drain plan. It fails the review despite spare modeled worker capacity.

9. Specify observability, experiments, and rollout

Every assumption needs a signal or a scheduled re-measurement. Preserve offered, admitted, rejected, completed, correct, degraded, cancelled, and retried populations. Measure queue age as well as depth; service time separately from waiting; demand per correct result; and version, tenant, region, outcome, and failure state where cardinality policy permits.

Write experiments as claims. “Validate that the 92% cache-hit assumption holds for the modeled key distribution and that misses do not violate the data-path budget” is testable. Record generator validation, warm-up, steady state, correctness checks, raw data, uncertainty, and transfer limits. Run nominal, skew, overload, dependency slowdown, zone loss, deploy, and recovery scenarios.

Roll out by risk. Shadow or replay when side effects permit, use a small cohort, compare matched populations, define abort thresholds, and prove rollback semantics. A rollback that restores binaries but cannot reconcile state is not a rollback plan.

10. Evaluate fit, record the decision, and revisit it

Performance trades against cost, security, operability, and organizational capacity. A faster derived store can enlarge the privacy surface. A sophisticated partition scheme can exceed the team’s ability to rebalance safely. More replicas can raise durability and read capacity while increasing write, repair, and coordination work. Record these effects rather than hiding them under “non-functional requirements.”

End with alternatives, assumptions, uncertainty, and triggers. A trigger is observable: cache hit rate below 85% for the defined population, peak demand above 2,700/s for four weeks, replay time above 30 minutes, or a new data-residency boundary. “Revisit if scale grows” assigns no decision.

A one-page design review scorecard

Use the scorecard to expose missing arguments, not to average away a fatal omission. Mark each row evidenced, modeled, assumed, or missing and attach the decisive artifact.

Any stop condition blocks the design regardless of the other rows. A score of nine out of ten is meaningless if the missing row is authoritative state or overload behavior.

Architecture decision record: the smallest durable argument

Decision: <mechanism and boundary>
Status/date/owners: <proposed | accepted | superseded>
Context: <workload, objectives, invariants, operating states>
Decision: <chosen option and interface/failure semantics>
Alternatives: <credible options and why they lost here>
Model: <demands, budgets, capacity, cost, uncertainty>
Evidence: <measured/modeled/simulated/inferred; reproduction path>
Consequences: <correctness, latency, capacity, cost, security, operations>
Overload/recovery: <bounds, admission, degradation, retry, repair>
Rollout/rollback: <compatibility, cohorts, abort thresholds, state plan>
Assumptions and transfer limits: <where the argument stops>
Revisit triggers: <observable thresholds, owner, review date>

Record why a credible alternative lost. “Option B is slower” is weak; “Option B adds a cross-zone coordination round trip to every write, exhausting the 20 ms write budget under the stated topology” is reviewable. Separate modeled claims from measurements. The record should survive both staff turnover and the next load shape.

The 60-minute design drill

Use the standard canvas under a clock to practice prioritization, not diagram speed.

For Mercury, use the fixture inputs, calculate the bounds, then attack the architecture: hot tenant, cache failure, slow store, full queue, duplicate replay, Zone B loss, mixed-version deploy, and traffic growth. A useful pre-mortem names the causal chain—“store latency increases, connections remain owned, admission queue ages, clients retry, correct goodput falls”—and the control that breaks it. “Database outage” is an event label, not an adversarial review.

The drill is complete when the next investigation is obvious. It need not select every implementation. Unresolved questions become owned evidence tasks with decision dates.

Run the adversarial review across states

A pre-mortem begins with a violated objective and works backward through mechanisms. Assign reviewers different attacks so the discussion does not converge on the architecture author’s favorite failure.

Skew: one tenant generates 35% of keys and 60% of cache misses. Does the partition and admission policy isolate the tenant, or does even hashing of keys conceal uneven work per key? Inspect per-tenant offered load, miss rate, store demand, and queue age. A global 92% hit ratio can coexist with one customer receiving nearly all misses.

Saturation: arrival rate reaches 3,600/s while worker capacity remains modeled at 4,000/s. The 400/s arithmetic margin is not safe capacity if service time rises with cache misses, workers share a constrained store, or bursts arrive faster than the 200 ms queue-age policy. Ask which threshold rejects first, which class retains service, and whether rejection itself is cheap.

Partial failure: Zone B becomes slow rather than unavailable. Timeouts retain API resources; callers hedge; the event log accepts writes that workers cannot drain. Reviewers should trace one first attempt and all repeated work, then identify the deadline and admission layer that prevents amplification. Fail-open and fail-closed choices must be stated separately for authorization, search freshness, and enrichment.

Recovery: Zone B returns with a log backlog and cold cache. If replay consumes all 48 workers, foreground writes wait; if foreground always wins, recovery time is unbounded. Require a recovery objective, reserved or dynamically governed capacity, progress and divergence signals, and a rule for pausing replay when foreground objectives approach their limits.

Change: a mixed-version deployment changes the cache key and event schema. Can old workers replay new events? Can rollback read state written by the new version? Performance review must include compatibility because a fast path that prevents rollback converts a latency regression into a migration incident.

For each attack, record four items: violated objective or invariant, causal chain, earliest decisive signal, and control with its own failure mode. “Add autoscaling” is not a control until its signal, delay, maximum, cold behavior, and interaction with queues are modeled.

Make uncertainty change the decision

Uncertainty should not live in a footnote. Put low, base, and high estimates beside the architecture and ask whether the chosen mechanism changes.

Suppose read CPU demand is between 2.0 and 3.2 ms, write demand between 5.0 and 7.5 ms, and the peak rate between 2,700 and 3,300/s. The weighted base demand is useful for arithmetic, but the review should calculate the corners that matter. If the high case still fits the same topology with explicit headroom and failure reserve, the decision is robust. If it requires twice the worker fleet or crosses a store limit, the team needs an experiment or a staged architecture—not another decimal place.

Distinguish three responses:

• measure now when uncertainty can reverse a near-term irreversible choice;
• design an option when the high case is plausible but not yet worth paying for;
• accept and monitor when the consequence is bounded and a trigger provides enough lead time.

This is also where organizational fit becomes quantitative. A partitioning scheme that needs weekly manual rebalancing creates operational demand. A cross-zone write path consumes both latency budget and incident complexity. A cache requiring perfect invalidation spends correctness risk. Name the team, tooling, on-call, security, privacy, and migration work as consequences with owners. Architecture cost is not only the invoice attached to compute.

Conditional decision rules

• Reject a design whose functional success, correctness invariants, authority, or exclusions are ambiguous.
• Do not select mechanisms until workload classes, skew, growth, objectives, and failure semantics constrain the choice.
• Treat every number as modeled, measured, simulated, or inferred; carry its unit, population, state, uncertainty, and transfer limit.
• Annotate critical paths, ownership, queues, authoritative and derived state, budgets, and failure domains on the architecture itself.
• Reserve and control capacity for overload, degraded operation, deploys, and recovery; nominal headroom is not a recovery plan.
• Require observability and experiments capable of falsifying consequential assumptions.
• Block review on any fatal omission; never average scorecard rows into a readiness percentage.
• Record alternatives and observable redesign triggers so architecture can evolve before a cliff.

With a shared design language in place, the next question moves beneath the boxes: how CPU execution turns instructions, dependencies, stalls, topology, and sustained load into useful work.

Sources and evidence scope

• RFC 2119, “Key words for use in RFCs to Indicate Requirement Levels” provides a disciplined vocabulary for normative interface requirements. It does not define product objectives or make an architecture correct.
• Google SRE, “Handling Overload” describes load shedding, per-customer limits, and graceful degradation in Google’s operational context. The mechanisms transfer; its thresholds and implementation choices do not.
• Google SRE, “Addressing Cascading Failures” documents queue, retry, deadline, and recovery interactions. It supports adversarial review, not the specific modeled Mercury numbers.
• The Mercury workload, budgets, architecture, queue bound, and worker capacity are modeled teaching evidence. examples/performance-engineering-system-design-handbook/part-01/design-loop-and-units/verify.mjs verifies arithmetic and internal consistency, not production behavior or vendor capacity.