The Performance Contract

Performance begins before a profiler opens and before an architecture diagram acquires a cache. It begins when a group of people decides which observable behavior must hold, for whom, under what demand, within which cost and risk limits.

That decision is a contract. It may be an external promise, an internal objective, a product constraint, or an engineering hypothesis awaiting validation. It always couples a user journey to a workload, a measurement boundary, a distribution of outcomes, and consequences when the objective is missed. Without those terms, “make it fast” is not a difficult requirement. It is not yet a requirement.

This book develops the mechanisms needed to satisfy such contracts—from processors and runtimes to queues, storage, networks, distributed coordination, experiments, and governance. The method keeps those mechanisms from becoming a catalog of tricks: ask what useful work enters, what correct outcome leaves, where time and resources accumulate, what failure does to the path, and which evidence would change a decision.

The first rule is therefore:

Do not accept a performance project until the affected user journey, workload, objective, measurement boundary, and economic constraint are explicit.

“Explicit” does not mean perfectly forecast. It means assumptions are visible enough to test, price, revise, and reject.

Performance is observable system behavior

Performance is the relationship between offered work, useful outcomes, elapsed time, resources, and variability. It includes more than request latency:

• latency: time from a defined start event to a defined outcome;
• throughput: completed units of work per interval;
• responsiveness: whether interaction produces timely, meaningful progress;
• freshness: delay between a real or authoritative change and the value a reader observes;
• efficiency: useful outcomes per unit of CPU, memory, I/O, network, energy, or money;
• predictability: how behavior varies across requests, tenants, regions, time, and operating modes.

Each term requires a unit and boundary. “Database latency” might mean client-observed duration including pool wait and network, server execution time, storage service time, or a sampled trace span. Those observations answer different questions. “Throughput” might count attempts, accepted messages, durable commits, or user-visible successful outcomes. Counting cheap failures can make a saturated system look productive.

Use goodput when the distinction matters: correctly completed, still-useful work per interval. A response produced after its deadline, a duplicate side effect, or an item later discarded may consume resources without contributing goodput. The correctness condition belongs inside the performance indicator.

Likewise, one number rarely represents a distribution. State population, outcome, interval, aggregation, and sampling. A percentile over all operations can hide a slow operation class; a percentile averaged across machines is generally not the fleet percentile; a daily percentile can hide a ten-minute overload. Objective construction and tail analysis will add precision; for now, an indicator is meaningful only when a reader can reconstruct what was observed.

The map rejects a common mistake: optimizing one node as if the others were fixed. Buffering may improve apparent availability while increasing freshness delay. A faster acknowledgement may move durability after the response. Higher utilization can reduce cost at nominal load and destroy latency headroom during bursts. Disabling encryption can make a benchmark faster while invalidating the system. Performance is a system property because the outcome emerges from these coupled choices.

“Fast” and “scalable” conceal the decision

Adjectives hide dimensions. Ask what would be different if the requirement were satisfied.

“The page must be fast” could concern first feedback, useful content, complete interaction, input responsiveness, or a business action. “The service must scale” could mean higher request rate, more tenants, a larger data set, wider geography, longer retention, more complex queries, or cheaper operation. These growth axes stress different resources and mechanisms.

Replace each adjective with five elements:

1. Journey: the person or system action whose outcome matters.
2. Workload: arrival pattern, mix, payload, data set, skew, locality, and operating mode.
3. Boundary: start and end events, including or excluding named queues and dependencies.
4. Objective: indicator, population or distribution, threshold, interval, and allowed misses.
5. Constraint: correctness, availability, security, durability, cost, energy, portability, or delivery limits.

Here are five transformations. The numbers are intentionally left for the owning organization; inventing a universal target would make the examples less rigorous.

These are contract shapes, not finished contracts. Workload dimensions, indicators, and budgets must still make each shape precise. A mature team refuses false precision as firmly as vagueness: a target copied from another product without user or economic evidence is a well-formatted guess.

Temporal correctness is still correctness

Functional correctness asks whether the result obeys its logical invariant. Temporal correctness asks whether the correct result arrives while it is still valid and useful.

An auction bid accepted after close can be logically valid data and an incorrect product outcome. A control-loop update arriving after the physical state changes can be numerically accurate and unsafe. Fraud scoring delivered after funds move has lost its prevention function. A recommendation returned after a user leaves contributes no useful outcome. Conversely, a monthly archival report may be entirely correct with hours of latency. Low latency is not intrinsically virtuous; fitness for the deadline is.

Make the relationship explicit:

useful outcome = functional invariant
              AND completion before the journey's validity deadline
              AND required durability, security, and freshness state

The deadline may be hard, soft, or economic. A hard real-time miss violates system correctness. A soft miss reduces quality or revenue. An economic deadline marks the point where additional work costs more than its expected value. The response to overload differs: reject before expensive work, degrade fidelity, serve bounded staleness, prioritize, or complete late. The contract chooses; an unbounded queue chooses accidentally.

Temporal requirements also interact with failure. A retry may increase success probability while consuming the remaining deadline and amplifying load. A cache may reduce latency while weakening freshness. Replication may improve read availability while adding write coordination. These are not arguments against the mechanisms. They are reasons to state the outcome before selecting one.

Negotiate across stakeholders

A performance contract crosses organizational boundaries because each stakeholder owns part of its meaning.

Users reveal which journeys and delays matter, how waiting is perceived, and when a result loses value. Their behavior is evidence, not a request for every interaction to be instantaneous.

Product owners connect behavior to value, segmentation, launch shape, and acceptable degradation. They decide whether freshness, completeness, or responsiveness should win when all cannot.

Engineers and architects translate demand into critical paths, resource demand, queues, state transitions, budgets, and feasible mechanisms. They expose where a proposed objective conflicts with physics, correctness, or delivery capacity.

Operations and SRE own deployability, observability, saturation, incident response, recovery, and the cost of maintaining headroom. They know whether a measurement and rollback can exist outside a design document.

Finance, security, compliance, and sustainability owners contribute constraints when spend, data handling, threat controls, geography, retention, or energy is material. Cost is not merely a late approval. A design that meets latency only at unaffordable utilization or staffing is not satisfying the product contract.

Negotiation should produce explicit conflict decisions. Suppose Mercury API can meet its interactive tail objective with either substantial idle headroom or aggressive caching. Headroom raises steady cost; caching changes freshness and invalidation risk. The charter names the user journey and the authoritative state, quantifies the workload envelope, chooses a freshness bound, prices headroom, and assigns the miss response. “Use a cache” is an implementation proposal; the contract explains whether it is eligible.

Document who may revise each term. Product cannot silently expand the workload after capacity approval. Engineering cannot narrow the population until a dashboard turns green. Operations cannot replace an end-to-end objective with host utilization because it is easier to collect. A decision record should preserve trade-offs and a revision trigger, not freeze assumptions forever.

Performance debt compounds quietly

Performance debt is the future decision and operating cost created by implicit demand, hidden coupling, missing evidence, or architecture that cannot be measured or bounded. It differs from “slow code.” A fast service can carry severe debt if nobody knows its capacity boundary or can reproduce its benchmark.

Common forms include:

• an API with no declared payload or fan-out envelope;
• an unbounded queue that turns bursts into delayed failure;
• synchronous dependency work hidden behind a convenient interface;
• high-cardinality state with no tenant or key-skew model;
• a benchmark without correctness checks, raw results, or environment;
• a cache whose hit rate is reported without miss cost or freshness;
• a dashboard that aggregates outcomes until the expensive class disappears;
• an objective with no admission or degradation policy;
• a cost model that counts provisioned capacity but not retries, transfer, or operations.

Debt compounds because later features inherit the invisible assumption. A “small” increase in fan-out multiplies dependency demand. A new tenant distribution invalidates the cache model. A retry policy converts a local slowdown into fleet overload. Teams then pay interest as incidents, emergency capacity, delayed launches, and experiments that cannot isolate causality.

Manage this debt with evidence and options. Add queue age before tuning workers. Preserve workload traces before replacing storage. Establish a correctness oracle before optimizing a query. Create a seam and rollback before a cache migration. Some debt is rational: a low-volume internal tool may not justify elaborate capacity modeling. Record the accepted exposure, signal that will reopen the choice, and cheapest next measurement.

The contract moves through a lifecycle

Performance engineering is not a validation phase after implementation. The contract is refined and tested throughout a loop:

SPECIFY → DESIGN → IMPLEMENT → VALIDATE → DEPLOY → OBSERVE → REVISE
    ↑                                                          │
    └──────────────── evidence changes the contract ───────────┘

Each transition has an evidence gate.

The arrow back to specification matters. Measurements can show that a target is economically irrational, a workload forecast is wrong, or a mechanism creates a correctness conflict. Revising a contract with stakeholders is not failure. Quietly changing the measurement boundary or excluding hard outcomes is.

Deployment is an experiment with users and failure consequences. Progressive exposure, automated rollback criteria, and comparable cohorts can bound risk, but only if metrics reflect the contract. CPU utilization may explain behavior; it does not replace user-visible latency and correctness. After rollout, observe both outcome and mechanism signals so a regression can be detected and diagnosed.

The lifecycle also prevents one-time “performance sign-off.” Demand, data, dependencies, compiler/runtime behavior, platform, cost, and team ownership change. Governance schedules objective review, capacity reassessment, benchmark refresh, incident learning, and architecture decisions; its controls become concrete once the relevant mechanisms and evidence practices are in place.

Neighboring disciplines share a system, not a definition

Performance engineering overlaps several disciplines. Clear boundaries improve collaboration without creating silos.

Security and correctness are constraints across all four. Sustainability adds energy and carbon terms when material. Organizational delivery constrains experiment and migration choices. The best owner for an analysis depends on the decision, but the charter should use shared units and boundaries.

A capacity plan that says “add ten hosts” without service demand and failure headroom is purchasing, not engineering. A reliability objective that counts a correct response after the journey deadline may miss temporal failure. A cost reduction that raises retry amplification can increase total spend. A local optimization that reduces CPU while increasing tail latency may or may not be valuable. The contract supplies the common denominator: useful outcomes under declared conditions.

Maturity is the quality of the decision loop

Maturity does not mean more dashboards or a dedicated performance team. It means the organization can predict, test, observe, and revise behavior with decreasing surprise.

Level 1: reactive firefighting

Performance is reported through complaints or incidents. Teams add capacity, indexes, caches, concurrency, or retries without a preserved baseline. Success means the visible symptom subsides. To advance, define one user journey and capture its workload, boundary, outcome distribution, and saturation signals.

Level 2: component measurement

Services have dashboards and microbenchmarks, but component numbers are weakly connected to end-to-end outcomes. Environments and correctness checks vary. To advance, trace critical paths, reconcile component and journey measurements, and retain reproducible experiments.

Level 3: objective-driven design

Design reviews include workload envelopes, budgets, queue bounds, failure modes, and alternatives. Pre-production validation tests representative modes. To advance, connect deployment gates and capacity/cost decisions to the same contract.

Level 4: lifecycle control

Progressive delivery, rollback, SLOs, workload drift, capacity, and unit economics form a decision loop. Incidents update models and tests. To advance, make governance cross-system and preserve reusable artifacts and terminology.

Level 5: continuously governed performance

Performance constraints participate in portfolio and architecture decisions. Teams forecast uncertainty, test overload and recovery, quantify the economic value of headroom, and retire objectives or mechanisms when value changes. Expertise is distributed through templates, evidence review, and practiced diagnosis rather than centralized heroics.

These levels are diagnostic, not a certification. A team can be mature for a payment path and reactive for an internal analytics pipeline, rationally. Choose the next capability that changes an important decision; do not implement ceremony to appear advanced.

Build the one-page charter

The companion artifact expresses a charter as data and verifies that no required field or evidence gate is absent:

$ node examples/performance-engineering-system-design-handbook/part-01/performance-charter/verify.mjs
performance charter: verified

It deliberately avoids invented target values. Fill them from user research, business consequences, regulatory or safety constraints, empirical baselines, and affordable design options. The structure is the reusable part:

1. Name one journey and one correct unit of useful work.
2. Describe nominal, peak, degraded, and recovery workload envelopes.
3. Declare start and end events and included dependencies.
4. Define indicators by outcome and distribution, including goodput where attempts differ from success.
5. Record correctness, durability, security, cost, and delivery constraints.
6. Allocate evidence to design, implementation, pre-production, and production gates.
7. Name owners and who may approve a revision.
8. List workload, architecture, objective, incident, and cost triggers that reopen the contract.

Now apply it to an existing service. Rewrite five vague statements from a roadmap, incident, or design document using the transformation table. Then produce the charter. Ask a reviewer from another discipline to find one missing workload dimension, one ambiguous boundary, one correctness conflict, and one cost the proposed indicator hides.

Do not optimize yet. The successful output of this exercise is a sharper decision and a measurement plan. The next question is the demand side of the contract: how can work be described precisely enough that a design and experiment transfer to production?

Sources and evidence scope

• Google’s Art of SLOs provides current practitioner guidance for turning stakeholder needs into indicators and objectives. This chapter uses it for objective framing, not as authority for universal targets.
• The chapter’s system-property map, lifecycle, maturity model, discipline boundaries, and charter are editorial frameworks. They make reasoning explicit but do not prove that a proposed system will satisfy its contract.
• The companion JSON contains no empirical benchmark result. Its validator checks structural completeness and ordering of evidence gates; it cannot validate workload representativeness, target value, feasibility, or stakeholder agreement.