Performance Regression Prevention and Continuous Validation

A shared CI runner says a codec change is 3.11% faster: 931 ns/op before, 902 ns/op after. The result is attractive, immediate, and wrong as a release decision. A background neighbor ran during the baseline, the CPU policy was uncontrolled, and there is one observation per version.

The registered experiment tells the opposite story. On an isolated runner with fixed CPU policy, one second of warm-up, and twelve alternating repetitions per version, the baseline median is 812.5 ns/op and the candidate median is 877.5 ns/op. The candidate is 8.00% slower. Maximum within-version coefficient of variation is 0.43%, below this benchmark’s declared 2% stability ceiling. The candidate also allocates six objects per operation against an exact budget of four.

The important result is not that more runs defeated fewer runs. It is that the second system knows what it is allowed to conclude. It names the operation, environment, populations, controls, practical boundary, ownership, and action. The shared-runner number is still evidence—evidence that the environment is not fit to gate this claim.

Continuous validation is the discipline of preserving that distinction while code, dependencies, payloads, query plans, configuration, infrastructure, and workloads change. The goal is not to block every measurable difference. It is to stop expensive, attributable regressions at the cheapest trustworthy boundary and to carry the remaining uncertainty toward environments that can resolve it.

A regression is a broken performance contract

A regression is a harmful change relative to an explicit comparison contract. “Latency rose” is incomplete. State the population, metric, workload, boundary, environment, baseline, uncertainty, practical threshold, and consequence.

Seven common regression classes need different evidence:

The classes can combine. A dependency update may change allocation, which alters GC scheduling, which changes p99 only under concurrency. A schema change may enlarge a payload, invalidate a cached plan, and raise network and parsing work. Tag the first observed mechanism without pretending it is the complete cause.

Correctness is inside the contract. A faster result that drops work, uses a smaller successful population, returns stale data outside policy, lowers answer quality, or moves cost to an unmeasured service is not an improvement. Record logical operations separately from attempts, and pair latency with goodput, errors, quality, and resource demand.

Put the cheapest trustworthy evidence first

The validation pyramid is ordered by attribution and transfer, not prestige. Lower layers are narrow, fast, and stable enough to identify a change. Higher layers include more of the system and workload, so they transfer better but carry more noise and confounding.

Unit cost assertions catch exact changes

Use deterministic assertions when the property is countable and the implementation can expose it without timing:

• number of allocations or copies for one declared operation;
• serialized bytes for a fixed schema fixture;
• requests, queries, storage reads, or messages created by one logical action;
• maximum queue, batch, fan-out, or recursion bound; and
• binary sections or generated-code size when the build is reproducible.

These assertions are powerful because they do not need a statistical test. They are also narrow. “Four allocations per decode” does not establish latency, allocator contention, or live-set behavior. A test that asserts an internal call count can freeze an implementation detail that a better algorithm would legitimately change. Tie exact costs to a user-visible or resource contract and review the boundary when the mechanism changes.

Mercury’s candidate moves from four to six allocations per 4 KiB decode. The budget is four, so this is a merge blocker. The gate does not claim the two allocations cause the entire 65 ns median shift. It says the candidate violates a stable, attributable budget and should not consume more expensive release evidence until the violation is explained or repaired.

Microbenchmarks isolate mechanisms

A microbenchmark should make one mechanism visible while defeating compiler elimination, constant folding, dead-code removal, setup contamination, warm-up effects, thermal drift, and hidden shared work. Record inputs, setup boundary, timer, threads, compiler/runtime, CPU identity and policy, affinity where relevant, counters, repetitions, and raw output.

Interleave baseline and candidate repetitions when environmental drift could bias one block of runs. Randomized interleaving can reduce alignment with time-varying system state, but it does not make a shared runner isolated. Warm up when the decision concerns steady behavior; retain cold-start evidence separately when startup is the decision.

Google Benchmark exposes warm-up, repetitions, randomized interleaving, machine-readable output, run context, counters, and memory-manager hooks. Those features make an experiment describable; they do not choose a correct workload or gate. Go’s current benchstat computes medians, confidence intervals, and A/B comparisons and distinguishes exact units from noisy measures. A team can use either tool and still produce a weak claim if it compares different inputs or treats statistical detectability as operational importance.

Component and integration tests recover interactions

Component benchmarks include enough runtime, storage, network, or concurrency to reveal interactions that a unit test removes. Representative integration tests include realistic payload mixtures, data distributions, pools, caches, downstreams, and correctness checks. They answer questions such as:

• Did lower allocation reduce tail latency under 128 concurrent decoders?
• Did a query-plan change survive production-like statistics and skew?
• Did a payload reduction lower end-to-end CPU, or merely move compression work?
• Did a new timeout preserve useful completions during dependency delay?

The Mercury component packet contains six independent runs per variant and 24,000 successful observations per variant. Successful-request p99 rises from 183 ms to 192 ms, a 4.918% regression against a predeclared 5% practical boundary. It does not block release by itself. It remains on the trend and enters the canary comparison because its estimate is close to the decision boundary and its successful population does not expose overload behavior.

A higher layer must not silently replace a lower failure. Passing an integration test does not excuse a deterministic allocation-budget violation. Conversely, a unit-cost pass cannot authorize release when canary goodput collapses.

Stabilize the comparison before testing a hypothesis

A baseline is a versioned evidence packet, not yesterday’s number. It should include source revision, dependency lock, build flags, fixture/data version, host class, firmware or kernel where material, runtime, topology, warm-up, run order, raw results, and validity interval. Recreate it when any dimension outside the comparison changes.

Prefer paired or blocked comparisons when the same runner, dataset, or time block can evaluate both variants. Separate independent repetitions from iterations inside one process. Ten thousand timed loop iterations may produce one experimental unit if they share warm state and one scheduling episode.

Normalize only against a control with a defensible relationship to the target. Dividing target latency by a colocated reference workload can remove shared host drift. It can also hide a real infrastructure regression if both slow together, or add noise if the reference stresses a different resource. Retain target, control, and normalized values. Never gate only the ratio.

For every signal, ask two independent questions:

1. Is the difference distinguishable from expected run variation under the registered design?
2. Is the difference large enough to matter to the performance contract?

A small p-value can accompany a 0.2% change that has no operational consequence. An 8% change can be important even when a noisy environment cannot estimate it precisely; the right action may be rerun in a better environment, not pass. Predeclare the practical boundary, uncertainty method, sidedness, minimum sample design, and inconclusive action. Do not repeatedly sample until a gate happens to pass.

Control charts and change-point methods compare current behavior with an established in-control process. They help decide when investigation is warranted; they do not identify the responsible commit or prove that the process distribution was stable. A workload or host-class migration should create a named baseline epoch rather than masquerade as a regression.

Make performance policy reviewable as code

A budget file belongs beside the component it protects. It should be small enough for code review and rich enough to prevent interpretation at failure time:

metric: mercury_codec_decode_4k
unit: ns/op
population: fixed 4_KiB contract fixture
environment: isolated-linux-x86-64-v3
comparison: paired_interleaved_medians
minimum_independent_repetitions: 10
maximum_cv_percent: 2.0
practical_regression_percent: 5.0
stage: merge
owner: api-runtime
inconclusive: rerun_once_then_owner_review
baseline_epoch: codec-v7

The file must distinguish a target from an observed baseline. A target of 900 ns/op should not be overwritten to 950 merely because recent main-branch runs cluster there. Updating an epoch requires a reason, evidence link, reviewer, and migration note. Code review should show old policy, new policy, and the change’s predicted cost before results arrive.

Mercury’s exact and measured gates produce different actions:

“Block release” rather than “block merge” is appropriate when a signal needs packaging, representative data, or integration context and can be evaluated before promotion. Do not defer a cheap, exact, attributable violation merely to keep merge velocity high.

Guard artifacts that timing can miss

Some regressions are clearer in structural artifacts than elapsed time.

Query plans. Store a machine-readable plan for representative schema, statistics, parameters, and data distribution. Compare node types, join order, estimated and actual rows, loops, filters, sort/spill behavior, and accessed partitions. PostgreSQL documents that plan costs are planner units rather than wall time, that sampled statistics can vary estimates, and that JSON/XML/YAML formats are intended for programmatic use. Do not freeze every node. Gate a material shape invariant or amplification and rerun after legitimate planner/version changes.

Mercury’s teaching query changes from an index scan over 1,200 rows to a sequential scan over 48,000 rows: 40× row-scan amplification under the registered statistics snapshot. That blocks release while the owner checks statistics, predicates, indexes, parameters, and whether the fixture represents production. It does not prove every production execution is 40× slower.

Binary size. Reproduce the toolchain and split growth by section, symbol, asset, debug data, and package compression. A 2% executable increase may be harmless on servers and severe for cold-started functions or constrained clients. Gate the delivery artifact that users load, not an arbitrary intermediate.

Allocation. Count operations and bytes when possible, then profile retention and allocator/collector cost under concurrency. Fewer allocations can still retain more memory.

Startup. Define start and ready milestones. Process creation, configuration load, cache warm-up, JIT, connection establishment, health readiness, and first successful request are different populations. A service that reports ready early and fails the first request has not improved startup.

Trends make regressions bisectable

A pull-request comparison asks whether one candidate differs from one baseline. A trend asks when the process changed and whether the new level persists. Keep immutable results keyed by source, environment, fixture, and baseline epoch. Plot raw run summaries, not only a smoothed line.

The fixture’s first eight commit medians average 812.75 ns/op; the next eight average 878.125 ns/op. Exhaustively testing valid splits with at least four points per side places the largest mean-level change before commit d3f91c, a 65.375 ns shift. That is a search boundary, not blame. The commit may expose an earlier dependency update, interact with a data change, or coincide with a runner migration.

Bisectability requires more than stored medians:

• old revisions must still build in a pinned toolchain;
• fixtures and data snapshots must be addressable by version;
• the test must finish cheaply enough to evaluate intermediate revisions;
• pass/fail or score rules must handle noisy results; and
• baseline epochs and infrastructure changes must be visible on the same timeline.

Automated change-point detection should open an owned investigation, attach the relevant window, and suggest a bisect range. It should not automatically revert unattributable production signals. For gradual drift, compare cumulative or exponentially weighted methods with abrupt-change detectors and validate false-alarm behavior on the service’s own history.

Canary evidence decides what isolated tests cannot

A canary is a partial, time-limited deployment evaluated against a control. It is useful when topology, production payloads, caches, downstreams, concurrency, hardware variation, or client behavior affect the result. It is not “send 1% and look at a dashboard.”

Register:

• assignment unit and traffic eligibility;
• canary and control versions, capacity, topology, and warm state;
• logical-operation population and correctness filter;
• latency, goodput, error, resource, and quality measures;
• minimum exposure, stop boundaries, and abort authority;
• interference paths through shared caches, queues, dependencies, or clients; and
• rollback and post-rollback recovery observation.

Mercury’s 20-minute teaching window reports p99(successful request latency) of 188 ms for control and 207 ms for canary, a 10.106% increase. Goodput falls from 7,730 to 7,420 correct deadline-compliant operations/min, a 4.010% loss. The registered stop boundaries are +6% latency or −3% goodput, so either signal stops release. The comparison still needs reviewer scrutiny: sample sizes, traffic mix, time alignment, shared-dependency interference, and confidence are outside this compact fixture.

Canary success does not establish every production state. Low exposure may omit hot tenants, large objects, rare plans, region loss, cold caches, or recovery traffic. Continue production trend comparison across the declared envelope, and connect every production anomaly back to a revision, configuration, workload mode, and infrastructure epoch when possible.

Flakes are defects in the evidence system

A flaky performance test consumes trust faster than compute. Repeatedly rerunning until green turns a probabilistic signal into an undisclosed policy. Deleting the test removes visibility but not the performance risk.

Classify the failure:

A quarantine record needs test, risk, owner, reason, start time, expiry, compensating signal, and removal test. Quarantined evidence must stay visible in release decisions. Track flaky-run rate, inconclusive rate, time to repair, and rerun compute. A suite with 99% green status because 30% of tests are silently skipped is not healthy.

Escape analysis repairs the missing gate

When production or canary catches a regression, ask why earlier evidence did not. Do not stop at “tests were insufficient.” Trace the claim through each layer:

The correction should live at the earliest layer that can reliably discriminate the failure. Copying a production dashboard into merge CI usually creates a slow, noisy gate. Reproduce the causal mechanism with a bounded fixture, then retain a later transfer check.

Release criteria and waivers are executable governance

A release decision should enumerate required signals, status, evidence links, unresolved uncertainty, and stop authority. “Performance tests passed” hides the important distinctions.

Use a waiver only when accepting the bounded regression is an explicit business and engineering decision. A copyable record is:

PERFORMANCE REGRESSION WAIVER
Scope: [metric, workload, versions, topology, population]
Failed gate and observed effect: [result, uncertainty, practical boundary]
User/resource impact: [latency, goodput, quality, capacity, cost]
Reason acceptance is necessary now: [specific release constraint]
Owner: [one accountable role/person]
Approvers: [service + performance/release authority]
Compensating control: [limit, alert, canary cohort, feature flag, capacity]
Expiry: [timestamp or release number]
Removal test: [evidence required to close the waiver]
Rollback/stop trigger: [production boundary and authority]
Evidence links: [raw packet, issue, change, dashboard]

No owner or expiry means the budget has silently changed. No compensating control means risk is merely documented. Renewal requires new evidence and an explicit decision; it is not an automatic extension.

For the Mercury packet, the merge decision is already blocked by stable codec and allocation signals. If those are repaired, binary and plan gates still block release. Startup and component p99 alone do neither. Canary latency and goodput independently stop promotion. This ordering prevents a noisy later result from overruling a precise early defect while still testing transfer.

Applied work: design a low-noise critical-library suite

For a critical serialization, scheduling, storage, or cryptography library, produce a suite with five deliberate layers:

1. exact assertions for allocations, bytes, operations, and correctness on boundary fixtures;
2. isolated microbenchmarks across representative sizes and modes, with pinned environment, interleaved repetitions, raw output, and practical thresholds;
3. concurrent component tests that include pools, memory pressure, slow paths, and failure outcomes;
4. consumer integration tests that expose payload, plan, startup, and end-to-end resource effects; and
5. canary/production comparisons stratified by version, shape, tenant class, host class, and workload mode.

For each signal, choose block merge, block release, or neither and defend the placement using stability, attribution, cost, user impact, and recovery time. Then introduce one allocation regression, one payload regression, and one environment perturbation. Confirm that the first two fail at the intended layer and the environment problem becomes inconclusive rather than green or red.

Run the companion packet:

cd examples/performance-engineering-system-design-handbook/part-06/performance-regression-ci
node analyze.mjs
node verify.mjs

Reconstruct the 8.00%, 2.043%, 3.226%, 4.918%, 10.106%, and 4.010% changes. Explain why the 3.11% shared-runner “improvement” is discarded, why the 40× plan scan is a release investigation rather than a universal latency claim, and why the first changed trend commit is not automatically the guilty change. Write a waiver for one failed gate, then reject your own waiver if any boundary field is missing.

Evidence and transfer limits

• Google Benchmark’s user guide documents repetitions, warm-up, randomized interleaving, counters, context, and machine-readable results. It does not supply Mercury’s workload, independence, or gate policy.
• Go benchstat documentation describes median summaries, confidence intervals, A/B comparison, and exact-unit handling. Its default test is not a universal choice for every distribution or decision.
• PostgreSQL’s current EXPLAIN guide explains plan nodes, cost boundaries, sampled-statistics variation, and machine-readable formats. The teaching plan is not a PostgreSQL performance claim.
• NIST’s control-chart guide frames current-versus-in-control comparison and investigation of nonrandom behavior. Production telemetry still needs workload, deployment, and environment annotations.
• Google’s SRE Workbook chapter on canarying defines canary/control evaluation and describes interference and change-related failure modes. Canary design must still match local correctness and traffic authority.
• All Mercury values are simulated teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-06/performance-regression-ci/. Passing the fixture proves arithmetic and declared decision logic, not statistical power or production transfer.

The decision rule is: gate stable, attributable, high-value signals early; move workload- and environment-dependent claims toward integration, canary, and production without allowing later noise to erase an earlier precise failure.

Continuous validation limits how often regressions escape. It cannot prevent every dependency fault, workload discontinuity, or interacting control loop. When the release evidence becomes an acute user-impact event, the next job is no longer benchmark interpretation. It is to stop positive feedback, preserve useful work and volatile evidence, and recover without creating a second incident.