Appendix B — Practical Probability and Statistics

A performance statistic is a compressed claim about a population. Before choosing a formula, write the population, sampling process, experimental unit, workload state, and decision. The same 10,000 latency observations can support a useful estimate, a biased comparison, or a precise answer to the wrong question.

Use this appendix when selecting a summary for a dashboard, comparing two implementations, estimating uncertainty, adjusting for workload, or reviewing a performance claim. It is a method-selection reference, not a substitute for experimental design. Chapter 5 owns the causal treatment of tail behavior; Chapter 53 owns full experiment design. Here the job is to keep the statistical contract intact at the point of calculation.

The object being measured

A population is the set of units about which the claim is intended: for example, successful admitted Mercury API reads in Region A during nominal load. A sample is the observed subset. An experimental unit is the independently assigned or repeated unit that could have received a different treatment: often a host, run, tenant, time block, or deployment cell—not each request emitted within one run.

A random variable X maps an uncertain outcome to a number. A realized observation is x_i. Keep those distinct from an estimate such as the sample mean x̄, and from a configured target. For request latency, define inclusion of failures, retries, cancellations, warm-up, and abandoned work. For throughput, define whether completions, correct results, or objective-compliant goodput are counted.

The expectation is a probability-weighted long-run center:

E[X] = Σ x P(X = x)                  discrete case
sample mean x̄ = (1/n) Σ x_i         estimate from observations

Expectation is not necessarily a typical request. In a mixture of fast hits and slow misses, the mean can fall in a region where almost no request occurs.

Variance measures squared dispersion around the expectation:

Var(X) = E[(X - E[X])²]
sample variance s² = Σ(x_i - x̄)² / (n - 1)
standard deviation s = √s²

The n - 1 denominator is the usual unbiased sample-variance correction under independent, identically distributed sampling. It does not repair dependence, drift, selection bias, or a mixed population.

Covariance measures whether two variables move together on their native scales:

Cov(X,Y) = E[(X - E[X])(Y - E[Y])]
Corr(X,Y) = Cov(X,Y) / (σ_X σ_Y)

Positive covariance between offered load and latency is descriptive, not proof that load caused latency. A rollout, tenant mix, cache temperature, or host class may move both. Correlation is undefined when either variable has zero variance and can hide nonlinear or segmented relationships.

Distribution shapes worth recognizing

Do not begin by naming a textbook distribution. Begin by plotting the empirical data and asking which mechanism can generate its shape.

“Heavy tail” has technical definitions; in field analysis, do not apply the label merely because p99 is large. Show the tail over an adequate range, state the candidate mechanism, and test sensitivity to truncation and censoring. A log-normal-looking body does not establish a log-normal tail.

Quantiles and order statistics

Sort observations as x_(1) ≤ ... ≤ x_(n). These are order statistics. A simple nearest-rank empirical quantile is:

q̂_p = x_(ceil(pn))

Software packages use several interpolation conventions, so record the method when small samples or boundary decisions make it material. p99 = 180 ms means an estimate of the 0.99 population quantile; it does not mean every request has a 99% chance of finishing by 180 ms, nor that the next window will reproduce the value.

Order statistics also explain fan-out. If k independent child latencies share CDF F, the maximum has:

P(max ≤ t) = F(t)^k
P(at least one child > t) = 1 - F(t)^k

Independence is often optimistic. Shared queues, hosts, network paths, and workload bursts create positive dependence. Use the formula as a model with a transfer limit, then compare it with end-to-end traces.

High quantiles need enough observations. With 1,000 observations, nearest-rank p99 is the 990th order statistic and only ten observations occupy the upper 1%. Segmentation, dropped samples, or one time-correlated burst can dominate it. Always report sample count, interval, population, and collection loss beside a tail estimate.

Mixtures: segment before explaining

Suppose a deterministic teaching sample contains eight cache hits and eight cache misses. Hits average 20.875 ms; misses average 112.375 ms. The combined mean is 66.625 ms, while its nearest-rank p50 is 24 ms and p95 is 190 ms. The combined mean describes the mixture’s total time contribution, but it describes neither path’s typical latency.

For component distributions F_j with weights w_j:

F_mix(t) = Σ w_j F_j(t), with Σ w_j = 1
E[X_mix] = Σ w_j E[X_j]

Quantiles do not combine as weighted averages. A rollout that changes the hit rate can move the aggregate p95 even if both conditional distributions are unchanged. Conversely, a slower miss path may be hidden if the hit fraction rises. Keep both the mixture and the mechanism-aligned segments: endpoint, outcome, request class, tenant, region, cache state, payload band, or fan-out band.

Segmentation can also mislead. Slice after seeing a desired result and the chance of a chance finding rises. Predeclare decision-critical segments where possible, preserve the aggregate population, and distinguish confirmatory from exploratory analysis.

Confidence intervals without ritual

A point estimate compresses the observed sample. A confidence interval is produced by a procedure whose long-run coverage is defined under stated assumptions. A 95% interval does not assign a 95% probability to a fixed parameter after the interval has been computed. It says that repeated intervals made by the procedure would cover the parameter about 95% of the time when the model and sampling assumptions hold.

For an independent approximately normal sample mean with unknown population variance, a familiar interval is:

x̄ ± t_(1-α/2,n-1) × s/√n

Requests nested within one run are rarely independent experimental units. Treating a million correlated requests as a million independent replications creates a narrow interval that ignores run-to-run, host-to-host, and time-block variation. Aggregate or model at the assignment unit, or use blocking and dependence-aware methods.

An interval is not an acceptance rule by itself. Compare it with the smallest operationally important effect, the performance objective, and the risk of acting incorrectly. A precisely estimated 0.2 ms improvement may not pay for a complex migration; a wide interval around a potentially severe regression may justify more data rather than acceptance.

Bootstrap intuition and its boundary

The nonparametric bootstrap approximates repeated sampling by drawing, with replacement, samples of size n from the observed units and recalculating the statistic. The distribution of bootstrap statistics estimates uncertainty induced by resampling those units.

repeat B times:
    draw n observed units with replacement
    compute statistic θ̂_b
use the bootstrap distribution for uncertainty

The unit resampled must match the independence structure. Resample paired run differences for a paired experiment; resample hosts or time blocks when requests are clustered within them. Resampling individual requests destroys that structure. Ordinary bootstrap intervals can perform poorly for extremes, tiny samples, highly discrete data, nonstationary series, or statistics with unstable tails. Block bootstrap, parametric models, or a redesigned experiment may be more defensible.

In the companion fixture, eight paired run differences average 3.750 ms in favor of the candidate. A seeded 10,000-resample percentile bootstrap gives [3.000, 4.500] ms. This is a reproducible demonstration of arithmetic, not production evidence: eight pairs cannot establish workload representativeness, and the percentile interval is not privileged over diagnostics of the paired differences.

Effect size: ask how much

A hypothesis test addresses compatibility with a null model. Engineering decisions also require magnitude. Report an effect on a meaningful scale:

• absolute latency change in milliseconds;
• relative change with a declared denominator;
• added goodput at an objective;
• CPU-ms saved per correct result;
• tail-risk or failure-rate difference;
• a standardized effect only when comparison across scales is useful.

For paired observations, compute d_i = baseline_i - candidate_i. The average paired effect is d̄; a paired standardized effect is d̄/s_d. In the fixture it is about 3.219 sample standard deviations of the paired differences. That large standardized value does not replace the more actionable 3.750 ms, and both remain conditional on the eight selected pairs.

Relative effects require stable denominators. Changing from 100 ms to 95 ms is a 5% reduction relative to baseline, but changing from a 1% failure rate to 0.5% is a 50% relative reduction and a 0.5 percentage-point absolute reduction. Report both when either could steer the decision differently.

Regression as workload adjustment

Regression represents an expected response conditional on predictors:

Y_i = β_0 + β_1 X_i + ε_i

Ordinary least squares chooses coefficients that minimize the sum of squared residuals. In a performance study, predictors may include offered load, payload size, request class, host, cache state, or time block. A treatment coefficient can estimate a conditional difference after accounting for specified predictors.

The fixture’s six illustrative points produce a slope of 0.032571 ms/(request/s), or about 32.6 ms per additional 1,000 requests/s over the fitted range. This is interpolation over six designed points, not a capacity law. Extrapolating it through saturation would erase the nonlinear queueing mechanism.

Review residuals and the design before interpreting coefficients. Check nonlinearity, unequal variance, dependence, influential points, interactions, omitted variables, and workload overlap between treatments. Weighted least squares can address known unequal precision, but estimated weights add uncertainty. Regression adjusts only for measured variables in the specified model; it does not manufacture a randomized counterfactual or prove causality.

Sampling, bias, and missing data

Sampling error is only one uncertainty source. Bias can remain even with enormous samples.

Classify missingness by mechanism rather than mechanically deleting rows. If data are missing because overload disabled export, the missingness carries performance information. Report counts before and after filtering, reasons, and sensitivity bounds. Imputation may help a model, but it cannot restore a population never sampled.

Multiple comparisons and stopping

If 20 independent no-effect metrics are each tested at a 5% false-positive rate, the chance of at least one false positive is:

1 - (1 - 0.05)^20 ≈ 64.2%

Dependence changes the exact value but not the decision problem. Predeclare a small set of primary outcomes, separate exploration from confirmation, and adjust when a family of claims must be protected. Bonferroni controls family-wise error by using α/m for m comparisons; false-discovery-rate procedures answer a different question when screening many hypotheses. Neither repairs biased sampling or metric fishing after repeated peeks.

Optional stopping also matters. Rechecking until an interval excludes zero changes the procedure’s error behavior. Use a fixed sample plan or a sequential method designed for repeated looks, and record abort rules that protect correctness and safety separately from statistical stopping.

Which method when

Two field exercises with answer guides

Field — mixture review. A dashboard’s p95 improves after a cache rollout, but miss-path p95 worsens and origin CPU rises. Ask for hit fraction, hit and miss distributions, avoided backend demand per hit, miss amplification, and cold-state behavior. A defensible answer does not accept the aggregate p95. It decomposes the mixture and checks whether a higher hit fraction is masking a more expensive or unstable miss path.

Principal — experiment review. Candidate B appears 3.8 ms faster across eight paired runs, with a seeded percentile interval from 3.0 to 4.5 ms. Before rollout, ask whether pairs share workload and host conditions, whether run order was randomized, whether correctness and tail objectives held, whether eight units span the production envelope, and whether the practical threshold was predeclared. A valid decision might be a bounded canary, more blocked runs, or rejection if the gain is below migration cost. The interval alone does not choose among them.

Field check

Before accepting a statistical claim, ask:

• What population, outcome, boundary, state, and interval does the statistic describe?
• What is the independent experimental or sampling unit?
• Could a mixture, time trend, or missingness mechanism explain the aggregate?
• Are the quantile definition, sample count, and tail observation count visible?
• Does the interval procedure match dependence, pairing, censoring, and the statistic?
• Is effect size reported in an operational unit, with a meaningful threshold?
• Does regression have workload overlap, plausible form, and residual checks?
• Were primary comparisons and stopping rules chosen before results were inspected?
• Can raw or file-backed inputs reproduce the reported arithmetic?
• Which decision changes if the uncertainty or transfer limit is larger than assumed?

Choose the simplest method that preserves the sampling and decision structure. When the structure is wrong, a more elaborate formula makes the answer less honest, not more rigorous.

With population, dependence, and uncertainty made explicit, a queue or capacity calculation can be treated as a bounded model rather than a prediction by typography. Select the operational law by the decision it must inform, and carry its assumptions beside the result.

Sources and evidence scope

• NIST/SEMATECH, e-Handbook of Statistical Methods provides authoritative engineering references for exploratory analysis, confidence intervals, regression, and comparison methods. Its process-measurement examples do not automatically transfer to nested, nonstationary service telemetry.
• NIST, “What are confidence intervals?” states the repeated-sampling interpretation used here.
• NIST, “Bootstrap Plot” explains bootstrap sampling intuition and warns that the method is not suitable for every distribution and tail-dependent statistic.
• NIST, “Least Squares” defines the least-squares criterion; the workload-adjustment guidance above adds performance-specific design and transfer limits.
• NIST, “Comparisons based on data from more than two processes” introduces families of comparisons and adjustment methods.
• The mixture, paired-run, bootstrap, standardized-effect, regression, and multiple-comparison examples are modeled teaching calculations. Run examples/performance-engineering-system-design-handbook/appendices/statistics-and-queueing/verify.mjs for their arithmetic. That fixture supplies no empirical production evidence.