Appendix K — Profiling and Diagnostic Command Cards

Mercury’s checkout p99 has doubled, yet its on-CPU profile contains no expensive function. The team concludes that application code is innocent. That conclusion asks an on-CPU sample to explain wall-clock time it never observed: runnable threads throttled by a quota, lock wait, socket wait, queue residence, runtime pauses, and abandoned downstream work are all missing candidates.

A profile is a sampled view under a capture contract. A heap dump is a sensitive state artifact. A packet trace is a partial observation at one interface. A distributed trace is shaped by instrumentation and sampling. None is a root-cause oracle. The diagnostic job is to choose the least intrusive evidence that can distinguish live hypotheses, preserve enough context to interpret it, and stop when capture risk or invalidity exceeds its value.

Use these cards during design validation, controlled experiments, investigations, and incident evidence capture. They enable you to:

• separate CPU execution, runnable delay, blocking, runtime pause, and end-to-end elapsed time;
• distinguish allocation rate from retention and lock frequency from blocked impact;
• locate storage/network delay without confusing buffering or retransmission with useful work;
• audit distributed traces for missing causality, sampling, and clock limitations; and
• compare baseline and treatment only after matching workload, state, configuration, and capture semantics.

The companion pack at examples/performance-engineering-system-design-handbook/appendices/diagnostic-command-cards/command-cards.json stores seven workflows and versioned examples. Its verifier checks structure, not command safety or tool behavior. Tokens such as PID, INTERFACE, FILTER, and UTC_WINDOW require locally reviewed values: do not paste an example into a system until owners approve target, privilege, overhead, data handling, retention, and stop conditions.

The workflow that precedes every command

Begin on paper. A command chosen before a question usually produces an attractive artifact with an undefined denominator.

1. State the decision. Example: “Decide whether the 80 ms regression comes from added CPU service demand or off-CPU wait, and whether to roll back build B.”
2. Define the boundary. Name operation, success rule, population, topology, workload mode, UTC interval, build/configuration, and the latency/resource objective.
3. List competing hypotheses. Each must predict a different observable. “It is slow” is not a hypothesis; “threads are runnable but CPU-throttled” predicts quota counters and runnable delay without a proportional increase in on-CPU service.
4. Check existing evidence first. Metrics, logs, traces, runtime recordings, scheduler counters, and prior experiment artifacts may already discriminate without attaching a new collector.
5. Select the smallest capture. Bound target, frequency/event threshold, duration, output size, attributes/payload, and privileges. Estimate overhead and storage before collection.
6. Define stop and invalidation rules. Stop on objective harm, event loss, uncontrolled output, policy breach, or overhead beyond budget. Invalidate a comparison if workload, state, clock, build, or capture settings drift materially.
7. Capture a manifest with raw output. Preserve tool/runtime/kernel version, exact command after secret removal, configuration, hashes, timestamps, clock status, errors/lost events, owner, and retention class.
8. Triangulate mechanism. Connect resource evidence to useful work and the path from Appendix J. Samples suggest where to test next; correctness and causal discrimination still decide.

The evidence manifest is part of the result. Without it, a flame graph cannot tell a later reviewer whether 40% means samples, CPU time, wall time, allocated bytes, or a filtered population.

Card 1: CPU time is not wall time

Question. Is elapsed time spent executing, runnable but not scheduled, sleeping on I/O or synchronization, or waiting outside the target process?

Collect useful goodput and latency distribution beside process/cgroup CPU, quotas/throttling, run-queue pressure, and thread states. Start with on-CPU sampling when service demand is the hypothesis. Add scheduler or off-CPU evidence when elapsed time grows without proportional on-CPU samples. Preserve symbols, binary/build IDs, process/thread identity, CPU topology, frequency/power state, and namespace mapping.

For Linux perf 7.0.6, an authorized bounded on-CPU capture can begin as:

perf record -F 99 -g -p PID -- sleep 30

-F 99 requests a sampling frequency; it does not guarantee uniform or zero-overhead observation. -g requests call graphs whose quality depends on unwind information and configuration. Check perf_event_paranoid, container privileges, lost samples, output size, and tool help on the target. Do not interpret symbol width as wall-clock contribution.

For an existing Go go1.26.0 CPU profile:

go tool pprof -top cpu.pprof

Record how the profile was obtained, duration, sample type, workload, and labels. A CPU profile can identify execution demand; it cannot reveal wait it did not sample. If on-CPU demand is stable while runnable delay rises, investigate quotas and scheduling. If threads sleep on one dependency, move to lock or storage/network evidence.

Card 2: allocation, retention, and RSS are different claims

Question. Is memory pressure caused by allocation rate, retained live objects, fragmentation, runtime heap policy, cache policy, mapped files, or a native/external owner?

Align process RSS and cgroup memory with heap committed/live, allocation bytes per useful operation, GC/reclaim, faults, OOM events, and lifecycle phase. Prefer allocation sampling before a full heap snapshot. Allocated bytes answer “where was memory requested?”; a live-heap or dominator view asks “what remains reachable?”; RSS includes more than managed live objects and need not fall immediately after collection.

Analyze an existing Go profile by the sample type the question requires:

go tool pprof -sample_index=alloc_space -top heap.pprof

For JDK 25, a class histogram example is:

jcmd PID GC.class_histogram

Confirm the command supported by that JVM with jcmd PID help, and review its documented impact. Heap artifacts may contain credentials, personal data, payloads, keys, and proprietary structures. Full dumps can pause a process and exhaust local storage. Require explicit privacy, encryption, access, transfer, and deletion controls; otherwise stop at aggregate counters or sampling.

Compare retention only at compatible lifecycle points—often after a named GC condition under the same workload—not arbitrary timestamps. Growth after deployment may be a larger cache target rather than a leak; falling allocation with rising RSS may indicate native memory or release behavior. Preserve that ambiguity until another measurement discriminates it.

Card 3: lock events need owners and impact

Question. Which resource and ownership interval serialize useful work, and is the observed delay lock wait, scheduler delay, safepoint coordination, or I/O inside a critical section?

Count alone overweights frequent short waits. Total blocked time alone can hide one harmed population. Collect waiter and owner stacks, wait and hold distributions, operation/tenant/key labels, runnable state, throughput, and queue age. Look for convoying, unfairness, hot-key concentration, lock-order cycles, and preemption while holding a lock.

A bounded JDK 25 Flight Recorder example is:

jcmd PID JFR.start name=locks settings=profile duration=30s filename=locks.jfr

Use jcmd PID help JFR.start on the exact JVM, choose event thresholds deliberately, and confirm the output path and data policy. The profile template records more than the lower-overhead default and may affect the service. Very short contention may be below thresholds; lowering them increases event volume and perturbation. A stack naming a monitor is a lead, not a redesign instruction: verify whether reducing hold time, sharding state, changing ownership, or bounding concurrency preserves invariants.

Card 4: follow storage and network time across layers

Question. Is delay before issuance, in an application pool, in a kernel/device queue, during device/service completion, in protocol transfer, on retransmission, or at the remote endpoint?

Use the Appendix J path to locate observation points. Correlate useful operations and payload distribution with application pool/queue metrics, block-device request and latency counters, socket state, retransmits/errors, route/failure domain, and remote service evidence. A host with low device utilization can still suffer one saturated shard, queue, path, or remote volume.

Versioned host examples from this environment are:

iostat -xz 1 30       # sysstat 12.7.7: block-device interval counters
ss -tinp              # iproute2 6.19.0: local TCP socket state

Interpret field definitions from the installed manuals; device models, kernels, and multipath layers alter meaning. ss -p can expose process identity and may require privilege. Buffered writes, page cache, I/O schedulers, virtualization, storage services, TLS, multiplexing, and offload separate application timing from wire/device timing.

Packet capture is a last-mile discriminator, not a reflex:

tcpdump -i INTERFACE -s 96 -c 1000 -w capture.pcap FILTER

This tcpdump 4.99.6 example bounds packet count and snap length, but even headers expose addresses, timing, ports, and traffic relationships; filters can fail open through operator error, and encryption does not remove metadata sensitivity. Obtain authorization, test the filter, watch loss and disk growth, and retain only what the question needs. Stop if loss invalidates timing or policy is at risk.

Card 5: align runtime pauses with scheduler reality

Question. Do runtime coordination, garbage collection, compilation, safepoints, quota throttling, host scheduling, or recovery after a pause explain the objective failure?

Capture timestamped runtime events, pause durations, allocation/heap state, runnable time, context switches, quotas, host pressure, runtime flags, warm-up state, and latency exemplars. The pause itself may be short while queue accumulation and cache/capacity recovery dominate the user-visible tail. Conversely, a long request overlapping a pause does not prove the pause contributed its full duration.

For JDK 25:

jcmd PID JFR.start name=runtime settings=default duration=60s filename=runtime.jfr

The default recording is designed for lower overhead than the profile template, but local event settings, event volume, and workload still matter. JFR timestamps events such as execution samples, locks, GC, and runtime activity; available events depend on the JDK/runtime. Pair them with operating-system scheduling evidence rather than treating the JVM as its own machine.

For pidstat from sysstat 12.7.7:

pidstat -w -p PID 1 30

Context switches do not diagnose their own cause. Compare the same runtime version, flags, heap, warm-up, allocation workload, CPU quota, host class, and event configuration. Changing recording settings during an incident is itself a production change.

Card 6: a trace is sampled causality, not complete execution

Question. Where does a logical operation wait, fan out, retry, cross a trust boundary, or continue after its caller deadline?

Select exemplars by operation, outcome, tenant/region class, latency band, and scenario. Retain trace/span IDs, parent-child relationships, span links for asynchronous causality, events, status, sampling policy, resource attributes, redaction configuration, and clock status. Correlate metrics and logs only through approved identifiers.

OpenTelemetry models a trace as spans and causal relationships, but span duration is wall time at instrumentation boundaries—not CPU service. Missing spans may mean unsampled work, propagation loss, instrumentation gaps, exporter loss, or an uninstrumented dependency. Head sampling can underrepresent rare tails; tail sampling can change inclusion based on completed traces; retries may appear as siblings, separate traces, or one overwritten span depending on instrumentation.

Use a backend-specific query only after writing its vendor-neutral intent:

QUERY traces
WHERE service.name = SERVICE
  AND operation = OPERATION
  AND outcome = OUTCOME_CLASS
  AND window = UTC_WINDOW
SELECT trace_id, spans, links, sampling_metadata

Record the OpenTelemetry specification/semantic-convention version and backend query semantics. Trace attributes and baggage can carry sensitive or high-cardinality data and can amplify telemetry cost. Apply allowlists, tenant isolation, retention, and baggage limits before broadening collection. A trace that ends at a queue handoff must not be read as proof that the asynchronous outcome completed.

Card 7: differential evidence needs a controlled denominator

Question. Which mechanism changed between baseline and treatment, and is the change large and certain enough to support the named decision?

Predeclare claim, useful-work denominator, operational threshold, correctness rule, matching/blocking variables, warm-up, steady-state window, independent runs, and comparison method. Retain failed runs. Match operation mix, rate/concurrency, data and skew, topology, runtime/kernel, build/configuration, power state, and capture settings. Normalize by useful work only when flow is conserved and the denominator itself is not the treatment effect.

For matched Go go1.26.0 profiles:

go tool pprof -top -diff_base=baseline.pprof treatment.pprof

A differential view can show sample movement or delta; it does not establish that runs were comparable. Confirm compatible sample type and duration and inspect totals as well as relative shares. A function’s percentage can fall while its absolute service demand rises if total work rises faster. Report effect size, run-to-run variation or uncertainty, raw profiles, manifests, and the correctness/objective result.

Declare the comparison invalid when a causal control differs materially. “Inconclusive” preserves more engineering value than a precise-looking difference between unlike workloads.

Mercury: investigate the missing 80 ms

Mercury observes checkout p99 rise from 142 ms to 222 ms after build B at the same admitted rate. Valid goodput is steady; host CPU average moves from 46% to 48%; the on-CPU profile is broadly unchanged. These observations are scoped to region A, valid retail checkouts, a ten-minute peak window, and the same topology. They do not yet prove equivalence of tenant/key mix or CPU quota behavior.

The team writes three hypotheses:

Existing quota counters reveal throttled intervals beginning with build B’s deployment configuration; scheduler evidence aligns runnable delay with the latency shift. The unchanged on-CPU profile is now consistent with the mechanism: work waits to execute. The team still checks that operation mix and per-checkout CPU demand remain within uncertainty, then tests the configuration correction under controlled load. It does not tune functions absent a CPU-service signal.

If quota evidence had been normal, the next bounded capture would have been contention, not a heap dump or unrestricted packet trace. Evidence selection follows the architecture path and predictions.

Common invalid moves

• Top frame equals root cause. Samples identify where observed events accumulate, not why callers produced them or what alternative preserves correctness.
• Low host CPU means CPU is available. Quotas, affinity, single-thread constraints, hot cores, frequency, and runnable delay can coexist with low averages.
• Allocation equals leak. High churn and high retention are different mechanisms with different evidence.
• Lock count equals impact. Wait/hold duration, population, concurrency, and ownership determine harm.
• Socket or device utilization equals capacity. Queueing, service distribution, sharding, remote limits, and useful bytes matter.
• Trace span equals service time. Spans include waits and exclude uninstrumented work; sampling changes the population.
• Before/after means treatment effect. Drift, warm-up, mix, failures, and capture configuration create confounding.

Practice packets

Choose the missing-time capture. A service has 30 ms on-CPU time and 180 ms elapsed time at p99. Produce four hypotheses for the other 150 ms. For each, name one existing signal, one bounded new capture, a predicted discriminator, and a stop rule. Reject any tool that cannot observe the proposed state.

Design a privacy-safe memory investigation. RSS grows 2 GiB/day in a multi-tenant process. Build a ladder from aggregate counters through allocation sampling to a possible heap artifact. State what each level can and cannot prove, who approves escalation, how sensitive output is protected, and the condition that forbids a dump.

Audit a differential claim. Two flame graphs show function encode falling from 18% to 12%. Baseline ran at 800 valid operations/s; treatment at 1,200 with a different payload mix. Explain why the percentage is insufficient, specify a matched experiment, and define the absolute resource-demand and correctness result needed for rollout.

Pocket evidence checklist

• What decision and competing hypotheses does this capture discriminate?
• Are boundary, useful-work unit, population, scenario, interval, and success rule explicit?
• Have existing signals been exhausted before attaching a new collector?
• Are target, privilege, frequency/threshold, duration, size, and sensitive fields bounded?
• Are overhead, loss, stop, rollback, retention, and deletion rules approved?
• Are clocks, versions, configuration, symbols, namespaces, and exact commands recorded?
• Does interpretation distinguish service, wait, runnable, queue, pause, and end-to-end time?
• Are baseline and treatment matched on causal controls and capture semantics?
• Are raw outputs, stderr/loss counters, manifests, hashes, and failed runs preserved?
• Is the conclusion no stronger than the sampled population and mechanism allow?

The decision rule is: run a diagnostic command only after its observable can distinguish named hypotheses within an approved capture budget; interpret the artifact only with its workload, boundary, versions, sampling, loss, clock, privacy, and comparison manifest, and escalate to more intrusive evidence only when the expected decision value exceeds operational and data risk.

Primary references and transfer notes

• Linux kernel perf security documentation describes access controls and data categories exposed by performance monitoring. Distribution policy, kernel configuration, namespaces, and local authorization still govern a target.
• Linux perf-record manual defines recording options and call-graph modes. Exact availability and overhead depend on the installed perf/kernel, unwind data, event source, and workload.
• Go diagnostics and Go pprof package documentation describe profile types and collection interfaces. Go-version, runtime, labels, profile duration, exposure controls, and workload remain part of the claim.
• Oracle JDK 25 diagnostic tools describes Flight Recorder, heap, thread, and native tools; the JDK 25 jcmd specification documents commands and reported impact. Confirm supported commands on the exact JVM and treat recordings/dumps as controlled data.
• OpenTelemetry traces specification defines spans, links, sampling, and propagation; semantic conventions define common attributes. Instrumentation/backend versions and sampling policy constrain transfer.
• W3C Trace Context specifies interoperable trace context and explicitly addresses privacy and security. Propagation is not proof of full capture or causal completion.

Appendix L defines the terms that these cards intentionally keep separate, including latency, service time, p99, lock freedom, and exactly-once claims.