Accelerators, Heterogeneous Compute, and Energy

Orchid Serve’s team reports a sevenfold kernel speedup. The old CPU implementation spends 27.6 ms in its central embedding computation. The device kernel finishes its equivalent arithmetic in 4.0 ms in a warm microbenchmark. The first end-to-end canary is slower than the CPU path.

Nothing paradoxical happened. The benchmark excluded host preprocessing, transfer into device memory, queueing for a device stream, four launches, result transfer, synchronization, output validation, and the time needed to form a batch. It ran with resident inputs and an already initialized device. The canary received irregular request sizes, crossed a host non-uniform memory access boundary, copied pageable buffers, and sometimes waited behind an export batch. It optimized one service term while creating several waits.

The team changes the unit of measurement from “one kernel invocation” to one accepted request that produces a validated result before its 35 ms deadline. Under the simulated teaching workload in the companion fixture, the selected batch-eight path has 6.5 ms of p95 batch dwell, 2.4 ms of host preprocessing, 1.95 ms of host-to-device transfer, 0.35 ms of launch, 10.8 ms of device execution, 0.35 ms of result transfer, and 1.5 ms of host completion work. The sum is 23.85 ms. Its safe goodput is modeled as 610 successful requests/s after headroom and correctness exclusions. The accelerator is now a plausible system change, not because its kernel is fast, but because the complete path satisfies the declared outcome.

That distinction controls every heterogeneous-compute decision: specialized hardware wins only when enough suitable work reaches it, stays fed, and completes usefully after transfer, scheduling, failure, and operational costs are charged back.

Specialization changes the path, not the conservation of work

Accelerators improve particular work by providing many parallel execution lanes, high-bandwidth local memory, specialized arithmetic, fixed-function data paths, or some combination. A graphics processing unit can apply regular operations across many independent elements. A tensor processor can devote silicon and data movement to matrix-like operations. A cryptographic engine can implement a bounded algorithm in a pipeline. A compression engine can retain dictionaries or execute codecs without consuming the application’s general-purpose cores. A network processor or smart network interface can classify, transform, route, or move packets near the device boundary.

These advantages do not make work free. They move work among resources and introduce new contracts:

• the host must prepare descriptors, buffers, commands, and completion handling;
• bytes must reach the execution unit or already be resident there;
• requests wait for batches, queue pairs, streams, memory, and scheduling slots;
• device-local memory has allocation, fragmentation, residency, and eviction behavior;
• failures can strand in-flight work at a boundary with ambiguous completion;
• software must preserve a portable, recoverable result contract; and
• power and cooling constrain sustained delivery even when a short run reaches a higher rate.

The fast path has resident data, compatible layout, an available stream, enough parallel work, overlapped copies, a warm runtime, and one validated completion. A slow path may stage pageable memory, reformat data, compile or load a kernel, allocate device memory, wait for a batch, or fault data across an address boundary. A failure path can encounter an enqueue rejection, device reset, unsupported operator, out-of-memory condition, lost completion, or invalid numerical result. The recovery path reloads code and state, warms caches, rebuilds memory pools, drains or replays work, and protects the CPU fallback from the sudden transfer of offered load.

Device utilization alone cannot distinguish those paths. Keep counters for offered originals, admitted originals, attempts, batch dwell, queue age, host and device service, bytes transferred, launch count, memory occupancy and allocation failures, correct completions, deadline misses, retries, fallback work, power, and reset/reload state. Normalize them to the same operation population.

The host-device pipeline determines elapsed time

A naive request timeline is serial:

[ L = Q_b + H_{pre} + T_{in} + L_k + K + T_{out} + H_{post} ]

where (L) is request latency, (Q_b) is batch or device-queue dwell, (H_{pre}) and (H_{post}) are host service, (T_{in}) and (T_{out}) are transfer times, (L_k) is launch/scheduling overhead, and (K) is device execution. Every term is in seconds or milliseconds for the same request population. Summing their p95 values would not generally produce request p95; the Orchid values are an intentionally coherent teaching path used to expose the budget, not a percentile-composition rule.

Throughput can improve when consecutive batches overlap. While batch (n) computes, one copy engine may transfer batch (n+1), and another may return results for batch (n-1), if the hardware, runtime, memory type, dependency graph, and streams allow it. The steady pipeline cycle is then bounded approximately by the slowest overlapped stage plus non-overlapped launch work:

[ C_{pipe} \approx \max(T_{in}, K, T_{out}) + L_k ]

For Orchid’s selected point:

[ C_{pipe} = \max(1.95, 10.8, 0.35)\ ms + 0.35\ ms = 11.15\ ms ]

The optimistic steady ceiling is therefore:

[ X_{pipe} = \frac{8\ requests}{0.01115\ s} = 717.49\ requests/s ]

That is not safe goodput. Host work, variability, result failures, allocation, sharing, thermal state, headroom, and recovery reserve reduce it to the fixture’s modeled 610 successful requests/s. The ceiling is useful because it reveals the current cadence term. It must not be reported as observed capacity.

Overlap is conditional. A blocking transfer can serialize streams. Pageable host buffers may require hidden staging. Two logical streams may share one copy engine. A data dependency can force the kernel to wait. A unified address API can simplify ownership while still migrating pages or synchronizing at runtime. “Asynchronous” means that submission returns before completion under a defined API; it does not mean that transfer and compute overlap or that the caller no longer owns a lifetime obligation.

Inspect an aligned device timeline rather than inferring overlap from high utilization. Mark buffer preparation, enqueue, actual copy, kernel ready, kernel run, result copy, synchronization, and caller completion. If the host submit thread is late, optimizing device code cannot close the gap. If the copy engine is continuously busy while compute has bubbles, reduce or reuse transfer. If launches dominate small work, fuse operations, retain a resident service, or leave the work on the CPU.

Operational intensity separates memory and compute ceilings

The Roofline model asks how many operations a kernel performs per byte transferred through the limiting memory interface. Let:

• (W) be useful operations for one kernel, operations;
• (B) be bytes transferred between the relevant device memory level and compute, bytes;
• (I = W/B) be operational intensity, operations/byte;
• (P_c) be sustained compute ceiling, operations/s; and
• (M_b) be sustained memory bandwidth, bytes/s.

The performance bound is:

[ P \le \min(P_c, I M_b) ]

Orchid’s teaching kernel performs 36 giga-operations while moving 3.6 decimal GB through the modeled device-memory boundary:

[ I = \frac{36\ Gop}{3.6\ GB} = 10\ operations/byte ]

The declared sustained ceilings are 15 tera-operations/s and 900 GB/s. Their ridge point is:

[ I_{ridge} = \frac{15{,}000\ Gop/s}{900\ GB/s} = 16.67\ operations/byte ]

Because 10 is below 16.67, the teaching kernel reaches the bandwidth roof first. Its bandwidth-limited ceiling is 9 tera-operations/s, giving a four-millisecond lower bound for 36 giga-operations. The 4.8 ms fixture observation is consistent with that bound and an effective 7.5 tera-operations/s. It does not prove memory bandwidth is the only stall; profiles and counters must show the bytes and activity at the named boundary.

Roofline analysis prevents two expensive mistakes. Buying more peak arithmetic does not raise the diagonal memory roof. Reducing arithmetic while preserving the same byte traffic can lower operational intensity and leave elapsed time unchanged. Conversely, a compute-bound kernel may not benefit from reducing a copy that is already hidden. Measure the actual operations and memory traffic for the representative input shape rather than copying peak numbers from specifications.

The model also has boundaries. Cache reuse changes which byte interface matters. Sparse operations can execute extra index work and irregular loads, so nominal zero count is not sufficient intensity. Tensor or vector units may require supported shapes and precision. A mixed kernel can move between roofs by input size. Plot important kernels separately and retain the host-transfer roof outside the device Roofline; device memory bandwidth does not pay for PCIe-like movement.

Layout, precision, sparsity, and fusion change different terms

Data layout determines whether adjacent lanes consume adjacent useful bytes, whether loads coalesce, whether padding is processed, and whether host conversion precedes execution. An array-of-structures convenient to the CPU can force strided device access. A structure-of-arrays may improve coalescing but raise conversion and ownership costs elsewhere. The decision must include the canonical storage layout and every boundary conversion, not only the final kernel.

Lower precision can reduce bytes and increase specialized arithmetic rate. It can also change rounding, overflow, convergence, ranking, or model quality. Orchid accepts a precision mode only when its versioned validation corpus reaches at least 0.999 application-defined agreement; the simulated selected path records 0.9994. That number is not a universal quality metric. A payment calculation, scientific solver, search ranker, and approximate embedding each require a different invariant and tolerance. Preserve a high-precision control, segment error by input shape, and test failure cases near thresholds.

Sparsity saves work only when the hardware and representation skip it cheaply. Irregular indices, metadata, load imbalance, and low reuse can cost more than dense execution. Record executed operations and transferred bytes, not theoretical nonzero counts. Test realistic sparsity patterns; random sparsity and block sparsity can drive entirely different data paths.

Operator fusion removes intermediate writes, reads, synchronization, and launches. In the fixture, a proposed fused path reduces modeled device traffic from 3.6 GB to 2.6 GB while useful work changes from 36 to 34 giga-operations. Operational intensity rises from 10 to 13.08 operations/byte. The kernel remains below the 16.67 ridge, but the bandwidth lower bound falls. Fusion is not free: larger kernels may consume more registers or local memory, reduce resident work, duplicate computation, make compilation slower, and limit portability. Compare the whole timeline and result, not the number of operators.

Occupancy is likewise a means, not an objective. It describes resident execution capacity relative to a device limit under a particular launch configuration. Higher occupancy can help hide latency, but register pressure, shared memory, instruction mix, and independent work determine whether it improves throughput. A lower-occupancy kernel with better reuse can beat a higher-occupancy kernel that moves more bytes. Use occupancy to explain an execution constraint, never as a substitute for useful completion rate.

Batch size has a latency frontier, not a maximum

Batching amortizes launch, descriptor, synchronization, and fixed host work. It may improve device occupancy and memory efficiency. It also waits for compatible items, consumes memory, extends cancellation, and lets a large or long request delay others.

The simulated Orchid frontier is:

The table’s pipeline ceiling is modeled batch/cycle, not observed goodput. Batch eight is the smallest point whose safe-goodput model reaches 580/s while p95 remains below 35 ms. Batch sixteen has a higher ceiling and is still the wrong interactive policy.

Form batches by compatible shape and remaining deadline. Put a maximum count, maximum bytes or tokens, and maximum dwell on every batcher. Flush early when the oldest item approaches its budget. Separate latency classes when a long item would pad or dominate the rest. Cancellation should remove work before dispatch when safe; after dispatch, decide whether to mask the result, compact a future batch, or let device work finish. Repacking an in-flight device command may cost more than finishing it.

A counterexample matters here: batching can lower throughput. If a larger batch exceeds device memory, spills an intermediate, changes an algorithm, destroys cache reuse, or causes repeated allocation, its per-item cost can rise. The batch curve must therefore measure bytes, launches, allocation, device service, host service, correctness, and queue dwell at each point.

A shared accelerator is a scheduler and memory system

Sharing increases utilization when workloads have complementary gaps, but it imports the same questions Chapter 15 applied to virtual CPUs: what is reserved, weighted, capped, queued, observable, and recoverable?

Three broad policies expose different failure behavior:

Device memory is often the first shared constraint. Model code, weights, buffers, caches, temporary workspaces, allocator fragmentation, and reload reserve. A scheduler can report idle arithmetic while no request fits the remaining contiguous or allocatable memory. Oversubscribing residency may trigger eviction or reload, moving the bottleneck to host transfer and making tails bimodal.

Fairness should use estimated work, bytes, memory residency, and deadline—not request count when shapes differ. One export with 20 times the service demand is not equivalent to one interactive request. Bound per-tenant in-flight work and queue age. Preserve admission capacity for health, control, and recovery operations. If the runtime cannot preempt a long kernel at the needed granularity, partition work or refuse that workload rather than promising scheduler fairness that the device cannot enforce.

Observe enqueue acceptance, queue pair or stream, scheduling delay, device service, memory residency, faults, evictions, preemptions when supported, cancellations, and correct completions. A tenant p95 can degrade while fleet utilization and aggregate p95 remain healthy. The promise and measurement boundary must match.

Offload engines have the same queue contract

Heterogeneous compute is broader than GPUs. CPU instructions can accelerate cryptography, checksums, and vector work without crossing a discrete-device queue. Lookaside engines accept descriptors for cryptographic or compression operations and expose asynchronous enqueue/dequeue behavior. Inline engines transform packets or storage data on a path. Smart network interfaces can execute steering, encapsulation, encryption, or bounded application functions near the network.

The adoption test remains unchanged:

1. name the exact operation and correctness result;
2. measure current CPU service and whether it is a system constraint;
3. include descriptor preparation, buffer mapping, transfer, enqueue, polling or interrupt, and completion work;
4. bound in-flight operations and queue age;
5. define unsupported input, overflow, error, reset, and software-fallback behavior; and
6. compare end-to-end goodput, latency, power, and cost under normal, skewed, failed, and recovering states.

Stateful compression illustrates a hidden boundary. Chunks in one stream may depend on prior history, so distributing them across queue pairs is not equivalent to stateless operations. A reset can lose history even if input bytes remain. Cryptographic offload must preserve key custody, algorithm support, authentication failure, ordering, and side-channel requirements. Network offload can save host CPU while making observability or upgrade behavior harder. “The NIC handles it” does not remove ownership.

Small work often belongs on the CPU. Crossing a queue, polling for completion, and synchronizing a result can exceed the saved service. CPU instructions also share the application’s scheduling and memory locality, which can make their lower peak rate the better request path. Offload only after a representative size curve finds the crossover and a fallback test proves semantics.

Energy belongs to a successful system outcome

Power is a rate; energy is accumulated power over time. For a steady interval with integrated whole-system energy (E) joules and (N_s) successful SLO-compliant operations:

[ e_s = \frac{E}{N_s} ]

When average system power (P) watts and safe goodput (X_s) operations/s are stable over the same interval:

[ e_s = \frac{P}{X_s} ]

The CPU baseline uses 235 W at 122 safe successes/s, or 1.926 J/success. Orchid’s selected path uses 430 W at 610/s, or 0.705 J/success. The accelerator consumes more instantaneous system power and less modeled energy per successful operation. Measuring only device board power would omit host CPUs, memory, transfer, cooling allocation, and idle reserve. Using thermal design power would not measure either interval.

Power caps move a frontier:

The lowest joules per operation does not automatically win if it violates required goodput. The highest throughput does not win if its extra energy and cost buy no valuable outcome. Plot correct, deadline-compliant results against energy and cost.

Short runs can mislead. Orchid’s simulated goodput is 666/s at minute five and 610/s at minute thirty, with thermal settling declared after twelve minutes. Possible mechanisms include power control, temperature, clock behavior, memory temperature, cooling, or another shared limit; the fixture does not assign a physical cause. Record power, clocks, temperature, throttling signals, workload, fan/cooling policy, and performance until sustained state is established. Then repeat at the failure and ambient conditions relevant to operation.

Energy also belongs in recovery. Device reload, model or table transfer, recompilation, replay, and fallback consume energy while goodput is low. A design that is efficient only while warm can be economically and operationally poor under frequent deployment or eviction.

Fallback must preserve meaning before capacity

Portability is not one source file compiling for two targets. It includes numerical semantics, layout, supported operations, memory behavior, asynchronous completion, cancellation, error mapping, telemetry, deployment, and performance at the required scale. A portable API can hide important differences; a vendor-specific kernel can expose them while increasing migration cost.

Define one result contract above the implementation paths. Orchid’s fallback returns the exact declared result variant, disables optional enrichment, and admits at most 100 requests/s against 110/s of simulated CPU fallback goodput. It does not pretend that 610/s remains available. Callers receive early overload responses for excess work rather than waiting behind an impossible queue.

Device loss needs an explicit sequence:

1. stop or fence new submissions to the failed context;
2. classify in-flight operations as completed, failed, or ambiguous from durable request identity;
3. prevent blind duplicates where effects are not idempotent;
4. transfer admitted work to reserved fallback capacity within remaining deadlines or reject it;
5. reload code and resident state—the fixture models 74 seconds;
6. validate output and sustained performance before restoring traffic; and
7. drain fallback gradually so the recovering device does not create another batch or cache surge.

If no CPU or alternate-device implementation can preserve the invariant, say that the service is unavailable for that operation. A numerically different “best effort” result is a product behavior change, not a transparent fallback.

Adoption economics include the operational tax

For a steady modeled interval, cost per million safe successes is:

[ C_{1M} = \frac{C_h}{3{,}600X_s} \times 1{,}000{,}000 ]

where (C_h) is hourly cost units and (X_s) is successful SLO-compliant operations/s. The fixture gives 1.321 units/million for the CPU path and 0.888 for the selected accelerator path. These are invented units. They demonstrate the denominator, not a purchase recommendation.

Add engineering and operational costs: kernel and portability maintenance, device-aware scheduling, scarce-device reservation, larger images, initialization, memory fragmentation, monitoring, security review, failure reserve, data movement, and slower rollout. Account for utilization by workload class and time; a cheap full device can be an expensive mostly idle fleet. Include the cost of CPU fallback or alternate devices required for failure.

Start with the CPU path. It remains the sound choice while work is small, irregular, branch-heavy, latency-sensitive, or still changing, because it avoids submission and transfer and usually provides the broadest recovery path. Leave it only after an end-to-end size curve shows both that CPU service is the constraint and that enough parallel, regular work exists beyond the crossover.

Beyond that point, choose the narrowest specialization that relieves the measured term. An asynchronous crypto, compression, or network engine can save host service for regular work, but only after its queue age, CPU saved, correctness vectors, reset behavior, and software fallback have been measured across the size curve. A dedicated accelerator can provide predictable capacity when steady work amortizes transfer and isolation is worth its reserve; its case rests on the safe-goodput distribution, cost per success, and an N-minus-one test that includes reload. A shared partition trades some flexibility for bounded memory and capacity, so tenant tails, memory headroom, and partition-loss recovery must justify the capacity stranded under skew.

Hybrid routing is appropriate only when input shapes have different measured crossover points. It adds a classifier whose errors consume latency or send work to the wrong capacity pool, and it creates mixed-version and telemetry obligations. Use it when per-shape demand and a live fallback campaign show that the paths preserve the same result—not merely because two implementations exist.

The conditional rule is strict: adopt specialized compute only when the measured whole operation has enough parallel, regular work to amortize transfer, launch, scheduling, memory, and operational complexity under the actual objective—and when failure can transfer to a bounded correct outcome.

Field exercise: locate the binding boundary

Reproduce the fixture at examples/performance-engineering-system-design-handbook/part-02/heterogeneous-compute/ and answer:

1. Why is the batch-eight 717.49/s pipeline ceiling not the 610/s safe-goodput result?
2. Which term binds the overlapped cycle, and which evidence would prove that term is really active?
3. Recompute the Roofline intensity, ridge, bandwidth roof, and four-millisecond bound with units.
4. Explain why adding peak arithmetic cannot improve that bound unless another term changes.
5. Compare CPU and accelerator joules/success and cost/million success.
6. Choose the 360 W cap only if you can change one objective; state the product and capacity consequence.
7. Identify one condition under which fusion raises operational intensity yet makes end-to-end latency worse.

A strong answer separates modeled ceilings from observed or safe results, does not compose percentiles, and asks for aligned transfer/device/host timelines before naming the constraint.

Principal drill: approve or reject heterogeneous serving

Orchid must serve three input shapes, one of which uses eight times the memory and has a 2% arrival share. Two tenants require isolation. A region must survive one device pool loss. Precision reduction passes aggregate quality but fails a rare threshold class. The lowest power cap misses peak goodput, and the CPU fallback supports only 18% of normal traffic.

Produce an adoption record containing:

• the unit of work, shape distribution, objective population, and quality invariants;
• host/device paths for normal, large-shape, device-loss, and reload states;
• a batch policy with count, byte, shape, dwell, deadline, and cancellation bounds;
• memory residency, fragmentation, tenant scheduling, and admission controls;
• Roofline and transfer measurements by shape;
• full-system energy and cost per correct SLO-compliant outcome after thermal settling;
• the precision exception path and its capacity effect;
• N-minus-one, ambiguous-completion, fallback, reload, and traffic-restoration tests; and
• a threshold at which the operational tax makes CPU-only or another architecture preferable.

There is no required hardware answer. Approve only if the evidence closes the whole-path and failure contracts. Reject an accelerator proposal that depends on average shape, peak specifications, or an unexercised fallback.

Evidence and transfer limits

• Williams, Waterman, and Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures” establishes the operational-intensity bound used here. A Roofline ceiling is not an observed end-to-end result.
• CUDA C++ Best Practices Guide 13.3 documents current CUDA transfer, overlap, memory-access, execution-configuration, and occupancy behavior. Those mechanisms are implementation-specific and must be checked against the deployed toolkit and device.
• AMD HIP performance guidelines document current HIP memory coalescing, reuse, resource, and occupancy considerations. Source portability does not imply equal execution behavior.
• OpenMP API specifications provide the current OpenMP 6.0 target/offload portability contract and errata. An API contract does not supply a performance result or device availability.
• DPDK Cryptography Device Library and Compression Device Library show queue-based asynchronous offload, capabilities, state, and operation-status surfaces. Exact drivers and releases require their own validation.
• Linux power capping framework documents a current host power-cap control surface; it does not by itself measure the full service or explain a performance change.
• MLCommons power measurement documentation illustrates calibrated external whole-system power measurement for a specific benchmark run and system under test. Orchid’s numbers are not MLPerf results.
• Every Orchid value is simulated teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-02/heterogeneous-compute/. The fixture checks arithmetic and decision boundaries only.

Part II has now supplied the machine mechanisms beneath service demand: execution, memory, scheduling, runtimes, storage, networks, virtual resources, and specialized devices. The next design task is to compose them without losing locality or introducing avoidable coordination. Chapter 17 begins with data layout and algorithm choice, where bytes touched often matter more than nominal operation count.