ML and AI Inference Systems

What does one inference request cost?

There is no useful single answer. Orchid Serve receives four requests at the same instant:

An admission system that counts all four as one request gives them equal price despite unequal accelerator time, memory lifetime, transfer, batching compatibility, and quality consequence. A scheduler that uses only input tokens still misses iterative output work. A utilization graph can show 96% while urgent work expires behind a long batch. Even “tokens per second” is incomplete when prefill and decode have different arithmetic intensity, memory traffic, and parallel efficiency.

The useful unit is a resource-and-outcome record:

request identity and class
  tenant, model and version, operation, privacy class, deadline

shape
  items/samples, input tokens, maximum output tokens, modalities,
  sequence state and batching compatibility

predicted demand
  host CPU, transfer bytes, accelerator-ms by phase, peak and live memory,
  external feature/vector work, output validation, energy or price

outcome
  accepted/rejected, first-result and completion latency, validated quality,
  charged work, fallback or approximation, failure/recovery state

That record is estimated before execution and corrected from observed work afterward. It makes the governing rule operational: schedule and price inference by actual compute and memory demand plus the required quality outcome, not by request count. Batch only inside declared deadline and fairness bounds.

Serving mode changes the completion boundary

Inference is not synonymous with one synchronous RPC.

Online inference places the model on a user-visible critical path. Queue dwell, feature retrieval, preprocessing, transfer, execution, postprocessing, policy checks, and response transport all consume one deadline. Admission must reject before useful completion becomes impossible.

Streaming inference returns incremental results. Time to first result and inter-result gap become separate objectives from total completion. State persists across scheduling turns, so preemption may free compute without freeing key/value memory. A transport write is not user consumption, and a partial generated response may have product or billing consequences.

Asynchronous inference durably accepts work and returns an operation identity. Completion can be polled or delivered later. The queue may absorb bursts, but it needs age, storage, retry, cancellation, deduplication, and result-retention bounds. “Accepted” means the durable job contract was met, not that inference ran.

Batch inference reads a declared data snapshot and produces a versioned output set. Throughput and cost often dominate per-item latency, but skew, stragglers, accelerator allocation, checkpoint/restart, and quality validation still determine useful completion. A million-item job that finishes quickly with the wrong feature snapshot is not goodput.

One product can use all four modes. Keep their identities and objectives separate so background exports cannot consume interactive failure reserve and streaming sequences cannot hold memory without a lifetime charge.

Follow the full critical path

Orchid Serve’s online path is:

authorize and classify request
  -> retrieve versioned features / embeddings when required
  -> tokenize, decode media, normalize, and validate shape
  -> route to a compatible warm model replica
  -> wait in a deadline- and memory-aware scheduler
  -> transfer inputs or bind resident buffers
  -> prefill / main model execution
  -> iterative decode or downstream stage
  -> postprocess, policy-check, and validate result
  -> record model/version/quality evidence and respond

Every arrow can queue. Feature service fan-out can dominate a fast model. Tokenization can saturate a shared CPU pool. Pageable buffers can add hidden copies. A device can be busy while its host feeder is late. A model replica can have compute headroom but insufficient contiguous or paged cache capacity for a long sequence. Output moderation or schema validation can become the tail after generation ends.

Measure stage distributions with a shared request identity, but keep authority clear. A trace is observational evidence. The accepted model version, feature snapshot, policy version, and durable side-effect identity determine the application result. Do not log raw prompts, images, embeddings, features, or outputs merely because a trace makes that convenient.

Model execution spends compute, bandwidth, memory, and transfer

Model parameter count provides a first memory term, not a device-fit proof. Fourteen billion parameters at two bytes each occupy 28 billion bytes, about 26.08 GiB, before runtime state, kernels, workspaces, activations, communication buffers, graph captures, allocator reserve, and sequence cache. The fixture gives an 80 GiB teaching device a 6 GiB runtime/workspace reserve. If live sequence state costs 128 KiB per token, the remaining 47.92 GiB holds at most 392,584 live tokens by arithmetic. Fragmentation, block granularity, safety reserve, and transient peaks reduce that ceiling.

Arithmetic intensity asks how many useful operations occur per byte moved at a named boundary. Prefill over a large input may expose parallel compute; one-token decode repeatedly reads model state and attends over a growing sequence, often producing a different bottleneck. Quantization can reduce bytes and change supported kernels, but the decision must include calibration population, output quality, overflow behavior, transfer, conversion, and operational portability. Sparsity matters only when the runtime and hardware exploit the actual structure.

Record memory by lifetime:

• resident model weights and runtime state;
• per-replica workspace and peak activations;
• per-request input/output buffers;
• per-sequence cache that grows and persists between decode turns;
• prefix or embedding caches with tenant and privacy scope;
• temporary state during model load, version overlap, and failure recovery; and
• allocator reserve and fragmentation loss.

A model that “fits” alone may fail while old and new versions overlap. Loading from object storage also consumes network, host memory, CPU, device transfer, and initialization time. Cold loading is a recovery workload and a rollout capacity event, not an exceptional first request.

Parallelism moves the bottleneck

Data parallelism places complete model replicas on several workers and sends different requests or batches to each. It is simple when one replica fits and traffic is sufficient, but duplicates weights, makes cache locality a routing concern, and can strand long sequences on one replica.

Model or tensor parallelism splits operations and parameters across devices. It permits a model that does not fit on one device and can reduce one execution’s compute time, while adding collective communication and a larger failure domain. Tail latency depends on the slowest participant and interconnect path.

Pipeline parallelism assigns model stages to devices and overlaps microbatches. It can increase throughput, but bubbles, unequal stages, activation transfer, and one slow microbatch determine cadence. An interactive request may traverse several device queues.

These mechanisms compose. A pipeline group may be replicated for data parallelism; a stage may use tensor parallelism. Capacity then belongs to a topology, not a count of devices. Record which links, host sockets, failure domains, and model versions form one schedulable unit. Never route half a tensor-parallel group to unrelated work and still count the group as available.

A deadline-aware scheduler spends slack explicitly

For request i, define remaining slack at scheduler arrival as:

[ S_i = D_i - E_i - R_{up,i} - R_{exec,i} - R_{fail,i} ]

where D is the end-to-end deadline, E elapsed time, and the three reserves cover unfinished upstream work, predicted execution/response, and bounded failure or variance. A positive S is not permission to wait until zero; prediction error and queue variability need margin.

The teaching request has a 200 ms deadline, has already spent 45 ms, reserves 12 ms for feature/preprocessing work, 60 ms for execution and response, and 15 ms for variance or one bounded recovery action. Its maximum arithmetic batch wait is 68 ms. The scheduler should normally dispatch earlier because the estimate is a limit, not a target.

A useful scheduling record is:

request: orchid/interactive/7f2
model: full-v18; allowed fallback: compact-v11
deadline: 200 ms end to end; elapsed: 45 ms
shape: 256 input tokens; <=64 output tokens
demand estimate: 56.8 accelerator-ms; 8.0 MiB peak request state
compatibility: text, full-v18 tokenizer, privacy cell eu-restricted
slack: 68 ms after reserves
queue policy: earliest feasible completion within tenant reservation
admission result: full-v18; batch dwell capped at 20 ms
post-run charge: observed phase work, live-token-ms, validated outcome

Earliest-deadline-first alone can choose work that cannot finish or starve large jobs. Shortest-predicted-job-first can improve mean latency while violating tenant and deadline guarantees. Priorities can starve lower classes. Use feasibility plus hierarchical reservations: reject work whose predicted completion misses its deadline; schedule feasible work by deadline/age inside tenant and class shares; cap consecutive decode turns; and reserve capacity for recovery and control traffic. Age estimates and penalize chronic underprediction rather than trusting callers.

Batching is a constrained packing problem

Dynamic batching amortizes launch, weight reads, and fixed host work when requests can execute together. Its costs are queue dwell, padding, synchronized completion, memory peaks, and fairness.

The fixture places seven 256-token inputs beside one 4,096-token input. Useful input is 7 × 256 + 4,096 = 5,888 tokens. Padding all eight to the longest shape executes storage or arithmetic over 32,768 token positions, a 5.565× amplification at that boundary. Eight requests is therefore not an eight-request batch in any predictive sense. Bucket compatible shapes, use a runtime/model that supports ragged representation where correct, or dispatch the long input separately.

Sequence length is only one compatibility dimension. Include model/version, modality, precision, tenant isolation, adapter, output constraints, privacy region, deadline, and statefulness. A larger compatible batch may still be wrong if its oldest request spends too much slack.

Generative serving adds two phases. Prefill consumes a new input; decode advances live sequences iteratively. Request-level batching waits for the longest sequence and leaves completed slots idle. Iteration-level or continuous batching can remove finished sequences and admit new work between steps. That improves packing but requires per-sequence state ownership, fair turn allocation, cancellation, memory accounting, and output-order semantics. Preemption that swaps cache to host storage trades device memory for transfer and tail latency; recomputation trades memory for compute.

Measure batch-size and padded-token distributions, queue dwell by class, useful versus padded tokens, accelerator phase time, live-token memory, preemptions, cancellations, and deadline misses. High average batch size is not a success metric if long requests inflate padding or interactive requests wait behind batch formation.

Upstream data can be the real model path

Feature retrieval and vector access need version, freshness, and fan-out contracts. An online fraud model may require account, device, and transaction features. A recommendation cascade may retrieve candidates from an embedding index before ranking. Those reads create network, cache, storage, and consistency work before the accelerator starts.

Name each feature’s authority, event-time or snapshot boundary, maximum age, missing-value policy, and model compatibility. A model trained with feature definition v12 must not silently consume an incompatible online v13. Prevent one inference from opening hundreds of independent RPCs; precompute when the freshness contract allows, colocate repeated data, batch vector reads, and bound candidate counts. Cache keys must include tenant, authorization-relevant scope, feature/model version, and data generation.

Preprocessing can be CPU-, memory-, or decode-bound. Separate image decode, tokenization, normalization, and serialization pools from device dispatch so their saturation is visible. Backpressure should reach admission. Filling an accelerator queue while the feature service is already late converts an upstream incident into expensive expired work.

Cascades trade quality, latency, and cost

A multi-stage system may use rules before a model, a small model before a large one, retrieval before ranking, or an early-exit head when confidence is sufficient. The route is a decision system and needs its own quality evaluation.

Plot viable configurations on a quality-latency-cost frontier for a named task population. A point is dominated if another has at least as much validated quality, no worse latency, and no greater cost. The frontier changes by language, tenant, request length, risk class, and load. Average quality can hide a subgroup regression just as average latency hides a tail.

Early exit requires calibrated evidence that the exit criterion predicts acceptable outcomes. A smaller fallback needs a task-specific floor and prohibited classes. Orchid’s fixture uses a modeled full-model quality score of 0.92 and compact-model score of 0.86 for an eligible evaluation population whose minimum is 0.84. Those numbers merely make the routing arithmetic explicit. They do not authorize fallback for safety-critical, regulated, or out-of-distribution work.

Return provenance: selected model/version, route reason, approximation class, and whether the result was complete. Product behavior must not represent a degraded guess as the same outcome when the distinction matters.

Overload needs an honest quality policy

The fixture prices short requests at 56.8 accelerator-ms and long requests at 467.6 accelerator-ms. At 140 short and 18 long requests/s, offered demand is 16,368.8 accelerator-ms/s. With each device protected at 70%, it contributes 700 schedulable ms/s, so the arithmetic minimum is 24 devices and normal demand consumes 97.43% of the protected envelope. That is already too little uncertainty for many real systems; it is intentionally visible.

At 1.25× arrivals, demand becomes 20,461 accelerator-ms/s. The same fleet cannot preserve all work. For eligible requests only, routing 60% of the short class and 50% of the long class to a smaller model saves a modeled 5,173.5 accelerator-ms/s and leaves 15,287.5, or 90.997% of protected capacity. The system still needs queue and failure evidence. If fallback quality falls below its floor, the route is unavailable and admission must reject or defer work instead.

Use an overload ladder:

1. stop shadow traffic and optional secondary stages;
2. reduce candidate counts or output limits within declared quality evidence;
3. route eligible classes to a warm validated smaller model;
4. defer asynchronous and batch work;
5. reject new interactive work early with retry guidance or a product alternative; and
6. protect already admitted sequences and the control/recovery plane.

Do not cold-load the fallback during overload. Reserve its memory, prove warm capacity, and test loss of a device group. Retrying inference can duplicate downstream effects; use an operation identity and idempotent effect boundary where inference triggers writes. Cancellation must stop future decode and release state, not merely close the client socket.

Versions, warm-up, and shadowing consume capacity

A model release includes weights, tokenizer/preprocessor, feature schema, runtime, kernels, adapters, policy, and output validator. Assign one deployment identity to that manifest.

Load the candidate without serving traffic, verify integrity, allocate runtime state, and run representative warm-up shapes. Warm-up should cover short/long, sparse/dense, modality, and relevant precision paths without polluting tenant caches. Mark a replica ready only after its declared paths pass.

Shadowing duplicates selected inputs to a candidate while returning only the control result. It consumes feature, CPU, transfer, device, network, logging, and privacy budget. Redact or avoid prohibited inputs, cap shadow rate by shape and tenant, and prevent shadow work from entering production queues. Compare paired quality and latency with the same versioned evaluation logic.

Canary by request class and shape, not random request count alone. Watch first-token/completion latency, deadline miss, quality guardrails, memory high-water, allocation failure, padding, preemption, and fallback rate. Rollback routing may be fast, but unloading the new model and rewarming the old path are stateful operations. Preserve both-version capacity or state the rollback delay.

Isolation includes memory and information

Tenant quotas by request count fail when one tenant sends long contexts. Charge predicted and observed accelerator-ms, live-token-ms, transfer bytes, feature work, and reserved model memory. Use hierarchical reservations with bounded borrowing. Limit per-tenant queued work, live sequences, context/output length, adapter/model variants, and cache occupancy.

Performance isolation is not privacy isolation. Shared batches, prefix caches, adapters, logs, device memory reuse, timing, and error messages can expose cross-tenant information. Zero buffers according to threat model, scope caches and encryption, avoid mixing prohibited tenants, authorize model/adapter access, and audit administrative paths. Privacy controls can reduce batch opportunities or require dedicated cells; price that cost explicitly.

Observe by model and version, request class, input/output shape buckets, tenant or privacy-safe cohort, route, region, device group, and quality population. Preserve distributions for queue dwell, first result, completion, phase execution, live memory, tokens/items per result, and cost per validated success. High-cardinality raw input labels are both an operational and privacy hazard.

Model-serving capacity worksheet

Outcome and workload
  mode; task/quality population; arrivals and concurrency; input/output/item
  distributions; deadlines; tenant/region/privacy mix; growth and surge

Execution
  model/runtime/version; prefill/decode or stage demand; arithmetic intensity;
  host CPU; transfer; accelerator-ms; validation; useful-success denominator

Memory
  weights; runtime/workspace; activations; live cache per token/sequence;
  fragmentation; version overlap; failure and cold-load reserve

Scheduling and batching
  compatibility dimensions; slack; queue bounds; padding/ragged work;
  reservations; fairness; cancellation/preemption; admission

Parallel topology
  data/model/pipeline groups; interconnect and host boundaries; replicas;
  failure domains; movement; warm capacity

Upstream/downstream
  features/vector reads/preprocessing; freshness/version; fan-out; caches;
  policy and side-effect boundaries

Quality and overload
  evaluation population and metric; frontier; early exit; allowed fallback;
  prohibited classes; degradation ladder; rejection; rollback trigger

Evidence and economics
  observed/modeled/simulated fields; traces/profiles/quality set; power/cost;
  uncertainty; transfer limits; owner and review date

Capacity the maximum independent bottlenecks and their combinations: normal traffic plus shadow, version overlap plus long sequences, one device-group loss plus cold load, feature latency plus batch dwell, and surge plus fallback. A peak kernel number cannot answer any of those.

Applied work

Design the mixed-shape scheduler. Define the record accepted at admission, demand estimator, compatibility keys, deadline slack, tenant reservations, large-job aging, decode-turn fairness, live-memory bound, cancellation, and post-run charge. Reproduce the 5,888 useful versus 32,768 padded tokens and explain whether bucketing, ragged execution, or separate dispatch preserves the 200 ms objective. Inject underpredicted long requests and one lost device group.

Choose the overload route. State the task population, quality measure, full and compact versions, eligibility and prohibited classes. Reproduce 20,461 accelerator-ms/s at surge and the modeled 5,173.5 saving. Verify warm compact capacity, upstream capacity, provenance, product semantics, rejection behavior, recovery, and the trigger that disables fallback when quality evidence fails.

Rehearse a version change. Load and warm a candidate while the current version serves peak traffic. Shadow privacy-eligible requests by shape, canary the high-risk cohorts, force an allocation failure, then route back. Prove feature/tokenizer compatibility, two-version memory, output validation, cache scoping, and the time until the old version can safely be removed.

Sources and transfer limits

• NVIDIA’s current Triton batching documentation provides concrete dynamic, sequence, delayed, priority, and iterative batching mechanisms. It is product-specific; compatible shapes, backend behavior, and measured frontiers determine whether those mechanisms help Orchid.
• The primary Orca OSDI paper develops iteration-level scheduling and selective batching for transformer serving. Its evaluated models, hardware, software, and workload do not transfer as capacity claims.
• The primary PagedAttention paper analyzes key/value-cache fragmentation and block-based management in one serving system. Its memory and throughput results are evidence for a mechanism, not a universal cache size or scheduler policy.
• NVIDIA’s current Triton model-configuration documentation documents one warm-up/readiness implementation. Warm shapes, readiness semantics, update delay, and backend behavior remain deployment-specific.
• Current MLPerf Inference rules demonstrate scenario-specific latency, quality, preprocessing, and implementation boundaries. A compliant benchmark still does not reproduce Orchid’s request mix, tenant isolation, failure, features, price, or privacy constraints.

The chapter’s numbers are reproduced by examples/performance-engineering-system-design-handbook/part-05/ml-inference/. They are modeled teaching evidence. A production decision needs observed shape and arrival distributions, phase-aligned traces, memory profiles, supported-device topology, quality evaluations, upstream behavior, tenant/privacy mix, version manifests, failure tests, power, and cost.

Decision rule

Schedule and price inference by actual compute, memory lifetime, transfer, upstream work, and validated quality objective—not request count. Batch only compatible work inside explicit latency and fairness bounds; reserve warm, quality-approved degradation before overload arrives.

Inference telemetry must distinguish waiting, useful execution, memory pressure, route, version, and quality without collecting prohibited inputs. The next chapter moves observation to devices where execution windows, network reachability, software versions, energy, and even telemetry delivery are intermittent.