Case Study: Cost-Constrained AI Inference

Orchid Serve’s finance review begins with two facts that appear to disagree.

The regional fleet meets its 1.5-second p95 completion objective. Accelerator utilization is high, and the service completes almost every request. The serving bill is still 17% above the product’s approved cost per successful response.

The platform team proposes a larger batch. In a warm microbenchmark, raising the maximum from eight to sixteen increases accelerator throughput by 11%. The proposal sounds like a direct way to reduce cost.

The production replay fails. p95 rises to 1.88 seconds, padding amplifies scheduled token positions by 3.7× in the mixed workload, and 8.3% of requests miss their deadline. More kernels finish per unit time, but fewer policy-valid responses finish while useful. Cost per successful response does not improve enough because expired work, long-sequence state, and fallback retries remain charged.

The product requirement was incomplete: “serve AI requests under 1.5 seconds for at most $0.00016 each without hurting quality.” It omitted which latency event, which request population, which quality definition, whether a timed-out response counts, which models and lengths may share a batch, whether truncation is visible, and what happens when a device or model version is lost.

Orchid turns it into four gates for one named population:

• p95 from authenticated edge admission to a complete policy-valid response is at most 1,500 ms;
• modeled cost per successful response is at most $0.00016;
• a versioned segmented task-utility index is at least 0.895;
• at least 96% of eligible offered requests satisfy the complete contract.

The quality index is fictional and task-specific. It is not a universal measure of intelligence, truth, safety, or user value. Every numerical observation in this case is modeled or simulated teaching evidence. The packet does not execute a model, benchmark a device, or measure a production user.

The design lesson is: request count is a poor capacity unit for heterogeneous inference; scheduling must price actual resource demand and quality value.

Decide what one successful response means

Orchid’s boundary begins after authentication but before model routing. It ends only after preprocessing, queueing, host-to-device transfer, model execution, iterative output, policy checks, schema validation, and response assembly. Feature or retrieval calls that a route requires are inside the cost and latency boundary. Client rendering and later user behavior are separate product outcomes.

A success satisfies all of these conditions:

request was eligible and admitted under policy generation P
selected model, tokenizer, adapter, and validator versions were compatible
complete or explicitly bounded output met the route contract
mandatory policy and schema validation passed
response completed before the request deadline
quality evaluation includes this route and request segment
all attempts and retained state are attached to one logical request identity

A response that arrives after the deadline is physical throughput, not successful goodput. A smaller-model response counts only for a class whose product contract permits fallback and whose measured quality clears the floor. Silent truncation is not success. A cancellation that leaves device work running still creates cost. A policy-invalid response cannot be made cheap enough to pass.

The denominator is eligible offered requests, not only completions. This prevents admission from improving p95 by dropping difficult traffic without consequence. Orchid reports offered, admitted, rejected, cancelled, fallback, early-terminated, completed, policy-valid, deadline-valid, and quality-eligible populations separately.

Latency also has multiple events. The fixture’s gate is completion p95 because its product consumes complete structured results. A streaming product would separately govern time to first token, inter-token gap, and terminal completion. Combining them into “model latency” would let a fast first token hide a stalled sequence or let a long but healthy completion inflate an interactivity metric.

The product contract permits a compact fallback for classification and selected short-assistant tasks. It prohibits fallback for premium document generation and specified high-risk segments. Early termination is permitted only when a calibrated terminal condition produces a complete semantic response or when the API returns an explicit bounded-output state. The gateway never labels a deadline cut as a normal completion.

Convert length, model, and output into schedulable demand

Four request classes arrive during the modeled warm regional peak:

There are 325 requests/s. Equal request pricing would give each class 0.31% of “one second of fleet capacity,” an obviously false equivalence. Input tokens alone are also insufficient. Autoregressive output repeats model work and retains sequence state. Model variants have different demand. An output cap is reserved admission demand even when typical responses finish earlier.

The fixture uses a deliberately simple token-equivalent unit, or TEU:

TEU_i = f_model,i × (0.25 × I_i + O_max,i)

where (I) is input tokens, (O_{max}) is the admitted maximum output tokens, and the model factors are 0.45 for compact, 1.0 for full, and 1.8 for premium. The coefficient 0.25 is a fitted teaching weight for this packet, not a hardware law. A real system fits separate phase demand from measured accelerator time, memory traffic, transfer, and batch state for each model/runtime/hardware version.

The arithmetic produces:

classification
  0.45 × (0.25 × 256 + 16)       =    36 TEU/request

short assistant
  1.00 × (0.25 × 900 + 120)      =   345 TEU/request

long assistant
  1.00 × (0.25 × 6,000 + 500)    = 2,000 TEU/request

document generation
  1.80 × (0.25 × 12,000 + 900)   = 7,020 TEU/request

Demand by class is 7,920, 31,050, 24,000, and 21,060 TEU/s, totaling 84,030 TEU/s. Classification supplies 67.7% of requests but 9.4% of modeled TEU. Document generation supplies 0.92% of requests and 25.1% of TEU. Request count reverses the capacity story.

TEU helps admission compare heterogeneous work. It does not replace physical counters. Orchid retains input and output tokens, accelerator milliseconds by prefill and decode, live-token milliseconds, peak and allocated cache bytes, transfer bytes, host CPU, queue wait, model residency, external calls, energy where measurable, and dollar allocation. Prediction error is recorded by class and version.

Output is uncertain. Admission reserves the declared maximum or an approved percentile plus safety policy; accounting charges observed work; scheduling updates remaining demand after each turn. A tenant cannot declare 32 output tokens, generate 800, and receive the cheap lane. Crossing the cap produces a typed stop or a separately admitted continuation, never unbounded continuation by accident.

Shape includes more than length:

• model, version, adapter, tokenizer, precision, and execution topology;
• modality and preprocessing path;
• tenant, privacy region, cache-sharing policy, and priority class;
• input length, output cap, current output, and live state;
• absolute deadline and remaining reserve;
• schema, tool, policy, and validation requirements; and
• fallback and early-termination eligibility.

Two 900-token requests may be incompatible because one requires a different adapter or privacy cell. Two compatible requests may still be a bad batch because the older one has insufficient slack.

The batch-16 result was the wrong experiment

The warm microbenchmark preloaded one model, used one input-length bucket, generated fixed short outputs, excluded policy checks, and drove a closed loop. It answered a narrow question: can this execution path process more scheduled tokens per second at batch sixteen?

Production asked a different question: how many eligible requests finish correctly within 1.5 seconds per dollar under a mixed arrival stream?

At batch sixteen, the scheduler waited longer to fill compatible groups. One long prefill padded shorter work in the static path. Completed short sequences could not always release their slots until the longest member reached the batch boundary. Queue delay correlated with length and tenant class. The closed-loop generator reduced offered load when latency rose, hiding the backlog that an open-loop arrival schedule exposed.

The replay records 11% higher microbenchmark throughput beside 1.88-second production p95, 3.7× padding amplification, and an 8.3% deadline-miss fraction. None of those figures is transferable beyond the fixture. Their purpose is to show why a local throughput gain can worsen unit economics.

The failed experiment changes the measurement plan. Every batching trial now declares:

• arrival process and whether it is open loop;
• request mix and length distributions;
• model and runtime manifest;
• warm/cold residency and cache state;
• batch-size and padded-token distributions;
• queue dwell, first-result, inter-result, and completion distributions;
• policy-valid correct goodput;
• cancelled and late work;
• live cache, preemption, transfer, and allocator state; and
• full hourly cost divided by successful responses.

The result also rejects “maximize average batch size” as a controller objective. Batch size is an intermediate packing observation. The controller protects completion, quality, memory, and cost gates.

Compare three scheduling shapes at the right boundary

Static, dynamic, and continuous scheduling solve different problems.

Static batching

A static batch waits for a fixed or planned group, executes that group together, and normally completes at a common boundary. It can exploit regular tensor shapes and amortize fixed work. It is attractive for homogeneous asynchronous jobs with generous deadlines.

In mixed generative serving, it pays queue dwell, padding, and longest-member delay. A short sequence that completes early may leave a logical slot unusable until the batch ends. Static batch eight reaches 287.0 successful responses/s in the fixture, with p95 2,260 ms and modeled cost $0.000184 per success. Its quality index is 0.914 because it uses the full route, but latency and cost fail.

Dynamic batching

A dynamic batcher collects currently compatible requests up to a size or delay bound, then dispatches without requiring a permanently fixed group. It trades a controlled wait for better packing. NVIDIA Triton Inference Server’s batcher documentation describes dynamic batching, maximum queue delay, priorities, queue limits, and timeouts for its implementation. It supports the mechanism distinction; it does not validate Orchid’s parameters or generative semantics.

Dynamic batch eight reaches 315.4 successful responses/s, p95 1,480 ms, quality 0.912, and cost $0.000162. It narrowly passes latency and misses cost. The small tail reserve makes it fragile under a cold path or device loss.

Continuous, deadline-aware scheduling

Continuous scheduling admits new compatible work between iteration boundaries, removes completed or cancelled sequences, and schedules prefill and decode quanta explicitly. It avoids waiting for the longest request to release every slot, but it introduces state ownership and fairness obligations.

The Orca OSDI paper presents iteration-level scheduling for transformer serving and distinguishes it from request-level scheduling. Orchid uses that research to justify evaluating finer scheduling granularity. The paper does not establish Orchid’s TEU factors, quality routes, device costs, or SLO results.

Orchid’s selected policy limits a prefill turn to 512 input tokens and a sequence to at most eight consecutive decode turns before another eligible sequence is considered. Those are fixture controls, not universal optima. The scheduler chooses feasible work within tenant and route reservations, accounting for predicted completion, live cache, absolute deadline, and remaining output budget.

The selected path produces 319.8 successful responses/s, p95 1,310 ms, quality 0.902, and cost $0.00015113. It clears the three primary gates. It gives up 0.012 of modeled utility relative to the full-only point for an eligible segmented population. Product owners must accept that trade; the scheduler cannot define quality value alone.

Memory is scheduled state, not free headroom

Each modeled full-model device has 80 GiB. The residency ledger closes exactly:

At 131,072 bytes per live token, the arithmetic 34 GiB pool can represent 278,528 live tokens. Applying the fixture’s 85% usable factor for block granularity and operating safety gives 236,748 whole tokens after flooring. This is an admission ceiling, not a measured allocator guarantee.

Key/value cache consumes memory over time. A sequence that pauses between decode turns may use no compute at that moment while retaining cache. Preempting compute does not automatically free memory. Swapping cache to host memory spends device-to-host bandwidth, host memory, later transfer, and latency. Recomputing it spends accelerator work. Dropping it changes or terminates the request contract.

Paged allocation can reduce external fragmentation and support flexible sharing. The PagedAttention paper describes block-based KV-cache management and the vLLM system’s evaluated design. It motivates explicit cache allocation and sharing analysis. Orchid does not import its reported throughput into this fixture, and paging does not remove the need for a safety reserve, tenant isolation, deadline policy, or cancellation.

Memory admission reserves expected live tokens and peak transient state. It also caps per-tenant live tokens, per-route sequences, and model or adapter variants. A long prompt can be feasible in TEU yet infeasible in current cache. A device can show low compute utilization while having no safe live-state capacity. Autoscaling on utilization alone may react too late because loading and warming a new replica takes longer than the queue can wait.

Weights are resident only after a manifest is verified. A deployment manifest includes model weights, tokenizer, adapter, precision, runtime, kernels, policy, schema validator, and feature contract. Loading consumes storage and network reads, host memory, host-to-device transfer, initialization, and warm-up execution. “Model ready” means the declared shapes and policy path passed, not that bytes exist on a device.

The 80 GiB full-device ledger cannot hold another 26.08 GiB full version. Orchid rejects in-place overlap. Candidate versions use a separate four-device canary pool. That choice spends fleet capacity but preserves honest rollback. A pointer switch cannot roll back to weights that were evicted and need minutes of stateful recovery.

Host-device transfer is part of the critical path. Inputs are tokenized in bounded host pools, placed in owned staging buffers, and transferred only after device admission. Queuing thousands of prepared buffers in host memory would hide backlog outside the device scheduler. Traces distinguish host preparation, transfer ready, transfer start, transfer completion, prefill, decode turns, output transfer, validation, and response.

Admission prices feasibility before priority

Orchid’s admission record contains:

logical request and tenant
model route and compatible fallback set
input length and admitted output cap
predicted TEU, phase milliseconds, live tokens, and transfer bytes
absolute deadline and post-model reserve
quality segment and minimum route
policy, tokenizer, adapter, and validator generations
queue, model-residency, and privacy cell

The first question is feasibility. Can one compatible warm replica complete the remaining work, preserve post-model policy reserve, and fit live memory before the deadline? If not, priority should not launch it. Orchid rejects, defers, narrows under an explicit product contract, or chooses an eligible fallback.

The second question is protected share. Capacity is hierarchical: control and recovery, region, model pool, tenant class, tenant, and route. Premium weight does not permit one 12,000-token request to monopolize all decode turns. Standard classification has a minimum because it supports a separate critical product path. Background evaluation and shadow traffic have no claim during overload.

The third question is ordering within eligible work. Orchid combines deadline slack, attained tenant share, age, predicted remaining TEU, and live-memory consequence. Shortest work alone can starve long requests. Earliest deadline alone can prefer a request that cannot finish. Strict priority can retain low-priority cache forever while serving high-priority compute.

The policy applies several bounded actions:

• admit full route: default when quality, deadline, memory, and cost close;
• admit compact fallback: only for eligible segments with a warm validated version;
• early terminate normally: only on a calibrated semantic stop condition, such as complete structured output;
• return bounded output: only when the response explicitly states its terminal bound and the product accepts it;
• defer: for asynchronous work with a durable identity, deadline, and queue-age bound; or
• reject: before tokenization and allocation when no valid route can finish.

Early termination is not timeout. If a structured classifier reaches a terminal class after 12 output tokens, stopping at 12 can be a complete response. If an assistant hits the wall clock halfway through a sentence, relabeling it “early exit” does not make it valid.

Fallback is a model route, not an emergency guess. It has its own task evaluation, prohibited segments, tokenizer and schema compatibility, warm residency, output policy, and rollback. The selected fixture routes 18% of eligible requests through fallback and early-terminates 7%. Those fractions are observations to monitor, not targets to maximize.

During overload, Orchid sheds work in this order:

1. stop shadow, profiling, and optional secondary evaluation;
2. stop admitting asynchronous and batch work to interactive pools;
3. cap optional outputs within declared product policy;
4. choose a warm compact route for eligible segments;
5. reject new infeasible work before allocating cache; and
6. preserve admitted sequences, mandatory policy validation, cancellation, and recovery control.

It does not cold-load the fallback after queues are already late. It does not lower quality for prohibited segments. It does not let clients retry immediately without a stable reason and budget, because retries amplify TEU and live state.

Build the quality-latency-cost frontier by segment

One scalar quality average can hide harm exactly as a fleet percentile hides a tenant. Orchid evaluates quality by task, language, tenant policy, input-length bucket, output-length bucket, risk class, model route, and load state. Mandatory safety and schema validity are zero-tolerance gates rather than weighted preferences.

The fixture compares five policies:

A point is dominated when another eligible point is no worse in quality, latency, and cost and is better in at least one. The selected point is not globally “best.” Premium only has higher modeled quality. Aggressive fallback is faster and cheaper. The product constraint selects the feasible point.

The frontier must be conditional on load. A route that passes at steady state may miss under a 1.4× spike or after losing four devices. It must also be conditional on segment. The selected aggregate quality 0.902 does not permit a regulated or multilingual segment to fall below its own approved floor.

Quality evidence combines a frozen versioned evaluation set, adversarial and policy suites, shadow comparisons, and guarded online outcomes where appropriate. Online clicks or continued conversation are not automatic proof: position, novelty, latency, truncation, and user effort can change them. Human evaluation records rubric, rater population, disagreement, and model/prompt version.

The frontier includes uncertainty. Completion predictions have calibration error; output length is unknown; device cost and fleet overhead change; quality samples have intervals. A point sitting exactly on every gate is operationally fragile. Orchid’s 190 ms tail reserve and $0.00000887 cost reserve are not large enough to ignore drift, so rollout gates remain conservative.

Changing the quality threshold is a product decision with an evidence record. The platform team can show the capacity and cost consequence, but it cannot silently trade accuracy or safety for utilization. Conversely, demanding premium quality for every low-risk classification has an economic consequence that the product must fund.

Close unit economics with all reserved capacity

Each of 28 devices supplies a modeled raw 5,000 TEU/s. At the 70% safe factor, one supplies 3,500 schedulable TEU/s and the fleet supplies 98,000 TEU/s. Offered demand is 84,030 TEU/s, consuming 85.7% of the safe envelope and leaving about 14.3% steady headroom.

That headroom is not all failure reserve. Some is prediction error, batch incompatibility, memory fragmentation, deployment, and transient variance. The fleet topology contains four-device groups. Losing one leaves 24 × 3,500 = 84,000 TEU/s, which is 30 TEU/s below even the steady offered model. The small arithmetic deficit proves that no-fallback full service cannot survive the group loss. The degraded policy must reduce demand, add already-warm reserve elsewhere, or reject work.

The hourly ledger is:

28 devices × $4.80/device-hour        = $134.40/h
hosts, storage, network, control,
policy validation, and observability =  $39.60/h
total                                 = $174.00/h

successful responses
319.8/s × 3,600 s/h = 1,151,280/h

cost per successful response
$174 / 1,151,280 = $0.000151136...

The fixture rounds the policy card to $0.00015113. The small difference is below the packet’s reconciliation tolerance. These costs are fixed assumptions, not public market rates.

Cost per request would divide by 325 offered requests/s and make rejection appear cheap. Cost per accelerator token would omit host, policy, network, reserve, and failed work. Cost per successful response aligns the economic unit with the product outcome, while the raw resource ledger keeps diagnosis possible.

Shared overhead allocation is explicit. Orchid assigns direct device and host work to requests, model residency to the routes that require it, reserved failure capacity to the product promise, and common control cost by a governed policy. Shadow and candidate rollout cost stays visible as deployment investment. Idle compact fallback residency is not “waste” when the overload contract requires it warm.

Tenant budgets include predicted and observed TEU, live-token milliseconds, output tokens, external calls, and route cost. A daily dollar cap cannot replace a microburst limit; a tenant can remain under budget and exhaust cache in one second. Orchid enforces per-request shape, concurrent live state, short-window TEU, and longer-window spend.

Break the design in four different ways

Steady-state success does not select the policy. Orchid runs four adversarial trials with open-loop arrivals, fixed model manifests, full attempt accounting, policy validation, and segmented quality reconciliation.

A 1.4× load spike

The spike makes offered TEU exceed the steady safe fleet. Orchid stops shadow work, defers asynchronous routes, reduces optional output caps under declared contracts, sends eligible classes to the warm compact model, and rejects infeasible arrivals before cache allocation. The modeled result is p95 1,440 ms, cost $0.000158, quality 0.897, and SLO attainment 96.8%. All gates pass narrowly.

The trial also checks queue age and restoration. When load falls, Orchid waits for admitted demand and live cache to return inside steady bounds, then restores routes one at a time. Re-enabling every full route at once can create a second queue wave.

A candidate model rollout

Four separate canary devices hold the candidate manifest. A bounded shadow fraction runs only when reserve exists. Candidate requests are selected by shape and segment, not uniform count, so long and premium routes receive evidence. The candidate must pass warm-up, policy, schema, quality, memory, and latency gates before serving.

The modeled trial produces p95 1,340 ms, cost $0.000153, quality 0.898, and 98.2% attainment. Passing does not authorize an immediate fleet switch. The quality margin is small, and peak, cold-start, and rollback coverage must complete. Rollback routes new requests to the old warm pool while candidate sequences finish or cancel under their starting manifest.

Loss of a four-device group

The capacity packet predicts a 30 TEU/s steady deficit before queueing effects. The loss detector fences the group, cancels or reconstructs ambiguous ownership, stops shadow traffic, invokes eligible fallback, and preserves control capacity. It does not retry every lost request simultaneously.

The modeled degraded result is p95 1,460 ms, cost $0.000159, quality 0.896, and 96.3% attainment. Again, the margins are intentionally tight. The recovery test includes model loading, host-device transfer, warm-up shapes, cache reconstruction or request failure semantics, and gradual restoration. Device count alone is not readiness.

Long-request domination

The generator injects a cluster of long prefills and maximum-output sequences. Without chunking, they occupy the execution path and retain cache while short work waits. With 512-token prefill quanta, no more than eight consecutive decode turns, live-token admission, and tenant shares, the modeled result is p95 1,410 ms, cost $0.000156, quality 0.899, and 97.1% attainment.

The fairness gate is not only short-request p95. Long eligible requests must make bounded progress and complete at their own objective. A scheduler that continually preempts long work can produce an excellent aggregate tail while starving premium document generation. Orchid measures maximum no-progress interval, attained share, completion by length bucket, preemption, and cache residency.

For all trials, cancellation must reach future decode scheduling and release live state. If a device kernel cannot be interrupted, the attempt remains charged until it reaches a safe boundary. A gateway timeout is not resource release. The control plane needs the request identity to prevent a retry from creating an unbounded second sequence.

Make traces explain latency, memory, quality, and cost together

One request trace carries the policy and model manifest, but raw prompts and outputs are excluded from general telemetry. It records bounded shape and route fields:

• input and admitted output-length buckets;
• model, tokenizer, adapter, policy, and validator generations;
• predicted and observed TEU by phase;
• queue, batch, prefill, decode, transfer, validation, and response times;
• batch membership, useful and padded tokens, and cancellation;
• allocated, live, swapped, recomputed, and released cache blocks;
• fallback, early termination, truncation, rejection, and terminal state;
• tenant class and privacy cell under cardinality control; and
• cost components and quality-evaluation eligibility.

Stage percentiles are not added. Prefill and decode overlap across requests, transfer can overlap execution, and the p95 of each stage comes from different requests. The trace identifies the critical wait for an individual completion and separately accounts all consumed work.

Prediction calibration is an operational gate. For each shape cell, Orchid compares predicted completion probability, TEU, output length, and peak live tokens with observations. A cell whose nominal p95 prediction succeeds only 70% of the time is disabled or padded. Underprediction is dangerous because it admits work that cannot finish; overprediction wastes safe capacity and can unfairly reject expensive tenants.

Quality evidence joins by logical request and route manifest, not by raw text in metrics. Offline evaluation retains controlled inputs and outputs in an approved evidence store. Online telemetry records result state and bounded segment. Security and privacy policy determine retention, access, and deletion.

Useful operating views include TEU offered/admitted/completed, correct goodput, p95 by class, cost per success, live tokens and cache age, batch dwell and padding, output-cap utilization, fallback and early-stop rate, cancellation-to-release, model residency, warm-up status, transfer bandwidth, and policy-invalid outcomes. Accelerator utilization remains a diagnostic measure, not the primary SLO.

Stage the policy without losing rollback capacity

Orchid rolls out the demand model before the scheduler.

First, it observes request shape, fits phase demand, and compares TEU predictions with accelerator milliseconds and live memory. Existing routing stays authoritative. The team verifies that request identity survives preprocessing, device scheduling, validation, and billing.

Second, it shadows admission decisions. For every request, the new policy records full, fallback, bounded, defer, or reject and the reason. Review focuses on disagreements near deadlines, quality boundaries, long contexts, and tenant budgets. Unknown shapes remain on a conservative route.

Third, it enables hard shape and memory safety: input and output caps, maximum live tokens, maximum queued TEU, compatible model manifest, and typed rejection. These controls can change client behavior, so schemas and retry guidance are versioned.

Fourth, the separate canary pool serves 1%, 5%, 10%, 25%, 50%, and 100% of eligible demand with hold periods covering peaks and model-cache cycles. Cohorts are stratified by route, length, tenant, language, and risk. Gates include p95, attainment, segment quality, policy validity, cost per success, queue dwell, live cache, transfer, fallback, early termination, and cancellation release.

Rollback changes admission authority for new requests. Existing streaming sequences retain the model and policy generation under which they began unless the failure contract explicitly terminates them. The old pool remains warm until rollback evidence closes. Candidate cache and weights are released only after ownership and active-sequence reconciliation.

The team rejected random-request canarying because 220 classification requests can outnumber three document requests while representing much less TEU and risk. It rejected in-place model replacement because the memory ledger has no room for overlapping full weights. It rejected “rollback by reload” because load and warm-up time exceed the product deadline.

Record the final design decision

Decision. Price admission in versioned token-equivalent demand plus physical memory, transfer, and outcome requirements. Use dynamic batching for compatible fixed-output paths and deadline-aware continuous scheduling for iterative generation. Limit prefill and decode quanta, preserve hierarchical tenant and route shares, and admit only work whose deadline and live-state budgets close. Use warm validated fallback and explicit early termination only for eligible segments.

Outcome gates. Completion p95 ≤ 1,500 ms, cost per successful response ≤ $0.00016, segmented utility ≥ 0.895, SLO attainment ≥ 96%, and mandatory policy/schema violations at zero.

Evidence. The packet derives 84,030 TEU/s from four request classes, 98,000 TEU/s safe fleet capacity, 14.3% steady headroom, a 30 TEU/s deficit after a four-device loss, an exact 80 GiB residency ledger, 236,748 usable live tokens under the fixture policy, $174 hourly cost, and approximately $0.000151136 per successful response. It fixes three scheduler alternatives, five frontier points, the failed batch-16 result, and four degraded-state trials.

Rejected defaults. Equal request price hides six-thousand-TEU differences. Input tokens omit output work. Maximum batch size worsens dwell and padding. Full-only misses cost and tail gates. Aggressive fallback misses quality. In-place version overlap does not fit. Utilization-based admission misses live memory. Cold fallback cannot rescue an active spike. Silent truncation corrupts the outcome unit.

Failure behavior. Stop optional and shadow work, protect admitted sequence state and mandatory validation, select only warm eligible fallback, reject infeasible arrivals before allocation, fence lost devices, reconcile attempt identity, and restore routes gradually after queue and cache recover.

Rollback. Preserve the old warm pool, route new work back under a new admission generation, let or explicitly terminate existing sequences under their original manifest, retain decision and quality evidence, and release candidate state only after reconciliation.

Revisit. Any segment falls below its quality floor; p95 exceeds 1,500 ms; cost exceeds $0.00016; attainment falls below 96%; prediction calibration drifts; input/output mix changes by 15%; fallback or early termination rises unexpectedly; cache allocation or cancellation release crosses its bound; a model/runtime/device version changes; or candidate overlap no longer fits the declared fleet topology.

The decision intentionally does not select a model family, accelerator, quantization, inference runtime, or cloud. Those choices change every fitted coefficient and frontier point. They do not remove the need to define success, schedule live state, preserve quality policy, and price failed work.

Applied work

Field exercise: the cheap route is expensive

A compact fallback costs 40% less accelerator time than the full route. After enabling it for all standard tenants, reported cost per admitted request falls 18%. Cost per successful response rises 6%. Completion p95 improves from 1,420 to 1,180 ms. Quality for short English classification stays above its floor, but structured extraction schema failures rise from 0.1% to 3.4%, and affected clients retry once.

Diagnose the economics and propose the next experiment.

Answer guide

The cheap route creates invalid responses for structured extraction. Those responses are not successes, and one retry adds host, transfer, device, validation, and queue work. Cost per admitted request hides both the smaller denominator of valid completions and the duplicate attempts. The improved p95 may also be conditioned on completions, excluding schema failures.

Segment by task and terminal outcome. Join logical request identities to all attempts. Compare compact and full routes on identical structured inputs with schema-valid completion, quality, deadline, and total work. Verify tokenizer, schema, and output-constrained decoding compatibility. The immediate policy should remove structured extraction from fallback eligibility while retaining evidence for short classification.

A viable alternative is a task-specific compact model or constrained decoder if it passes the full contract. The wrong turn is increasing the quality scalar while leaving schema validity inside an average; mandatory validity should be a gate. The most uncertainty-reducing observation is cost and result state per logical request, including retries.

Principal exercise: one launch, one lost group

A product launch is expected to raise short-assistant arrivals by 60% for twenty minutes. At the same time, the candidate model must reach 25% serving exposure for a contractual evaluation. Losing one four-device group is a required design condition. The business will not accept a lower quality floor for premium tenants, but standard tenants permit a bounded compact fallback. Additional devices cannot arrive for six weeks.

Produce an admission, residency, scheduling, quality, cost, and rollout plan. State which requirement you would renegotiate if the model cannot close.

Answer guide

Recalculate TEU and live-token demand with the 60% increase rather than scaling request count. Short-assistant demand moves from 31,050 to 49,680 TEU/s, adding 18,630 and raising steady total to 102,660 TEU/s—already above the 98,000 safe fleet before group loss. The requirements cannot all be met with unchanged routes.

Protect candidate residency in its separate pool only if the remaining serving fleet can meet the launch contract. Shadow work stops. Standard eligible traffic receives a prevalidated compact route and bounded output. Premium reservations and quality floor remain. Asynchronous work defers, and infeasible new standard work receives typed rejection before cache allocation. Prefill/decode quanta and live-token limits protect short requests without starving admitted premium sequences.

The candidate exposure should be measured by TEU and segment as well as request count. Twenty-five percent of cheap classifications is not 25% of relevant serving evidence. If the contractual evaluation demands exposure during the launch, reserve its exact demand and include it in the cost/SLO plan; do not call it free shadow traffic.

Run a trace replay for launch mix plus group loss, including warm fallback, candidate routing, cancellation, cache pressure, and restoration. Gates stay segmented. If no combination meets capacity, renegotiate one explicit product constraint: reduce standard output caps, defer a task, accept bounded rejection, move candidate exposure outside the peak, or fund/borrow already-warm external capacity. Do not lower premium quality silently or assume autoscaling that cannot arrive.

The common wrong turn is applying the existing 14.3% headroom to a 60% increase in one class without recomputing demand. Another is unloading the compact model to make room for the candidate, eliminating the fallback required by the failure condition.

Field review card and source limits

When reviewing cost-constrained inference, ask:

• What makes a response successful beyond “the model returned”?
• Are admission, first result, inter-result gap, and completion separate clocks?
• Which input, output, model, adapter, tenant, and policy segments matter?
• What physical evidence calibrates the token-equivalent unit?
• Are predicted maximum output and observed output both accounted?
• How much live state persists while a sequence is not computing?
• Does the residency ledger include weights, runtime, cache, transfer, transient peaks, and safety?
• Can old and candidate manifests actually coexist, or is rollback a cold reload?
• Which batching shape is compatible with the workload and deadline?
• What prevents long prefills or decode streams from dominating turns?
• Can fallback and early termination be distinguished from timeout and truncation?
• Does every fallback segment independently clear quality and policy gates?
• Is cost divided by correct deadline-valid success with all attempts included?
• What happens during a spike, model rollout, device-group loss, and long-request burst?
• Does cancellation release future scheduling and live state end to end?
• Which gates stop rollout even when utilization and throughput improve?

Serving DNNs like Clockwork is additional primary evidence that predictable inference serving requires controlling and modeling system behavior rather than relying only on best-effort utilization. Its model set, hardware, and results do not transfer directly to Orchid. Orca supports iteration-level scheduling as a researched mechanism. PagedAttention supports explicit KV-cache management. Triton documents one current dynamic-batching implementation. None establishes Orchid’s 512-token quantum, eight-turn bound, TEU factors, quality values, p95, memory fit, or price.

Run the evidence packet:

cd examples/performance-engineering-system-design-handbook/part-08/cost-constrained-ai-inference
node analyze.mjs
node verify.mjs

The packet checks deterministic arithmetic and declared gates. A production decision requires model and tokenizer manifests, hardware and runtime versions, representative input/output distributions, fixed open-loop arrivals, warm and cold trials, raw phase traces, cache and transfer evidence, segmented quality evaluation, policy reconciliation, full cost allocation, repeated failure recovery, and documented uncertainty.

Orchid does not become efficient by keeping every device busy. It becomes economically defensible when each admitted unit has a plausible completion path, each live token has an owner and lifetime, every quality trade is explicit, and failed or cancelled work remains visible in the denominator. The same authority and rollback question becomes harder in storage migration, where two representations of durable data—not only two model versions—must coexist while traffic continues.