Cost, Latency, and Performance Engineering

The Cheap Model That Raised Total Cost

A document assistant switches to a model with a lower price per token. The finance dashboard shows immediate savings. But the new model needs longer prompts, invokes retrieval twice as often, and produces drafts that reviewers edit for three extra minutes. Users resubmit slow requests, the queue grows, and monthly spend rises.

The team optimized one invoice line while making the system more expensive.

Cost and latency are properties of the complete user task. Optimize the request path and the human workflow under explicit quality and safety floors—not a model price or benchmark in isolation.

Price the Completed Answer

The document assistant’s useful unit is an accepted answer, not a model call. That boundary changes the calculation. Model input and output belong in it, but so do retrieval, reranking, storage, network transfer, indexes, queues, caches, observability, tool calls, retries, failed requests, evaluation, and incident work. Most important, it includes the review and correction time created by the answer.

Suppose 10,000 questions produce 8,000 accepted answers. Dividing the invoice by 10,000 makes failed and abandoned work disappear. Dividing the complete operating cost by 8,000 keeps it visible. The same denominator should follow a non-AI baseline where one exists; otherwise a team can celebrate an inexpensive AI path that costs more than search, a form, or direct human handling.

The calculation should preserve its assumptions: provider and model version, region, currency, pricing date, traffic and context distributions, peak concurrency, reserved capacity, and utilization. Use ranges for uncertain inputs. Prices change; an auditable model of the workload lasts longer than a copied price.

This wider boundary also reveals environmental cost. Track the compute and energy evidence a provider or operator can support, distinguish measured data from estimates, and consider avoidable work such as duplicate inference, unused long context, idle reserved capacity, and unnecessary retraining. Do not turn an unavailable carbon figure into a precise invented one. Treat energy and emissions as lifecycle constraints to measure and improve alongside service and financial efficiency.

Follow One Request Through Time

The team next traces an answer from the user’s action to an accepted result. Admission and policy checks lead to retrieval, reranking, context assembly, model queueing and inference, validation, network delivery, rendering, and review. The trace records the release identity from Chapter 33—application, prompt, corpus, policy, tools, router, and model—so a change in cost or latency can be tied to a changed system.

Start with the user’s deadline and work backward. If the assistant has 2.5 seconds at p95, each critical-path stage needs an allocation and the path needs headroom. An average is not a budget: a healthy median can conceal the long-context route whose queue time makes the product unusable. When streaming helps, measure both time to first useful output and time to completion. Server timing alone can hide network, rendering, and sequential client work.

Agent paths make the critical path easy to misread. Three 800-millisecond tools can finish in roughly one tool interval if they are independent and run concurrently; a planner that invokes them sequentially spends all three intervals. Parallelism can also create a load spike, so it needs cancellation, concurrency, and downstream-capacity limits. Every agent should have hard bounds on steps, retries, wall time, tool calls, and spend. At the boundary, it returns a safe partial result, escalates, or stops visibly rather than continuing work the user cannot see or cancel.

The trace should connect duration and cost to outcome. For this assistant it captures tokens and context composition; cache eligibility, age, and invalidation; queue, retrieval, reranking, inference, validation, tool, and delivery time; retries and duplicate suppression; estimated stage cost and reconciled charges; citation and authorization results; and reviewer effort. Privacy, access, and retention policy still govern what may be recorded. Cohorts such as task, risk, language, tenant, device, and context length turn a blended total into an explanation.

Use different horizons for different decisions. A per-request budget stops a runaway loop. Daily route and segment views expose drift or a broken cache. Monthly views support capacity and portfolio choices. None of them is complete without the accepted-outcome denominator.

Find the First Expensive Mechanism

The trace contradicts the original diagnosis. The cheaper model call is not the main problem. Duplicate chunks enlarge the context, two retrieval attempts serialize on the critical path, and weak citations create three extra minutes of review. The queue then encourages resubmission, which adds duplicate work and makes the tail slower.

The team first removes exact duplicate chunks and repairs retrieval so one permission-aware search produces a smaller, stronger evidence set. It caches document parsing, where reuse and invalidation are clear, but not generated answers whose evidence and authorization can change. Independent metadata lookup moves off the critical path. These changes attack observed mechanisms without yet changing model behavior.

Other levers belong to different workloads. Batching improves throughput for compatible offline work but adds queue delay and can mix privacy boundaries. Precomputation helps stable artifacts but creates freshness and deletion obligations. Structured output can reduce parsing and repair while remaining substantively wrong. Quantization, distillation, and smaller models can reduce inference work but require fresh evaluation of behavior, robustness, calibration, and segment performance. Removing agent steps can remove wasted work or remove the recovery step that made the task complete.

Change one major mechanism at a time when practical. Then compare cost per accepted answer, the latency distribution, task success, reviewer effort, and control effectiveness against the same traffic and evaluation portfolio. A saving that moves work into support, incident response, or an unmeasured human queue is a transfer, not an improvement.

Route by Eligibility, Not Pressure

The team considers a smaller-model route for routine questions. A router is a policy boundary: it can choose a smaller or larger model, a specialized model, a deterministic path, a human queue, or abstention. Its permitted features, task and risk classes, eligible releases, uncertainty threshold, prohibited routes, fallback, logging, override, and recalibration need to be explicit.

On the evaluation set, the smaller route meets the overall target but misses the citation floor for two languages. The blended result would permit release; the segment result does not. The team disables that route for those languages and sends uncertain classifications upward. It also refuses to weaken the route merely because the large-model queue is busy. Capacity pressure may justify admission control, a slower disclosed mode, or escalation. It does not justify silently lowering the safety or quality floor.

This is where cost engineering and reliability meet. A router, cache, timeout, or batch queue changes the system’s operating modes. Each needs the same versioning, monitoring, failure containment, and recovery discipline as any other production control.

Close the Budget With Evidence

After the retrieval and critical-path changes, the assistant’s illustrative p95 latency falls from 3.1 to 2.2 seconds and total cost per accepted answer falls by 24%. Citation precision, authorization tests, reviewer correction time, and task acceptance remain within their stated floors. Those figures describe a modeled exercise, not a transferable benchmark.

The decision record keeps the denominator, demand distribution, stage allocations, headroom, pricing assumptions, control floors, experiment design, segment results, owner, and rollback condition. The smaller-model result remains in the record as a rejected route, with the language boundary that prevented its release. That is more useful than a dashboard announcing a blended saving: it says where the optimization is valid and where it is not.

AI Cost and Latency Budget

• Task unit: completed outcome and non-AI baseline.
• Demand: volume, concurrency, context distribution, languages, peaks, and growth range.
• Latency: user-facing target, stage allocations, percentiles, timeout, and headroom.
• Cost: model, data, infrastructure, tools, observability, evaluation, people, and failure cost.
• Floors: task quality, safety, privacy, security, fairness, accessibility, and human workload.
• Controls: per-request caps, rate limits, route policy, cache policy, cancellation, and alerts.
• Evidence: trace fields, test suite, segments, pricing date, experiment, and decision.
• Review: owner, cadence, variance threshold, and remediation playbook.

Rework the Three-Tool Agent

An agent has a 2.5-second user deadline and three tool calls whose p95 durations are 700, 800, and 900 milliseconds. Draw the current critical path before proposing concurrency. Which calls are truly independent? What downstream capacity would concurrent calls consume? Where will cancellation occur if one result makes the others unnecessary?

Now add the outcome boundary. Name the completed task, the non-AI baseline, the quality and safety floors, the maximum steps and spend, and the visible behavior when the deadline expires. Propose one cost change and one latency change, then describe the trace and segmented evaluation that could disprove each improvement. Finish with the condition under which persistent cost, latency, or value failure should trigger the Chapter 39 decommissioning decision rather than another round of optimization.

Source Notes

• Google SRE, Service Level Objectives, official guidance on user-relevant indicators, latency distributions, objectives, and error budgets; verified 2026-07-20.
• Google SRE, Production Services Best Practices, official operational guidance connecting error budgets, releases, and actionable monitoring; verified 2026-07-20.
• NIST AI Risk Management Framework Core, voluntary guidance on assessing and documenting environmental impact and sustainability, evaluating performance under deployment-like conditions, and monitoring system behavior in production; verified 2026-07-20.
• Vendor pricing is intentionally not embedded as a durable claim. Any real budget should record the provider’s current official price page, model/version, region, currency, and verification date.
• See Reliability, Resilience, and Fallback Design for response when budgets or dependencies fail.