Cost Engineering

A successful feature with a shrinking margin

An image-processing feature removes backgrounds from product photographs, creates thumbnails, and sends uncertain cases to a third-party model. Customers like it. Usage is growing. So is the infrastructure bill, which would be unremarkable if the two were growing together.

They are not. In this modeled example, the feature once cost $180,000 a month to produce 10 million accepted images. It now costs $350,000 to produce 14 million. Total usage rose by 40 percent; cost per accepted image rose from 1.8 cents to 2.5 cents. More customers cannot repair those economics. At the current shape of the system, growth makes the problem larger.

The bill does not yet explain why. GPU workers may be poorly utilized. Failed jobs may call the model repeatedly. Originals and intermediate files may never leave hot storage. The dashboard may pull full-resolution previews across regions. One customer tier may create most of the work. Each possibility suggests a different change and threatens a different product promise.

Cost engineering begins here: turn spend into a claim about system behavior, locate the behavior that changed, and alter it without quietly transferring the cost into latency, reliability, quality, support, or engineering effort.

The denominator changes the diagnosis

“Our cloud bill is too high” has no engineering boundary. Even “cost per request” can reward the wrong thing. This feature receives image-processing requests, but customers value usable images. Failed attempts, automatic retries, and discarded low-quality results consume resources without creating that value. The primary denominator should therefore be an accepted image.

A second denominator can reveal who is creating the work. Cost per tenant or per plan tier may show that a few catalog importers account for most processing, or that a free tier consumes more model calls than its limits assumed. The two views answer different questions:

• cost per accepted image says whether the production path becomes more or less efficient as it runs;
• cost per tenant says whether pricing, quotas, isolation, or product policy match the workload customers actually create.

The denominator must stay attached to a product promise. For this feature, an accepted image has a visible quality bar, a target time to first usable result, and a recovery window during which support can restore the original. A cheaper result that customers reject is not the same unit. Nor is an image that arrives after the workflow in which it was needed.

This is why unit economics prevent local victories. A cheaper model may lower cost per call while rejecting more outputs. Moving enhancement work out of the request path may lower compute peaks while increasing abandonment. Deleting originals may reduce storage while making support unable to recover a damaged catalog. The useful measure follows the whole outcome.

Follow one accepted image through the bill

The first cost model should be rough enough to make quickly and honest enough to expose uncertainty. Trace the path that creates the unit:

cost per accepted image
  = allocated baseline capacity
  + upload and validation
  + deterministic transforms
  + third-party model calls
  + retained originals and derivatives
  + network movement
  + failed and repeated work
  + material operating and support burden

This is an engineering ledger, not a finance allocation system. It separates fixed cost from cost that grows with each unit. Minimum replicas, control planes, observability, support contracts, and deliberately reserved headroom exist before the next image arrives. Compute time, storage, egress, and per-call vendor charges usually grow with traffic, though discounts and commitments may make the relationship uneven.

Now follow one upload. It lands in the customer’s region, crosses to a GPU pool, is written as an original, an intermediate file, several thumbnails, and a final asset, and is read by the dashboard. An uncertain result invokes the external model. A timeout causes the queue to redeliver the job. Without an idempotency boundary, the redelivery repeats transforms and may invoke the model again.

The expensive line on an invoice is not always the controlling cause. The model provider may be the largest named charge, but retry amplification may be the behavior making that charge grow. Object storage may look inexpensive per gigabyte, while keeping every intermediate forever creates an unbounded multiplier. Network egress may be the symptom of a dashboard that fetches large originals instead of local thumbnails. The ledger connects the charge to the decision that produced it.

Attribution will be imperfect. Shared clusters blur service boundaries, tags go missing, and staff time resists a clean per-unit number. Record those limits, sample where necessary, and improve the model where a decision depends on it. False precision does not make a bill causal.

Segment until the expensive behavior has a name

Fleet averages hide cost tails just as they hide latency tails. Segment accepted and failed work by tenant, plan, region, payload size, feature path, result quality, retry count, and time of day. Then compare the expensive cohort with a normal one.

Suppose the image service reveals four facts:

1. Large catalog imports contain many byte-identical images under different filenames.
2. Jobs that time out can invoke the external model two or three times before one result is accepted.
3. GPU workers are sized for import peaks but remain mostly idle overnight.
4. Originals, intermediates, and dashboard previews stay in the primary region and hot tier regardless of age or where they are read.

The rising unit cost now has a shape: repeated work, failure amplification, idle capacity, unbounded retention, and unnecessary data movement. Those are not five independent “cost tips.” They are observations about one path.

The cohort matters. If one paid tenant legitimately imports an enormous catalog, a blunt global limit may punish the customer the product was built to serve. The decision could instead be a contracted batch window, an isolated worker pool, a price tier that reflects the workload, or an optimization aimed at that import shape. Cost attribution informs product policy; it does not replace it.

Make causal changes, not cheap-looking ones

The first changes should remove work whose absence does not change the customer-visible unit. A content hash can reuse deterministic transforms for a duplicate image. An idempotency key and retry budget can prevent one logical job from paying for several model calls. Smaller local derivatives can serve the dashboard without moving full-resolution originals across regions.

Storage needs a policy rather than a cheaper bucket. Keep current assets where the product reads them quickly, move older originals to a colder tier when the recovery promise allows slower access, and delete intermediates when they no longer serve replay, audit, or support. Every retained copy must inherit deletion, privacy, and access-control obligations. A lifecycle rule that cannot honor those obligations is not finished.

Capacity also follows the work’s timing. The service may keep warm capacity for interactive uploads while scaling batch workers against queue age and delivery deadline. Predictable baseline demand may justify a reservation or commitment; a short launch peak may justify temporary headroom instead. Turning off all idle capacity would look efficient until workers take too long to start or a missed batch window creates a backlog the system cannot drain.

These changes create obligations of their own. Content-addressed reuse needs correctness boundaries when transform code or quality settings change. Tiered storage needs restore monitoring. Autoscaling needs a downstream concurrency limit. A local thumbnail cache needs invalidation and authorization. An architecture that saves a small vendor charge by adding three fragile services may increase total cost through on-call load, migrations, and specialist knowledge.

For the image service, a concise decision record could read:

Unit: accepted image
Drivers: duplicate transforms, retry-amplified model calls,
         peak-sized workers, hot intermediates, cross-region previews

Changes:
- reuse deterministic outputs by content and transform version
- make model invocation idempotent and bound retries
- separate interactive headroom from deadline-driven batch capacity
- tier originals after the support window and expire safe intermediates
- serve region-local thumbnails to the dashboard

Guardrails:
accepted-image quality, time to first usable image, upload success,
support restore window, deletion correctness, backlog age

Recheck:
cost per accepted image and per tenant, calls per accepted image,
failed-attempt ratio, GPU utilization, storage by class and age,
egress per dashboard view

The record is useful because another engineer can challenge the causal claim. If duplicate work is rare, the cache will not pay for its complexity. If model quality falls, the unit definition catches a false saving. If restore requests become slow or unreliable, storage savings have crossed a product boundary.

Some expense is evidence of a promise

Cost optimization is not a mandate to make every line smaller. Cross-zone redundancy may protect an essential workflow. Logs may be the only evidence available during a security investigation. Spare capacity may be prudent before a launch with a hard deadline. A more expensive dependency may have the quality, audit trail, or support path the product requires.

Keep such spending when the promise is explicit and the evidence supports it. “Three replicas is the default” is weak. “This path keeps cross-zone redundancy because losing it violates the recovery objective; offline derivatives can run with lower availability” is a defensible allocation.

The same discipline applies to people and contracts. A managed service may cost more per operation while removing upgrade, recovery, and on-call work the team cannot responsibly staff. A commitment discount may lower unit cost for stable demand while becoming expensive idle capacity if the product changes. A vendor price can move, but so can the internal cost of operating an alternative. Architectural complexity, switching work, support burden, and opportunity cost belong in the decision when they are material.

Performance and cost can also disagree over time. Adding capacity may be the right mitigation for an active latency incident. The later cost review asks whether that capacity should become a measured baseline, remain a launch buffer, or trigger a design change. One metric does not own the system.

Leave behind a system that can remember

A one-time cleanup is easy to celebrate and easy to reverse. The image service needs a unit-cost view that joins spend with accepted outcomes, then segments the result by the dimensions that explained the regression. It also needs direct recurrence signals: model calls per accepted image, retry count, storage age, queue age, GPU utilization, and egress per dashboard view.

Budgets and alerts should point to investigation, not indiscriminate shutdown. A rising cost per accepted image asks whether product mix, quality rate, capacity, or the work per unit changed. A tenant budget can trigger review or isolation before it becomes a hard quota. Lifecycle policies need owners and restore tests. Commitments need renewal dates and workload assumptions.

In a system-design conversation, connect an economic consequence to the choice that creates it: replicas create storage and transfer, caches create memory and invalidation, retained detail creates future deletion work, and vendor calls create a metered failure boundary. In a project deep dive, give the before and after unit cost, the causal driver, the promise preserved, and the guard that made recurrence visible. A percentage saving without those boundaries is a finance headline, not yet an engineering account.

Rehearse one cost review

Choose a system you know and write one product unit at the top of a page: a completed checkout, delivered message, generated report, accepted image, or tenant-month. Trace the compute, storage, database, network, vendor, failure, and operating work needed to create it.

Then make the exercise resist the first answer:

1. Double one payload class without doubling successful outcomes.
2. Make one tenant dominate the expensive tail.
3. Add retries that preserve availability but repeat paid work.
4. Require a recovery, audit, privacy, or quality promise that forbids the easiest cut.
5. Choose one causal change and name the new obligation it creates.
6. Define the measure that would prove both the saving and the preserved promise thirty days later.

You understand the model when you can explain why the total bill changed, which useful unit should carry the cost, which cohort or mechanism drives the increase, and what evidence would make you reject your preferred optimization.

Field reference

1. Define the unit: choose the outcome the product or customer values.
2. Attribute the path: include baseline capacity, marginal work, retained data, movement, dependencies, failures, and material human ownership.
3. Segment the tail: find the tenant, tier, region, payload, feature, or failure path that explains the increase.
4. Name the mechanism: idle capacity, repeated work, retention, egress, skew, failure amplification, or vendor metering.
5. Protect the promise: state the reliability, latency, correctness, quality, privacy, recovery, audit, or support boundary.
6. Change the cause: prefer a reversible, measurable intervention whose complexity is proportionate to the saving.
7. Recheck and assign ownership: watch unit cost, the protected guardrails, and the signals that reveal recurrence.

Related links

• Performance Engineering
• Cost and Architectural Economics
• Observability, Performance, and Cost
• Build, Buy, or Partner