Cloud, Containers, and Runtime Platforms

When sixty healthy workers make the system slower

Consider an image-processing service for a marketplace. Sellers upload product photos through an HTTP API. The API records each upload, puts a job on a queue, and returns. Workers scan the file, produce several sizes, write the results to object storage, and update a metadata database.

The service usually runs eight workers. During a catalogue import, the queue’s oldest job reaches four minutes, so an autoscaler adds workers. All sixty start successfully. Their health checks pass. The platform reports the deployment as healthy.

The oldest job is now 38 minutes old.

The new workers opened too many database connections and sent too many files to the malware scanner. Requests began timing out, retries created more work, and useful throughput fell. Nothing was wrong with the container scheduler. It placed the requested workers and replaced the ones that failed. The mistake was the contract given to it.

A runtime platform is that contract between a workload and its operating environment. It says what may run, where it may run, how it is found, what it may access, when it receives traffic, how it scales, and what the platform should do when it stops. “Run it in containers” answers only a fraction of those questions.

We can repair the image service by following that sequence. Along the way, VMs, containers, orchestrators, managed platforms, and serverless functions become choices with consequences rather than names to place on a diagram.

One product contains several workload shapes

The image service is not one workload. Its public API is a long-lived request service. It needs warm capacity, stable network identity, bounded latency, graceful connection draining, and protection from duplicate side effects. Its image processor is a queue consumer. It needs idempotent jobs, controlled concurrency, retry limits, safe cancellation, and a repair path for files it cannot process. A catalogue reprocessing task is a batch job. It needs an input range, checkpoints, a progress measure, and a pause mechanism. The metadata database and original images are durable state.

Those differences come before the platform decision. An HTTP service, queue worker, scheduled job, stream processor, and stateful store do not become the same kind of program because they share a repository or a container image.

Start a design answer by writing the workload in verbs:

accept upload -> preserve original -> enqueue work -> scan -> transform
              -> preserve derivatives -> publish status

Then attach an operating claim to every step. The API may acknowledge an upload only after the original and job intent are durable. A worker may disappear after any instruction, so repeating a job must not corrupt its output. The scanner has a fixed concurrency allowance. The database has a safe connection and write rate. A seller’s image is durable data; a worker’s scratch directory is not.

This is the first senior-level distinction: replaceable compute must not quietly become the only owner of truth. Local disk is useful for caches, buffers, and scratch work when its loss is expected. Durable state belongs in a store with an explicit replication, backup, restore, and retention model—or in a deliberately stateful runtime whose identity and recovery rules are equally explicit.

For common databases, queues, object stores, and caches, a managed service is often the sensible owner. It delegates much of the patching, replication, backup tooling, and failover machinery. It does not delegate the application’s schema, access pattern, quotas, idempotency, restore expectations, or behavior during degradation.

Choose the boundary you need

A virtual machine offers a machine abstraction: a guest operating system, processes, users, filesystem, network interfaces, and disks. It is a good fit when a workload needs host-level control, a custom operating-system component, careful machine tuning, a legacy daemon layout, or deliberately pinned capacity. The cost of that control remains visible. Someone must own images, patching, process supervision, bootstrapping, configuration drift, and host replacement unless a higher platform layer does it.

A container packages a process and its stable runtime dependencies. Operating system mechanisms such as namespaces and control groups isolate its view and resource use, while containers on a host normally share the host kernel. That makes containers lighter than separate VMs and useful as repeatable artifacts; it does not make every container boundary equivalent to a VM or a complete security policy.

The image API and workers are good container candidates because they can be made disposable. Their image contains code and pinned runtime dependencies. Environment configuration and production secrets arrive at runtime. Original and derived images live in object storage, job truth lives in the queue and database, and scratch files may vanish with the process. The programs handle termination signals, stop accepting new work, finish or safely abandon current work, and leave incomplete jobs eligible for retry.

Calling a container a “small VM” tends to preserve the wrong assumptions: mutable filesystems, manual changes inside a running instance, long-lived local identity, and important logs or data stored only on that instance. A useful test is severe but simple: if the platform kills this container and starts the same artifact on another host, what truth is lost? If the answer is more than disposable work, the state design is incomplete.

An orchestrator adds a control loop:

declared desired state -> observe actual state -> act -> observe again

It places workloads, replaces failed instances, connects them to discovery, applies resource policy, and coordinates rollout. That loop is valuable for a fleet of independently deployed services and workers. It is also literal. A bad desired state—an unsafe replica count, a broken health check, a global configuration error—can be enforced faster and more consistently than a human operator could spread it.

Orchestration earns its operational surface when the organization needs its placement, reconciliation, rollout, policy, and multi-workload capabilities. It is not automatically the right first layer for a small internal service. A managed application platform may provide build integration, routing, scaling, logs, and deployment with fewer controls to own. The trade is deliberate constraint: less scheduling and networking freedom, platform-specific behavior, and a narrower migration path.

Serverless functions and event runtimes hide still more of the server lifecycle. They can suit bursty, short-lived image jobs, scheduled cleanup, or glue around managed services. The service still owns the work. It must know whether an event can be delivered again, how concurrency is limited, how long execution may last, how cold startup affects latency, what local state survives, and what identity the handler receives. Large native libraries, long scans, GPU use, warm in-memory models, or predictable sustained demand may make a container worker pool easier to control and cheaper to reason about.

No option wins in isolation. The question is which responsibility the team needs to retain and which it can safely delegate.

Discovery is not authority

Once the API instances have been placed, clients need a stable way to reach them. A public load balancer can terminate or pass through TLS and select healthy targets. Internal callers may use DNS names, a service registry, a gateway, or client-side discovery. Workers need no inbound service address at all when the queue is their rendezvous point.

Discovery answers “where can I send this work?” Identity and authorization answer “which caller may ask this service to do it?” Network reachability is not proof of either. The image API needs a workload identity for object storage, the job queue, and the metadata database, each with only the operations it uses. The worker needs its own narrower identity. A parser defect in an untrusted image should not inherit permission to administer the cluster or read unrelated buckets.

The traffic contract should also say where timeouts live, which failures may be retried, how many retries fit the budget, and how an instance drains. During a rollout, discovery must stop new traffic before the old API process exits. For workers, the equivalent sequence stops new job claims, gives current work a bounded time to finish, and then makes unfinished work available again. A shutdown that merely waits forever is no safer than one that kills work immediately.

Control-plane dependencies deserve separate attention. DNS, registries, identity systems, secret stores, orchestration APIs, and cloud APIs may not serve every image request, yet their failure can prevent new instances from starting, credentials from refreshing, capacity from scaling, or a bad release from rolling back. A running API should not consult the deployment service on every request. An emergency path should not rely exclusively on the same control plane that is failing.

Configuration turns into behavior

The same image should move through environments unchanged. Values that vary— storage locations, scanner endpoint, timeouts, feature flags, concurrency, resource limits, and secret references—are runtime configuration. They should be explicit, validated before the process declares itself ready, auditable when changed, and safe to roll back. Secrets require narrower storage, access, redaction, rotation, and revocation than ordinary configuration.

Resource declarations are part of configuration too. If the worker requests too little memory, the platform may repeatedly kill it on large images. If it requests far more than it uses, the scheduler may leave capacity idle. CPU limits can change latency under contention. A configuration value that says MAX_WORKERS=60 is therefore not housekeeping; it is a production decision about pressure on every downstream service.

Health signals must match the decision they drive. A startup check answers whether initialization completed. A readiness check answers whether this instance should receive new traffic or work. A liveness check asks whether restarting this process is likely to repair it. Shutdown behavior determines whether current work can finish or be retried safely.

Combining those questions in one shallow /health endpoint creates surprising control loops. If liveness fails whenever the scanner is slow, every worker may restart during a scanner incident and add more retries. If API readiness fails for every replica whenever the database is briefly unavailable, the load balancer may remove all capacity and hide any degraded response the service could have offered. Conversely, a check that proves only that the process can return 200 may send traffic before configuration, migrations, or required local state is usable. Define health from the action the platform will take, not from the convenience of one endpoint.

Scale the bottleneck, not the symptom

Return to the sixty workers. The queue’s age was a useful signal: customers were waiting. The scaling policy failed because worker count was not the only capacity constraint.

Suppose one worker can finish four images a minute under the representative file mix, while the scanner safely accepts 80 concurrent scans and the metadata database allows 100 worker connections. The system cannot infer a safe fleet size from queue depth alone. It also needs per-worker concurrency, downstream budgets, startup time, retry behavior, and enough reserved capacity for the public API.

A better controller can use oldest-job age and completion rate to decide that more capacity is useful, then clamp the result to the scanner and database budgets. Workers use a bounded connection pool and a bounded number of active jobs. Retryable failures back off instead of immediately multiplying load. If the scanner is saturated, the system may defer low-priority catalogue work while keeping interactive uploads moving. Scale-in stops job claims and drains or releases leases before terminating processes.

Different workload shapes require different signals. Request concurrency, tail latency, or active connections may reveal pressure in the HTTP API. Queue age and completion rate reveal pressure in workers. Partition lag matters to a stream processor. Remaining work divided by the completion window matters to a batch job. Memory occupancy or accelerator residency may dominate an inference service. CPU is useful when CPU is the constraint; it is not a universal proxy for demand.

Autoscaling also arrives late. Metrics are sampled and smoothed. A VM must boot, or a container must be placed and pull its image. The process then loads configuration, establishes connections, warms caches, and joins discovery. Capacity planning still needs a warm minimum, headroom for ordinary variance, and an overload policy for demand that grows faster than capacity can arrive.

The revealing question in a design discussion is: if ten new instances appear, what receives ten times the pressure? A database, cache, registry, model store, secret manager, third-party API, or shared lock may move the bottleneck rather than remove it. Sometimes the right response is more instances. Sometimes it is backpressure, admission control, batching, a smaller unit of work, or an explicit refusal to accept work the system cannot complete.

Replicas fail together more often than diagrams admit

Three API replicas on one host are protected from a process crash, but not from host loss. Three hosts in one availability zone may survive a machine fault but share power, networking, storage, or capacity risk. Replicas in several zones may still share one database, identity provider, account quota, registry, configuration value, or deploy wave.

A failure domain is any boundary inside which one fault can affect several components at once. For the image service, trace these domains explicitly:

• a process can crash or leak memory;
• a host can fail or be drained;
• a node pool can run out of capacity;
• a zone can lose a shared infrastructure dependency;
• a region can become unreachable;
• the scanner, queue, database, object store, identity service, or registry can degrade;
• one bad secret or configuration can reach the whole fleet;
• one deployment can replace every good instance;
• one cloud account or project can exhaust quota or receive a destructive policy change.

Placement reduces some of these risks. API and worker replicas can be spread across hosts and zones. Bulk workers can use a separate pool and quota so that a catalogue import cannot evict the public API. Multiple regions may be justified when the product’s recovery promise and data model support them. None of that removes shared dependencies; the design must state what degrades when each one is unavailable.

For example, if the scanner is down, the API may continue accepting durable uploads while publishing a delayed-processing status, provided the queue has a bounded capacity and the product can honor that promise. If object storage cannot preserve the original, the API must not acknowledge success. If the metadata database is unavailable after an image has been written, an idempotent reconciliation path can repair the missing record. These are useful reliability claims because they name the surviving operation, not just the number of replicas.

Deployment is another failure domain

The revised worker image changes its scanner client and concurrency controller. Replacing all workers at once would put the old and new assumptions beyond comparison. Instead, send a small share of representative jobs to the new version. Compare completion rate, oldest-job age, scan errors, retry volume, database pressure, memory use, and output correctness. Decide the rollback thresholds before exposure grows.

A rolling deployment limits simultaneous replacement when readiness is trustworthy. A canary limits exposure while behavior is compared. Blue-green deployment can make traffic switching fast, though shared data migrations may still resist rollback. Shadow traffic can compare read-only behavior if its side effects are isolated. A feature flag can separate deployment from activation, but the flag itself needs an owner, safe default, audit trail, and removal plan.

Whatever the mechanism, the unit of rollout should match the suspected failure domain: percentage, host pool, zone, region, tenant group, or job class. Platform status is not the final evidence. A rollout can converge perfectly while user latency, queue age, or output correctness gets worse.

Explain the contract before naming the product

In an interview, a runtime answer becomes credible when another engineer could operate from it. For the image service, a concise version might be:

The public API is a warm, long-lived request service; the processor is an idempotent queue worker; originals and derivatives live in object storage, and metadata lives in a managed database. I would deploy immutable API and worker containers across hosts and zones, with separate identities, quotas, health behavior, and rollout policies. The worker fleet scales from queue age and completion rate, but scanner concurrency and database connections cap it. A scanner outage delays processing rather than losing accepted uploads, and a canary rolls back on correctness errors, retry amplification, or growing queue age.

The platform name can follow. If a managed application platform supplies those properties with less operational burden, use it. If the organization already operates an orchestrator and needs mixed workloads, placement policy, and bounded rollouts, use that capability. If each job is short, bursty, and fits the event runtime’s limits, a serverless worker may be simpler. If native libraries, accelerators, long execution, or stable warm capacity dominate, a container pool or VM fleet may be the clearer boundary.

Now change one assumption: each worker loads a 12 GiB model and needs several minutes to warm. Reconsider minimum capacity, placement, scale-in, rollout, and failure recovery. Then change another: regulations require originals to remain in one region. Reconsider discovery, failover, and what “available” can honestly mean. A memorized platform recommendation breaks under those changes; an operating contract adapts.

The sentence worth carrying into a design is not “use containers.” It is this: when the workload starts, moves, scales, drains, or fails, the platform and the application must agree about who owns the work, the state, and the blast radius.

Related foundations

• Security and Privacy
• Observability, Performance, and Cost
• Distributed Systems Fundamentals