Infrastructure, DevOps, and SRE

The patch is one line; the change is not

Imagine this system-design prompt:

Two hundred services run on managed Kubernetes. A security requirement gives
you thirty days to replace every node with a patched image. Product teams
deploy independently. Some workloads have inaccurate resource requests, weak
disruption budgets, long-lived connections, and only host-level alerts.
Design the migration without stopping delivery.

The visible diff might change one image identifier in a shared Terraform module. The operating change is larger. New kernels and runtimes meet workload assumptions that nobody wrote down. Draining a node alters placement, endpoint membership, connection state, and spare capacity. A rollout that is safe for a stateless API may stall a singleton controller or evict too many replicas of a service whose disruption budget describes hope rather than reality.

This is the distinctive pressure in infrastructure, DevOps, and SRE interviews. Services, data paths, APIs, and code still matter, but the candidate must reason about a change shared by many owners. A platform abstraction concentrates leverage and blast radius at the same time.

The names of the tools cannot complete that argument. “Use Terraform,” “roll the nodes,” or “put a canary in one cluster” leaves the hard questions open: what the canary represents, which signal stops expansion, whether rollback can repair what already happened, and who decides when product risk now exceeds security risk.

Give the migration a promise

Start by defining success in terms of the people and systems that depend on the platform. For this migration, the promise might be:

• every in-scope node runs the patched image before the deadline;
• customer journeys remain within their existing SLOs during the change;
• teams retain a working path to deploy urgent application fixes;
• no workload loses durable data or violates an availability requirement because of eviction;
• the platform can stop, diagnose, and recover at each rollout boundary.

These conditions are more useful than “zero downtime.” That phrase hides which journeys matter, how much impairment is acceptable, and how it will be measured. A checkout SLO, a build-queue wait target, or a control-plane availability promise gives the interviewer something concrete to challenge.

The deadline introduces a real trade-off. Moving too slowly leaves known security exposure. Moving too quickly converts a security repair into an availability incident. A senior answer does not dissolve that tension with “safety first.” It makes scope, exception authority, rollout evidence, and the cost of delay explicit enough for a responsible decision.

Find where one change crosses a boundary

Before choosing a rollout rate, draw the boundaries through which failure or authority can travel.

At the cloud boundary, regions and availability zones determine whether a bad image or a capacity shortage remains local. Instance quotas, IP space, load-balancer targets, storage attachment, identity, and managed-control-plane limits can prevent the replacement pool from existing even when the Terraform plan is valid.

At the Linux boundary, the kernel, container runtime, cgroups, process signals, file descriptors, memory pressure, CPU throttling, disk I/O, and sockets determine what a workload experiences. An OOMKilled container is evidence of a decision made at this layer; it does not say whether the cause is a leak, an inaccurate limit, changed accounting, or a burst the service was supposed to absorb.

At the orchestration boundary, requests and limits affect placement, readiness controls endpoint membership, disruption budgets constrain voluntary eviction, affinity spreads or concentrates replicas, and autoscaling consumes the remaining headroom. The scheduler can honor declared constraints while still producing a bad result if those declarations are wrong.

At the network boundary, DNS, connection draining, load-balancer health, service discovery, timeouts, retries, TLS, and connection pools decide whether traffic follows the new placement cleanly. Reachability is not correctness, and a successful health check is not proof that a long request can finish.

At the ownership boundary, the platform team owns the migration mechanism but not every workload invariant. Service teams know which jobs are safe to interrupt, which queues can be replayed, which local volumes matter, and which release is urgent. Security owns the risk that created the deadline. Product and incident leaders own consequences that no single dashboard can settle.

This boundary map is a practical interview habit. It keeps “infrastructure” from shrinking into cluster configuration and shows where the answer needs an owner, an invariant, or a recovery path.

Build evidence before movement

A canary is useful only if failure will become visible. Before replacing a node, establish the evidence that describes normal behavior and the signals that will govern expansion.

Begin with user and platform promises: success rate and tail latency for critical journeys, queue age for asynchronous work, deploy success and wait time, and error-budget burn. Add the mechanism evidence needed to explain a change: node readiness, unexpected reboots, pod evictions, pending pods, restart and OOMKilled rates, throttling, disk and network errors, endpoint churn, connection resets, DNS failures, and capacity by zone. Keep deploy, configuration, image, and cluster events on the same timeline.

“CPU is normal” cannot clear the migration. Checkout can be failing while CPU is low because no consumer can acquire a database connection, retries are amplifying a slow dependency, endpoints disappeared during a drain, a queue is aging, or TLS handshakes changed. Host health explains part of the mechanism; the SLI tells you whether the promise survived.

Inventory is evidence too. Find workloads with one replica, local or attached state, restrictive affinity, missing requests, impossible disruption budgets, long termination periods, unusual architectures, privileged access, or no clear owner. Do not quietly “fix” all of them inside the patch rollout. Classify which conditions block migration, which can be constrained by a temporary policy, and which become follow-up work with named owners.

Make the first consequential action small

The migration can now proceed as a series of claims that evidence may confirm or reject.

1. Build the patched image and node-pool definition through the normal CI/CD path. Pin inputs, preserve provenance, scan the artifact, review the IaC plan, and make the promotion record auditable.
2. Create a separate canary pool rather than mutating every existing pool in place. Limit its region, zone, cluster, node count, permissions, and eligible workloads.
3. Place representative workloads on it deliberately. Include a request-heavy service, a queue consumer, a stateful or storage-attached workload where permitted, a daemon on every node, and a service with long-lived connections. An empty canary proves only that a node can join.
4. Exercise ordinary work, scaling, restart, drain, and rollback. Observe long enough to cover traffic cycles and scheduled jobs that a five-minute smoke test would miss.
5. Replace a small batch in one failure domain. Cordon old nodes so new work stops arriving, drain within disruption and termination constraints, and verify the replacement capacity before destroying the old pool.
6. Expand by explicit gates: one zone, then a small part of a region, then a larger cohort. Pause on SLO burn, unexpected eviction, pending workloads, capacity loss, or unexplained network and runtime changes.
7. Keep an exception path for workloads that cannot yet move. The exception needs an owner, compensating control, expiry, and migration plan; otherwise it becomes the permanent end of the program.

This sequence separates creation, scheduling, eviction, and destruction. That separation matters because rollback has limits. You can direct workloads back to the old pool while it still exists. You cannot use a Terraform rollback to restore an interrupted non-idempotent job, undo a duplicated external effect, or recover evidence discarded during a hurried drain.

Capacity is part of the control plane for the change. The cluster needs enough headroom to hold old and new nodes during replacement, absorb an unavailable zone, and reschedule disrupted work. Aggressive bin-packing may save money in steady state and remove the exact margin the migration needs. State how much surge capacity you will buy, where quotas may block it, and when it can be released after the patched fleet is stable.

When the canary is green and checkout is not

Now perturb the prompt. During the first regional batch, host CPU remains normal. Checkout p99 and errors rise, queue age climbs, and several newly placed pods restart. The infrastructure answer becomes an incident answer.

First, treat the customer signal as authoritative enough to halt expansion. State the affected journey, regions, error-budget burn, and current scope. If the impact warrants an incident, separate incident command, technical work, communication, and evidence capture so that the person debugging is not also trying to coordinate every stakeholder.

Then compare the changed cohort with the control cohort. Ask:

• Did errors follow new nodes, one zone, one workload version, or the drain itself?
• Are pods restarting because of memory, failed probes, runtime errors, or termination behavior?
• Did endpoint churn or connection resets rise as nodes drained?
• Is queue age growing because consumers are absent, throttled, retrying, or blocked on a dependency?
• Did another config, application deploy, certificate, quota, or network change land in the same interval?

The comparison protects against two forms of thrashing: blaming the patch because it is salient, and ignoring it because the image passed a smoke test. Logs, metrics, traces, kernel and runtime events, deployment records, and customer reports are parts of one argument.

Mitigation comes before perfect explanation. Stop the rollout. Preserve the old pool. Move or reschedule the affected cohort if doing so is safe. Reduce retry amplification, shed noncritical work, restore queue consumers, or route traffic away from the impaired zone according to the evidence. After each control, watch checkout success and queue age rather than declaring recovery from pod count alone.

Suppose the investigation finds that the new runtime accounts memory differently, pushing one consumer past an unrealistically low limit. Raising the limit may be the right immediate repair if capacity permits, but it is not the whole mechanism. The follow-up includes a representative workload test, resource-policy review, capacity model, rollout check for restart rate, and a way to find similarly exposed services before expansion resumes. The finding must change the migration, not merely close the incident.

Automation should preserve the operator’s chance to think

Infrastructure coding exercises often look small: parse logs, validate a manifest, inspect certificates, clean up cloud resources, repair a pipeline, or write a drain helper. The senior bar is operational code whose speed does not multiply an unsafe assumption.

For this migration, an interviewer might ask:

Write a tool that identifies nodes eligible for replacement and drains them in
batches.

Begin by separating discovery, decision, and mutation. Discovery returns a fresh inventory with stable node identities, zones, workloads, and relevant constraints. A pure decision step rejects nodes whose batch would violate surge capacity, zone spread, a disruption budget, an allowlist, or a workload exception. Mutation receives the approved identities and rechecks critical preconditions before cordon or drain; the world may have changed since the inventory was read.

Make preview the default. Require explicit environment and cluster scope, bound concurrency and rate, emit an audit record, avoid secrets in output, and report succeeded, failed, skipped, and still-running operations separately. Cancellation should stop new work without abandoning operations already in an uncertain state. A rerun should reconcile current node and workload state instead of repeating every command blindly.

Tests deserve the destructive edges: a singleton workload, an impossible disruption budget, a node that changes between plan and apply, partial API failure, exhausted capacity, a stuck termination, and interruption halfway through a batch. These are not embellishments around the algorithm. They are the algorithm’s operating contract.

What the familiar rounds reveal

Infrastructure loops use familiar interview labels, but they move between layers unusually quickly.

In practical coding, expect a deploy helper, health check, log parser, inventory tool, configuration validator, metrics emitter, or repair to a flaky pipeline. Keep the program small enough to finish, but make idempotency, scope, partial failure, dry-run behavior, permissions, tests, and operator-readable output visible in the design.

In system design, a CI/CD platform, Kubernetes service, observability system, multi-region runtime, secrets service, or internal developer platform is a contract before it is a toolchain. Define what teams receive, which decisions the platform centralizes, where tenants and privileges are isolated, how change is promoted and reversed, how capacity is reserved, and how teams debug the abstraction when it leaks. Include adoption: a platform with no migration or support path is only an architecture diagram.

In debugging, move from user harm to recent change, saturation, and dependency boundaries. Linux commands, cloud consoles, Kubernetes events, logs, metrics, and traces are ways to test a hypothesis. Listing them without an order is not a diagnosis. Say what each observation would distinguish and which evidence would cause you to mitigate immediately.

In a project deep dive, prepare a platform migration, SLO program, CI/CD change, incident, cost reduction, automation project, capacity correction, or on-call reform. Give the before state, the risk or toil, the constraints, the sequence of change, the control you kept, the adoption and support burden, and the outcome. If the first rollout failed or the platform became a bottleneck, do not edit away the useful part of the story: explain how the mechanism and ownership model changed.

In behavioral and senior screens, reliability conflicts are rarely solved by technical authority alone. An unsafe launch, disputed severity, noisy pager, security exception, platform mandate, or cost reduction needs a specific consequence and a decision owner. Show how you made risk legible, offered a smaller or staged path, escalated when the decision exceeded your authority, and avoided making the platform team a permanent gatekeeper.

SLOs, cost, and on-call are one operating system

An SLO earns its place when it changes action. Choose an SLI tied to a user journey or platform promise, set a target that reflects business tolerance, page on urgent symptoms a responder can influence, and use error-budget burn to govern rollout pace. A dashboard that nobody trusts and an alert that has no useful control are not reliability mechanisms.

Capacity and cost belong in the same decision. Right-sizing can remove burst headroom. More aggressive autoscaling can oscillate or overload a dependency. Consolidating clusters can save operating cost while enlarging the failure domain. Start cost work with attribution by service, tenant, environment, or workload; then name the SLO, recovery objective, or failover margin that must not regress. Stage the change and keep a rollback threshold.

Human capacity is finite too. A noisy alert consumes attention whether or not it appears on a cloud bill. Strong answers connect paging thresholds, ownership, runbooks, escalation, post-incident learning, and toil reduction. They also recognize a failure mode in SLO programs: if honest measurement is used to punish teams, teams will defend the number instead of improving the system.

Failure patterns worth interrupting

Listen for the sentence in your answer that closes a question too early:

• “We use Kubernetes.” Scheduling does not define tenancy, networking, identity, rollout, observability, capacity, or support.
• “Just roll back.” Rollback may not undo changed data, completed external effects, drained work, client behavior, or a dependency failure.
• “Retry with backoff.” Retries can deepen an outage or duplicate an uncertain side effect without budgets, idempotency, and reconciliation.
• “Automate it.” Broad credentials and unbounded concurrency turn a fast script into a fast incident.
• “The hosts are healthy.” Customers experience journeys, queues, dependencies, and tail latency rather than fleet averages.
• “The SLO is on the dashboard.” A target matters when it changes paging, release pace, capacity, or ownership.
• “We will mandate the platform.” Teams need migration help, debuggability, support boundaries, and a principled escape path.

Each correction reopens the operating question: what promise, evidence, control, or owner is still missing?

Practice one change until it becomes difficult

Use the node-image migration as a rehearsal, then vary one constraint at a time:

1. Mark the promise, failure domains, owners, canary cohort, stop signals, capacity margin, rollback limit, and exception process.
2. Add a singleton controller with local state. Decide whether to redesign it, protect it temporarily, or grant a time-bounded exception.
3. Remove spare capacity in one zone. Recalculate rollout order and explain what cost you would accept to preserve recovery.
4. Make the new image pass smoke tests but increase connection resets under long-lived traffic. Identify the evidence that reveals the defect.
5. Let the security deadline arrive while one critical workload still cannot move. State who owns the residual decision and what compensating controls are credible.
6. Design the drain helper. Test stale inventory, partial failure, cancellation, bounded concurrency, and safe rerun.

Rehearse two neighboring prompts with the same discipline. For CI/CD, define artifact trust, promotion, permissions, policy gates, canaries, rollback, auditability, and the team’s escape path. For multi-region service runtime, define the product flow, data consistency, dependency locality, capacity, failover trigger, recovery objective, and drill. The tools will differ; the quality of the reasoning should transfer.

After each attempt, identify the first place you relied on a noun instead of a mechanism: “the platform,” “observability,” “automation,” “the cloud,” or “rollback.” That gap is the next practice assignment.

A compact infrastructure answer frame

When a prompt sprawls, make these lines specific:

Promise and SLI I am protecting:
Failure, privilege, traffic, and ownership boundaries:
Recent change, saturation, dependency, and customer evidence:
Smallest cohort and the signal that stops expansion:
Immediate control and what it cannot undo:
Durable mechanism, owner, and evidence of improvement:
Capacity, cost, security, and human-load trade-off:

You are ready when a small infrastructure diff no longer looks small. You can follow it from source review through artifact promotion, cloud capacity, kernel and runtime behavior, scheduling, traffic, user impact, incident control, and team ownership without losing the governing promise. That is what interviewers are trying to sample: whether shared systems become more legible and recoverable when you change them.

Related links

• Networking and the Web
• Cloud, Containers, and Runtime Platforms
• Observability, Performance, and Cost
• CI/CD and Release Engineering
• SLOs and Operational Readiness
• Incident Response