Infrastructure Primitives

Small interfaces, large promises

Generate an ID. Decide whether a request is allowed. Return a cached value. Find a service endpoint. Deliver configuration. Evaluate a flag.

Each instruction sounds small enough to solve with one familiar product. That is what makes infrastructure-primitive prompts revealing. The primitive will sit beneath other services, often on their hot path, and its callers will build their own behavior around whatever it promises. A vague answer does not stay vague for long; it becomes somebody else’s outage.

The answer begins with an invariant, but an invariant alone is only a slogan. It has to determine where decisions happen, who owns the state, and what the system refuses to do when it can no longer protect the promise.

For a unique-ID generator, collision freedom is non-negotiable while global ordering may be optional. A cache can serve some values stale, but must not cross a tenant or permission boundary. Discovery can return a recently dead endpoint if clients have bounded retries; it must not pretend that membership is perfect knowledge. Configuration can keep the last-known-good version during an outage, while a malformed new version must not spread. A flag may be slightly stale but must bucket the same subject predictably.

The useful opening is therefore not a component list. It is a short contract:

I want to establish the invariant, identify the caller and hot path, and define
degraded behavior. Then I can place state and coordination around that promise.

That sequence keeps a design honest when the interviewer changes the scale, region, strictness, or trust boundary.

Four decisions before architecture

First, name the subject of the decision. A limiter for an IP address is not the same product as a tenant quota. A cache of public catalog descriptions is not a cache of authorization decisions. A flag evaluated for an anonymous browser cannot receive the same rule bundle as a flag evaluated inside a trusted service.

Next, locate the decision. If the primitive is consulted on every request, a remote call may turn it into both a latency tax and a shared failure domain. Local libraries, gateway plugins, sidecars, agents, and cached bundles trade central control for bounded staleness and client-version drift. Say which cost the product can carry.

Then identify the minimum coordinated state. Coordination is sometimes the mechanism that protects the invariant: a worker lease prevents two ID generators from claiming the same identity; an atomic counter can enforce a strict quota. But coordinating every read across regions is not a mark of seriousness. Keep the strongly owned state as narrow as the promise allows.

Finally, choose failure behavior before drawing replicas. “Highly available” does not answer whether a fraud-sensitive limiter fails closed, a catalog cache serves stale, or a configuration client keeps its last-known-good version. The caller experiences those policies, so they belong in the API contract.

The control-plane/data-plane split often emerges from these decisions. The control plane owns definitions, versions, leases, validation, authorization, review, and audit. The data plane performs the frequent decision, preferably from regional or local state. Not every primitive needs a grand platform, but this split is a useful consequence when safe writes and fast reads have different needs.

Worked prompt: design a rate limiter

Suppose the interviewer says, “Design a rate limiter for our APIs.”

Do not choose an algorithm yet. “Our APIs” leaves the decision almost wholly undefined. Ask who is limited, which actions consume capacity, whether the limit represents abuse protection or a purchased quota, and how much temporary over-admission is acceptable. Ask where traffic enters and whether enforcement must span regions.

Imagine the interviewer narrows the prompt:

Protect an expensive search API used by many business tenants. Each plan has a requests-per-minute allowance, brief bursts are acceptable, and a single tenant must not exhaust the search cluster. Traffic enters through several gateways in each region. During a control-plane outage, existing limits should continue to work.

Now the invariant has shape: one tenant must not consume more than its allowed share for long enough to endanger the cluster or violate plan policy. A tiny amount of regional over-admission may be acceptable; a design that requires a cross-region round trip for every search is not.

Put the decision near admission

Each gateway can evaluate a token bucket keyed by tenant and route. A bucket permits bursts up to its capacity and replenishes at the plan’s sustained rate. The control plane stores plan rules, tenant overrides, effective versions, and audit history. Gateways receive versioned rules and keep them locally, so a control-plane interruption pauses changes rather than stopping searches.

Local buckets alone would let every gateway spend the tenant’s full allowance. The design needs a way to divide budget. The control plane gives each region a disjoint, time-bounded envelope derived from the tenant’s allowance and recent demand. A regional allocator subdivides its envelope into small, expiring token grants for gateways. The gateway spends those tokens without a network hop on each request and renews before exhaustion. Unused authority expires or is explicitly reallocated; two regions never receive the same slice. The coordinated state is not a counter for every request but the slower allocation of bounded authority.

operator -> validated plan rule -> regional budget envelope
                                      |
                                      v
gateway <- expiring token grant <- regional allocator
   |
   +-> allow search
   +-> reject with retry guidance
   +-> use a deliberately bounded degraded budget

The lease size controls the trade-off. Large grants reduce allocator traffic but allow more overshoot when gateways fail or regions become isolated. Small grants enforce the allowance more closely but make the allocator hotter. The right size depends on request cost, plan semantics, and the amount of over-admission the search cluster can absorb.

If the interviewer requires an exact global billing quota, this architecture may be insufficient. A single authoritative counter or reservation service can provide stricter accounting, at the cost of latency and availability. That is not a patch to mention quietly: it is a changed product promise and should change the design aloud.

Make retries and policy changes visible

The decision API should return more than a boolean. It needs the policy scope, remaining or approximate budget when safe to expose, retry guidance, rule version, and whether fallback was used. Operators need an explanation path: given a tenant, action, region, and time, show which rule applied and why the request was allowed or rejected.

New rules should support shadow evaluation before enforcement. If a plan change would reject an unexpected share of traffic, the team should see that before customers do. Production overrides need ownership, expiry, and audit; otherwise an emergency exception becomes the permanent policy.

Failure behavior should follow the action rather than one platform-wide default. The expensive search route can spend a small cached emergency budget while the allocator is unavailable, then reject when that budget is exhausted. A low-cost read might fail open while downstream circuit breakers protect the service. Login abuse or payment attempts may require conservative rejection or a secondary check. “The limiter fails open” is rarely a complete policy.

Let one hot tenant challenge the partitioning

Suppose one tenant produces 40 percent of traffic. Sharding by tenant still puts that tenant’s allocation state on one shard. Possible responses include a dedicated allocator partition, sub-allocation by route or gateway group, hierarchical limits, and tenant-dedicated capacity. Each answer changes the accuracy or isolation story. Splitting a counter without explaining how the pieces remain bounded only moves the hot key.

Operate the limiter by observing both decisions and what they protect:

• allowed, rejected, shadow-rejected, and fallback decisions by rule and route;
• grant exhaustion, renewal latency, allocator errors, and overshoot estimates;
• hot tenants, high-cardinality growth, and rule-propagation lag;
• search-cluster saturation, latency, and load shed alongside limiter activity;
• override use, rule versions in service, and gateways running old clients.

A limiter dashboard full of healthy request rates can still conceal a search cluster being overwhelmed. The protected dependency completes the evidence.

Transfer the reasoning, not the components

The rate limiter supplied a pattern: define the permission to act, distribute bounded authority toward the hot path, and make degraded behavior explicit. The other primitive prompts preserve that discipline but change the promise.

Distributed unique-ID generator

The hard promise is no collision within a declared namespace. Ordering, compactness, opacity, and index locality are separate requirements; do not silently bundle them into “unique.”

At modest single-region scale, a database sequence or central service may be the best design because it is simple and easy to audit. At higher throughput, a control plane can assign disjoint worker identities or sequence ranges and let workers generate locally. An ID may combine time or logical time with a region identifier, fenced worker identifier, and per-tick sequence when rough ordering is useful.

The design becomes real at restart and clock rollback. A worker must not emit under an expired lease or reuse an identity still owned by another process. It tracks its last emitted time; on backward movement it can wait through a small rollback, use a reserved logical sequence if the format supports one, or stop and renew ownership. “NTP will fix it” does not protect collision freedom.

Active-active regions can receive disjoint region identifiers or ranges. If the prompt requires a single globally monotonic sequence, say what that costs: coordination moves into the hot path, so latency rises and partition availability falls. Collision alarms, lease conflicts, clock drift, sequence exhaustion, rejected generation, and worker-version distribution are the signals that test the actual promise.

Distributed cache

A cache promises faster or cheaper reuse, not truth. Begin with the authoritative store and name the data’s staleness tolerance. Catalog copy can often survive a longer TTL. Permission and revocation decisions may need a very short lifetime, versioned invalidation, or no shared cache at all.

Cache-aside is a reasonable baseline when a service owns loading. Key design must include tenant and permission-relevant scope; values should carry a serialization version and bounded size. Jittered TTLs, per-key request coalescing, refresh-ahead for known hot keys, and safe stale-while-revalidate behavior keep an expiry event from becoming an origin outage.

The revealing failure is not a cache node restart. It is the cache returning a plausible value that is unsafe: another tenant’s object, a revoked permission, or known-bad data after a correction. Operate the design with hit and miss latency by key family, origin load, invalidation lag, stale serving, hot-key skew, evictions, and memory cost. A high hit rate is not a success if it hides dangerous staleness.

Service discovery

Discovery promises endpoints likely to serve a request. It cannot provide perfect membership knowledge in a changing network. The design must therefore pair registry freshness with client behavior.

An orchestrator or registration API can publish instance identity, address, zone, version, capability, readiness, and lease or heartbeat state. Clients consume that through DNS, a gateway or sidecar, or a watched local cache. DNS is broadly compatible but coarse; client libraries allow richer routing but drift across languages; sidecars centralize policy but add an operating layer.

When a process dies, some client will briefly retain its endpoint. Short connection timeouts, bounded and safe retries, circuit breakers, and jittered refresh make that tolerable. During a registry partition, aggressively removing every unseen instance may create a false outage; retaining the last-known set indefinitely may route to dead or reassigned addresses. Choose a lease and stale window that match the runtime, then measure empty results, client cache age, unhealthy routed traffic, heartbeat lag, churn, retry rates, and traffic distribution.

Discovery metadata can support compatibility-aware routing, but it should not quietly become the entire deployment system. Rollout policy and rollback need their own owner.

Configuration service

Configuration changes production behavior without a deployment. Its promise is safe, attributable, reversible change—not arbitrary key-value access.

The control plane owns schemas, versions, environment and tenant scope, authorization, review, diffs, and audit. Clients poll, watch, or read through a local agent, then report the version they actually loaded. Polling is simple and naturally bounded but slower for urgent changes. Streams propagate faster at the cost of fan-out, ordering, and reconnection complexity. A local agent reduces duplicate work while adding another cache boundary.

Bad configuration deserves more attention than control-plane downtime. Schema checks catch type and range errors; semantic checks catch broken references and incompatible combinations. Roll a risky version through one cohort, tenant, or region before broad release. Clients reject invalid updates and retain a last-known-good version. Rollback names an earlier immutable version rather than attempting to reconstruct old values by hand.

Ordinary configuration should reference secrets rather than become a casual secret-distribution system. Secret storage adds stricter access, encryption, rotation, exposure, and audit requirements. For configuration itself, watch publish failures, propagation lag, stale clients, version distribution, rollback frequency, and last-known-good use. “Published” and “running everywhere” are different states.

Feature-flag platform

A feature flag resembles configuration until evaluation enters the hot path. The promise is predictable targeting: the same relevant context and rule version should yield the intended variant, while rollout and kill-switch changes propagate within a known bound.

The control plane stores flag ownership, environment, variants, targeting rules, approvals, expiry, and audit. Trusted server SDKs can evaluate locally from a versioned rule bundle. Browser and mobile clients should receive only rules and attributes safe to expose; sensitive targeting remains server-side. Central evaluation keeps one implementation but adds a network dependency to every decision.

Percentage rollout needs stable bucketing. Hash a stable subject identifier with the flag key and a deliberate salt so a user does not flicker between variants. Changing the salt, bucketing unit, or rule order can move cohorts and damage an experiment even when every service is healthy.

Exposure logging belongs off the request path. Kill switches need a narrow, tested propagation path and a defined stale fallback. Flags also need owners and expiry because permanent flags leave dead branches and operational ambiguity in application code. Measure rule-bundle versions, propagation lag, default fallbacks, evaluation errors, cohort stability, exposure-event lag, kill-switch latency, and overdue cleanup.

Pressure reveals the contract

After presenting a baseline, invite one constraint that can overturn it. A useful follow-up changes a promise, not merely the number of replicas.

Try these aloud:

• The ID must now be globally monotonic, not merely collision-free. Where does coordination move, and what happens during a partition?
• The limiter now protects payment attempts rather than search. Which degraded behavior changes, and why?
• A cached value controls authorization. What can still be reused safely?
• The discovery registry is healthy, but half the clients hold an old endpoint. Which mechanism and signal were missing?
• A valid configuration causes an incident. Which semantic check, cohort, and rollback path limits the damage?
• A flag changes its bucketing key midway through an experiment. What user and analytical behavior follows?

Record a ten-minute answer to any one prompt. On playback, find the first sentence that commits to the invariant, the owner of coordinated state, the hot-path decision, and the degraded mode. Then introduce one of the pressures above. If the architecture changes, carry that change through the API, consistency choice, operating signals, and summary. If it does not change, decide whether the design is robust or whether the original commitment was too vague to be challenged.

The transferable sentence is short:

Name the invariant, place the decision, own the state, define failure behavior.

Its value lies in the consequences. Infrastructure primitives disappear into the systems above them when they work, but their contracts do not. Every caller eventually discovers exactly what the primitive meant by unique, allowed, fresh, healthy, current, or enabled. A senior design makes that meaning explicit before production has to.

Related links

• Caching and Content Delivery
• Distributed Systems Fundamentals
• Scaling and Partitioning
• Consistency, Idempotency, and Transactions
• Reliability and Failure Engineering
• Security, Abuse, and Privacy
• Observability and Operability