Admission Control, Rate Limiting, and Load Shedding

At 4,000 modeled work units per second, Mercury completes 3,900 useful units. At 5,000 offered units, it completes 3,700. At 8,000, it completes 1,900. At 10,000, it completes 1,200.

The request counter still rises. Every connection is accepted. Workers are busy. Yet useful throughput has fallen to 30% of the system’s known 4,000-unit capacity. The missing work did not vanish: it became queue scans, timeouts, cancellations, retries, cache churn, partial downstream calls, memory pressure, failure handling, and responses that arrived after their value expired.

With admission enabled in the same deterministic model, goodput stays near 3,980 units/s at the highest offered load. The system is not doing more work. It is refusing work that has become unlikely to complete usefully, before that work consumes the scarce resource.

The controlling thesis is: protect the rate of useful completions, not the rate of accepted traffic. Admission control converts an overload contract into an early decision. Rate limiters, concurrency bounds, queue policies, reservations, and degradation are instruments for that decision; none is the objective by itself.

Read the collapse curve before choosing a limiter

The modeled sweep is deliberately simple:

These are modeled teaching data, not a universal curve. A real service can plateau rather than collapse, fail abruptly on memory, exhaust a connection pool at low CPU, or violate correctness before throughput changes. Reproduce the actual curve with offered load, accepted load, started work, useful completions, errors, deadline misses, resource demand, queue age, and recovery time on the same axis.

Define goodput at the business boundary. It may be correct responses before deadline, committed durable writes, unique events incorporated, frames rendered at required quality, or queries returning an acceptable result. A 200 response after the client deadline is not useful completion. A duplicated side effect is not two units of goodput. A degraded result counts only if its quality contract says it is useful.

Admission begins at the last point where rejection is still cheap. That can be an edge gateway before authentication-expensive parsing, a service before allocating a large buffer, a worker before a scarce accelerator, or a storage engine before starting a transaction. Rejecting earlier saves more work but has less local information. Layer gates so broad abuse/policy checks happen early and resource-specific protection happens near the constraint.

Capacity protection and policy enforcement are different jobs

A policy limiter enforces entitlement: a customer may call 200 times per second, a tenant may consume a purchased quota, or an unauthenticated client may receive a small abuse budget. A protection gate keeps a dependency, pool, or service inside a safe operating region. The same token bucket can participate in both, but the semantics differ.

Policy enforcement asks:

• Who is entitled to what rate, burst, or share?
• Over which identity, method, region, and time window?
• Is the decision billable, auditable, or security-sensitive?
• What consistency does the entitlement require?

Capacity protection asks:

• Which resource is currently constrained?
• How much demand does this request add?
• Can it finish before its deadline?
• Which useful class should survive overload?
• What rejection or degradation keeps recovery possible?

A fixed 10,000 requests/s tenant quota does not protect a service if requests become 20× more expensive. A local concurrency bound protects one worker but may violate global tenant fairness. A globally consistent quota store can enforce entitlement precisely yet become a mandatory dependency whose latency or partition blocks the service. Keep enforcement records and protection signals separate even when one component evaluates both.

Limiter mechanisms encode different burst and delay contracts

Token bucket

A token bucket replenishes at rate (r) tokens/s up to capacity (B). An operation costing (c) tokens is admitted when at least (c) tokens exist. It permits bursts up to the stored balance while constraining the long-run rate. Tokens can represent requests, bytes, predicted CPU milliseconds, database units, or another declared demand.

The fixture uses 200 tokens/s, 400 burst tokens, and one token/request. A full bucket admits 400 requests immediately and takes two seconds to refill from empty. That is a traffic contract, not a capacity proof. If each admitted request fans out or request cost changes, the token price or downstream protection must change.

RFC 2697 specifies a particular single-rate three-color marker in byte units with two buckets. It is evidence that token-bucket semantics require explicit rate, bucket size, unit, initialization, and color behavior—not a prescription for application APIs.

Leaky bucket and paced release

“Leaky bucket” is used for related but not identical algorithms. In a queueing form, arrivals enter a bounded bucket and depart at a steady rate, smoothing bursts but adding delay. In meter descriptions it may resemble a continuously draining counter. A design record must state whether excess arrivals wait, are marked, or are dropped; maximum queue length; and the latency cost of smoothing.

Pacing is valuable when downstream service prefers regular arrivals. It is harmful when a deadline-bound request waits behind work that will not finish in time. A rate regulator can turn a burst into a long queue unless the queue also expires and rejects.

Fixed and sliding windows

A fixed-window counter is cheap and understandable: allow (N) operations per named interval. It permits boundary bursts—up to (N) just before reset and (N) just after. A sliding log precisely retains recent event timestamps at higher memory and update cost. Sliding counters approximate the window with buckets, trading accuracy for bounded state.

Window choice is part of the promise. “1,000 per minute” does not define whether 1,000 can arrive in one millisecond, how clocks align, what happens across regions, or whether rejected attempts count. Do not use a billing-grade distributed window on the hot path merely to protect a local thread pool.

Concurrency limits

A concurrency limit bounds in-flight work. It naturally responds to service time: by Little’s Law, the same arrival rate consumes more concurrency when operations slow. That makes it useful for protecting pools and downstreams when rate alone is misleading.

Concurrency is still a proxy. Ten 2 GiB analytical queries and ten small reads are not equivalent. Blocked requests can occupy concurrency without consuming the resource being protected; asynchronous work can release a frontend slot while continuing downstream. Define when the permit is acquired and released, whether queued requests hold permits, and which work continues after client cancellation.

Use rate limits to shape arrivals and enforce time-based entitlement. Use concurrency limits to bound simultaneous occupancy. Compose them when both burst and residence time matter.

Put the gate where it can predict waste

Admission signals operate at different horizons:

The decision should compare the request’s probability of useful completion with an explicit threshold. A practical deadline test is:

[ Q_{pred} + S_{pred} + R_{remain} + U < D_{remain} ]

where (Q_{pred}) is predicted queue delay, (S_{pred}) is service demand, (R_{remain}) is response-path time still required, (U) is an uncertainty margin, and (D_{remain}) is the remaining end-to-end deadline.

The fixture supplies 45 ms queue age, 35 ms service, 10 ms response path, and 15 ms uncertainty against 80 ms remaining. The 105 ms predicted cost fails the test, so reject now. Waiting briefly is defensible only if an updated upper bound fits the deadline and the queue has a hard cap. If the same request arrived with 120 ms remaining, the inequality would permit admission under the model; it would not guarantee success.

Reject malformed, unauthorized, or impossible requests for their own semantics, not as load shedding. Under overload, ensure the rejection path is cheap and bounded: avoid expensive logging, large error bodies, remote quota transactions, and synchronous telemetry exports.

Local, distributed, and hierarchical control

A local limiter is fast and available with the process. It sees current worker pressure. Across (n) replicas, identical local rates can overshoot a global entitlement by roughly (n), and uneven placement can leave capacity unused on one worker while another rejects.

A strongly coordinated distributed limiter can make a precise global decision. It adds network latency, consistency cost, and a new failure dependency. A weakly consistent counter can remain available but overshoot or undershoot during delay and partition. Neither is universally correct.

Hierarchical control separates horizons:

service capacity budget
├── region safety budget
│   ├── premium class reservation
│   │   ├── tenant lease
│   │   └── per-worker concurrency gate
│   └── free class reservation
│       ├── tenant lease
│       └── per-worker concurrency gate
└── recovery and control-plane reserve

The global layer allocates coarse leases or budgets slowly. Regions and workers spend local allocations quickly. Unused reservations may be borrowable under explicit rules, with a revocation time that fits recovery. A worker still rejects when its actual constraint is reached, even if a global ledger says entitlement remains.

Partition limiter keys by the identity and resource that require isolation. A single global lock for all tenants creates a limiter-induced noisy neighbor. Conversely, purely per-tenant keys can let many tenants collectively overwhelm a shared dependency. Enforce both shared and tenant bounds.

Applied design: free and premium classes

Mercury’s modeled downstream capacity is 4,000 work units/s. Premium requests cost three units and arrive at 1,000/s; free requests cost one and arrive at 6,000/s. Offered demand is 9,000 units/s—more than twice capacity.

Reserve 2,400 units/s for premium and 1,600 for free:

• premium admits 800 requests/s × 3 units = 2,400 units/s;
• free admits 1,600 requests/s × 1 unit = 1,600 units/s;
• total admitted demand = 4,000 units/s.

This is a policy example, not a claim that premium should always receive 60%. The reservation must follow product promise, value, fairness, and capacity evidence. Preserve a small control/recovery reserve in a real design rather than operating at a modeled mathematical ceiling.

Borrowing improves utilization: if premium uses only 1,200 units, free traffic may borrow some slack. The borrow is preemptible. Define how quickly premium can reclaim it, how free queues are drained, and whether already-admitted work completes. Instant revocation can waste work; slow revocation can violate the premium promise.

Use separate class queues and limits so premium arrivals do not sit behind a large free backlog. Within each class, tenant weights or max-min-inspired allocation prevent one noisy tenant from taking the whole share. “Premium first” without a cap can starve free traffic forever. “Equal requests” is unfair when requests cost different resources. Allocate in normalized work units and audit actual service.

Fairness is an allocation policy, not a queue option

Max-min fairness is a useful mental model: increase each flow’s allocation together until one reaches its demand or a shared resource saturates, then continue among the rest. Weighted variants express differentiated entitlement. Real services complicate the model with multiple resources, deadlines, indivisible tasks, priorities, and burst credit.

Define fairness at the boundary where the promise is made:

• tenant share of useful capacity, not accepted requests;
• per-class minimum and maximum;
• burst credit and debt horizon;
• treatment of idle reservations;
• maximum starvation time;
• multi-resource accounting; and
• behavior during regional or dependency loss.

Priority should affect admission and scheduling deliberately. Reserve capacity for critical health, control, and recovery operations so overload cannot prevent the system from recovering. Do not let client-supplied priority become an unverified bypass. Propagate a server-authorized class across downstream calls so low-priority work cannot become high priority by fan-out.

A noisy-neighbor dashboard needs offered, admitted, completed, rejected, and consumed demand by tenant/class, plus queue age and objective attainment. Aggregate goodput can look healthy while one tenant receives nothing.

Shedding and degradation preserve different forms of value

Load shedding rejects or abandons work. Graceful degradation changes the work so it costs less while remaining useful under a declared contract. Examples include returning cached-but-acceptable data, reducing search breadth, omitting optional enrichment, lowering media quality, using a smaller model, sampling telemetry, or returning partial results with explicit completeness metadata.

Build a degradation ladder before the incident:

Use hysteresis: enter degradation at a higher pressure than the exit threshold. Require stability across enough windows to avoid flapping. Recovery is not normal operation in reverse. Cold caches, reconnect storms, retry backlog, and reconciliations add demand. Ramp admitted load and features while measuring protected goodput.

Partial results need correctness semantics. State which fields or partitions are absent, whether the result is cacheable, and whether a client may combine it with a retry. A cheaper model or approximate answer must still meet the quality floor used in the goodput denominator. Silent incompleteness converts availability into correctness failure.

Rejection semantics control the feedback loop

A rejection tells the client whether the request violated a client-scoped policy, encountered service-wide temporary unavailability, failed before execution, or may have produced an effect. Those distinctions govern retry safety.

For HTTP, RFC 6585 defines 429 Too Many Requests and allows a Retry-After field. RFC 9110 defines 503 Service Unavailable for temporary overload or maintenance and also permits Retry-After. Neither standard chooses tenant identity, rate algorithm, retry budget, or effect semantics for the application.

A useful rejection envelope includes:

• stable reason category: policy rate, concurrency, deadline, overload, or dependency;
• scope: tenant, operation, endpoint, region, or service;
• whether execution began and whether an effect is possible;
• retry eligibility and a bounded delay or not-before time when meaningful;
• idempotency or request identifier;
• degradation alternatives; and
• no sensitive capacity detail that aids abuse.

Do not emit the same retry delay to every client; synchronized retries recreate the spike. Clients need exponential backoff with jitter, deadline and attempt budgets, and a stop condition. Servers must include retries in offered-load and admission accounting. Chapter 25 develops the full retry, timeout, hedging, and idempotency contract; here, the admission rule is never to turn one rejected unit into unbounded future attempts.

Queueing is not a kinder rejection when the request cannot finish. A bounded short queue can absorb arrival variance below the deadline. Beyond that, reject explicitly so the client can choose another action. Report queue rejection separately from policy quota exhaustion.

Adaptive concurrency is a controller, not an oracle

A static concurrency limit is safe but can leave capacity unused as service time and resources change. Adaptive algorithms adjust a limit from latency or gradient signals. Their attraction is also their risk: they infer congestion from delayed, noisy outcomes and can interact with autoscaling, retries, placement, and downstream limits.

The Netflix open-source Gradient2 implementation is one concrete example. Its documented update compares a long-term RTT estimate with current RTT, clamps a gradient, allows a queue term, and smooths the new limit. The fixture reproduces one illustrative update:

• long-term RTT = 20 ms;
• current RTT = 50 ms;
• current limit = 100;
• gradient clamps to 0.5;
• unsmoothed target = (0.5 \times 100 + 10 = 60);
• smoothing factor 0.2 yields next limit (100 \times 0.8 + 60 \times 0.2 = 92).

This is not a recommended parameter set or validation of the algorithm. It demonstrates why the limit should move gradually rather than drop from 100 to 60 on one sample. RTT can rise because of network or downstream delay unrelated to local concurrency; an average can hide expensive classes; rejected samples can bias the observed population.

Every adaptive gate needs:

• minimum and maximum limits;
• cold-start and no-sample behavior;
• sampling window and censoring rules;
• increase/decrease rates and smoothing;
• per-class or cost normalization when demands differ;
• interaction with queue bounds and deadlines;
• manual/static fallback and rapid disable path; and
• step, spike, slowdown, loss, and recovery tests.

Measure the controller output beside actual concurrency, queue age, service time, goodput, rejections, and capacity changes. A limit that oscillates may look responsive while wasting capacity and causing client retry waves.

Fail open and fail closed are incomplete phrases

When limiter state is unavailable, a design can admit, reject, spend a cached lease, fall back to a conservative local budget, or allow only selected classes. The right choice depends on what the limiter protects.

Fail closed is appropriate when admission enforces a safety, authorization, legal, or irreversible-cost boundary whose violation is worse than unavailability. It can turn a limiter outage into a service outage and block recovery operations.

Fail open preserves availability when the limiter is only a soft policy signal and downstream layers have their own protection. It can overload the protected resource exactly when shared state is unavailable.

Prefer an explicit failure matrix:

Distributed limiter consistency follows the same reasoning. A small bounded overshoot may be acceptable for traffic shaping; it may be unacceptable for paid quota or a scarce nonrenewable resource. State the maximum overshoot under replication delay, partition, restart, and clock error. Use leases or escrow-like allocations when local availability and a global cap must coexist, accepting temporarily unused allocation as the cost of safety.

Measure protected goodput and the cost of saying no

An overload dashboard should distinguish:

offered → parsed → policy-eligible → admitted → started → completed
                                              ↘ late / cancelled / invalid
rejected by reason → client retry attempts → eventual useful completion

Track rates and resource demand, not counts alone:

• offered attempts and unique logical operations;
• admitted demand units and accepted requests;
• useful completions before deadline by class/tenant;
• partial/degraded completions by quality tier;
• queue age and service-time distributions;
• rejection rate, reason, scope, and processing cost;
• retry amplification and retry success;
• CPU, memory, I/O, lock, connection, and downstream demand;
• limiter decision latency, availability, state age, and overshoot;
• time in each degradation state; and
• recovery time and backlog drain.

Test rejection capacity itself. If a rejected request performs authentication, schema fetch, distributed quota update, high-cardinality logging, or a large serialized error, the cheap path may become the new bottleneck. Bound log volume and cardinality while preserving sampled evidence.

Goodput can remain flat while fairness or quality fails. Break it down by class and objective. Also measure admitted work that completes after the caller has abandoned it; this “orphan work” is often the gap between busy resources and useful outcomes.

Admission policy specification

Use this compact artifact in a design or operational review:

ADMISSION POLICY
Protected operation and useful-completion definition:
Scarce resource(s), capacity evidence, and safe operating region:
Decision point and rejection cost budget:

WORK MODEL
Classes / tenants / authorized priority:
Demand unit and pre-admission estimator:
Arrival, burst, skew, retry, and failure assumptions:
Deadline and correctness invariants:

HIERARCHY
Global / region / worker budgets:
Reservations, weights, borrow rules, and revocation time:
Rate, burst, concurrency, queue, and cost bounds:
Maximum distributed overshoot or unused lease:

STATE MACHINE
NORMAL entry/exit and full behavior:
DEGRADE entry/exit and quality contract:
SHED entry/exit and surviving classes:
RECOVER ramp, warm-up, and backlog gates:

FAILURE AND CLIENT CONTRACT
Limiter unavailable/stale/partitioned behavior:
429/503 or protocol-specific rejection semantics:
Effect possibility, retry eligibility, delay, jitter, and attempt budget:
Static fallback / disable / operator override:

EVIDENCE
Offered, admitted, completed, late, rejected, and retried demand:
Goodput and fairness by class/tenant:
Overload, dependency slowdown, limiter loss, and recovery tests:
Abort, rollback, and policy review owner:

Field checklist

• Is this gate enforcing entitlement, protecting capacity, or both?
• What is the useful-completion denominator, including deadline and correctness?
• Does the token/request price follow actual scarce-resource demand?
• How much burst can each layer admit simultaneously?
• Are concurrency permits held for the entire resource-consuming lifetime?
• When is queueing still likely to finish before the deadline?
• Which classes have reservations, may borrow, and can be starved for how long?
• What lower-cost result remains explicitly useful in each degradation state?
• Are retries counted as offered load and bounded by jittered budgets?
• What happens during stale state, partition, restart, and clock error?
• How much work does rejection itself consume?
• Does recovery ramp slowly enough for cold state and retry backlog?

Decision drill: reject now or queue briefly?

A premium read has 80 ms left on its propagated deadline. The local queue’s comparable-class p95 age is 45 ms; predicted service is 35 ms; the response path needs 10 ms; model uncertainty is 15 ms. The predicted 105 ms does not fit. Reject before acquiring the downstream permit and return a scoped retry response only if the logical operation’s deadline and attempt budget allow it.

Now assume queue age is 10 ms and all other terms remain. The modeled total is 70 ms, leaving 10 ms margin. Admission is defensible if the queue is bounded, the service estimate includes cold and failure states, and admission does not violate a class reservation. A strong answer does not say “premium always queues” or “fail fast always.” It shows the inequality, the evidence type, and what observation would change the decision.

Durable decision rules

1. Reject before scarce downstream work when the probability of useful completion falls below the explicit admission threshold.
2. Separate policy entitlement from local capacity protection; coordinate only as strongly as the invariant requires.
3. Price admission in the resource unit that constrains useful work, not request count by habit.
4. Reserve capacity for promised classes and recovery, define borrowing and starvation bounds, and measure fairness on useful service.
5. Treat degradation as a quality contract and recovery as a staged load state, not an incident-only flag.
6. Make rejection semantics dampen retries and reveal effect possibility without making the reject path expensive.
7. Bound every adaptive controller and give it a static fallback, hysteresis, observability, and overload/recovery test.

Evidence and transfer limits

• RFC 2697 specifies one byte-oriented token-bucket marker. It illustrates explicit rate and burst semantics but does not define an application admission policy.
• RFC 6585 defines HTTP 429 Too Many Requests and its optional Retry-After use. It does not choose identity, counting, fairness, or retry safety.
• RFC 9110 defines HTTP 503 Service Unavailable and Retry-After semantics. Application effect and idempotency rules remain separate.
• Google SRE: Addressing Cascading Failures presents production guidance on early rejection, load shedding, deadlines, and testing overload. Its thresholds and system examples are environment-specific.
• Google SRE: Handling Overload discusses quotas, client-side throttling, and request criticality in Google’s RPC environment. The number and meaning of classes are not universal.
• Netflix concurrency-limits Gradient2 source documents one open-source adaptive-concurrency update. The arithmetic in this chapter is illustrative, not a production recommendation.
• All load-sweep, class-reservation, token-bucket, deadline, and adaptive-limit numbers are deterministic modeled evidence in examples/performance-engineering-system-design-handbook/part-03/admission-overload/. The fixture has no queue simulator, retries, failures, multi-resource coupling, or production measurements.

Admission answers whether work should enter the scarce path. The next mechanism must ensure that admitted work carries a deadline, times out at the right boundary, does not multiply through retries or hedges, and can safely distinguish an uncertain response from an uncertain effect.