Case Study: The Hot-Key Cache Collapse

At 09:58, the catalog dashboard says the cache is healthy: 98.7% of 120,000 reads/s are hits. At 10:00, a launch directs 28% of all reads to one product key. At 10:03, that key expires simultaneously across application processes. At 10:07, a routine deployment removes a cache cell whose traffic share is 30%. By 10:08, the origin receives more work and completes less useful work. Operators add cache clients, which produce more concurrent misses.

The incident is not explained by a bad hit ratio. The ratio is excellent. It is explained by a state transition that the aggregate erased: one key carries 33,600 reads/s; a hit is cheap; a miss is an expensive origin query; missers are correlated; and the origin loses goodput beyond its 4,200/s knee.

The case’s design lesson is: cache architecture is governed by miss cost, key skew, and cold behavior—not average hit ratio. Every number here is deterministic modeled or simulated teaching evidence for a fictional catalog service. It is not a benchmark of Redis, a database, or a cloud cache.

Reconstruct the eight-minute collapse

The first useful artifact is a timeline, because the order of events distinguishes ordinary capacity exhaustion from correlated miss amplification.

The misleading metric was not false. It answered “what fraction of cache lookups hit?” It did not answer “which keys missed?”, “how expensive is each miss?”, “how many refresh leaders exist?”, “can the origin sustain the cold mode?”, or “is the system draining backlog?”

The primary unit is one logical product-detail read. Cache and origin attempts are capacity units. A response is correct when it contains a value inside its freshness contract or is explicitly marked as declared stale. A timeout is not goodput. A fast stale response is not correct if price or safety policy forbids that age.

Key popularity turns a small miss fraction into a large burst

At steady state:

120,000 reads/s × (1 − 0.987) = 1,560 misses/s

That average is below the origin’s safe 3,200/s admission bound and its observed 4,200/s goodput knee. The launch key alone receives:

120,000 reads/s × 0.28 = 33,600 hot-key reads/s

If synchronized expiry turns those reads into misses, offered work is eight times the 4,200/s knee. Even a short burst can fill connection queues, retain memory, trigger client timeouts, and launch retries. Once the origin is overloaded, recovery requires offered work below completion capacity for long enough to drain the backlog. Restoring a 98% cache hit ratio does not guarantee that condition.

The popularity curve matters because cache risk is concentrated. The top key carries 28%; the next keys follow a long tail. An aggregate cache can have plenty of memory and a high hit ratio while one shard, network link, serializer path, or origin row becomes a hot boundary. Redis documentation, for example, exposes keyspace hits and misses as useful aggregate statistics and separately documents hot-key observation. Those implementation facilities do not replace application semantics, but they illustrate why aggregate hit accounting and key concentration are distinct signals. See Redis key eviction and hit/miss statistics and Redis hot-key observability guidance.

The origin curve makes overload visible

The fixed teaching observations show why raw throughput and goodput diverge.

The curve is an observed fixture, not a universal queueing formula. Its shape can arise from connection churn, lock contention, buffer growth, garbage collection, timeout work, query-plan changes, or downstream quotas. The discriminating evidence includes origin service demand, active and pending connections, queue age, completed useful queries, cancellation delay, retry attempts, and dependency saturation.

Cell loss creates a second calculation. Losing a cell responsible for 30% of traffic exposes 36,000 reads/s. The remaining 70% retains its steady miss fraction:

36,000 exposed reads/s + (1,560 steady misses/s × 0.70)
  = 37,092 origin attempts/s

That value is not “a 30% cache outage.” It is nearly twelve times the selected safe origin admission boundary. If cache bypass is the automatic response to every cache error, the application converts cache unavailability into origin unavailability.

Amazon’s Builders’ Library describes this modal risk directly: a service can become dependent on its cache, and cold or unavailable cache state can surge traffic into a downstream service. It recommends testing with caching disabled and bounding fallback through mechanisms such as stale service or load shedding. See Caching challenges and strategies.

Six controls solve different parts of the problem

The review compares each technique in warm, expiry, cell-loss, and cold-region states.

Replicate the hot key

Replicating a read-only hot value across cache shards reduces one shard’s CPU and network concentration. It does not prevent all replicas expiring together, and invalidation or update fan-out becomes more expensive. Replication is useful when cache-node capacity is the constraint; it is incomplete when origin refresh is the constraint.

Add a process-local cache

A one-second local entry absorbs repeated hot-key reads before the distributed cache. Across 240 processes, the worst simple refresh population becomes about 240/s rather than 33,600/s, assuming each process refreshes once and invalidation/freshness semantics allow the local TTL. Fleet scaling now increases refreshers, and processes may show different versions. Local caching is therefore scoped to content whose one-second age is acceptable.

Redis’s client-side caching reference notes that clients need invalidation and disconnection rules; losing an invalidation channel can require flushing local state. Ledgerline’s cache is fictional, but the mechanism warning transfers: local cache safety includes coherence and reconnect behavior, not just hit speed. See Redis client-side caching reference.

Coalesce identical refreshes

Within each of 24 coalescing domains, the first miss becomes a leader and followers wait for that result. In the simple one-second model, only 24 hot-key refreshes/s reach the next layer, avoiding 33,576 origin attempts/s relative to direct hot-key fallthrough.

Coalescing needs identity, bounds, and failure semantics. The key includes content version, locale, authorization class, and freshness mode where those dimensions change the value. Followers inherit a bounded deadline. If the leader stalls, the system does not release all followers as new leaders. A failed result may be negatively cached briefly or served stale according to policy; otherwise the origin is hammered repeatedly.

Refresh early and randomize timing

Eligible readers begin background refresh in the final 20% of the soft TTL; a ±15% jitter prevents synchronized deadlines. Only one leader per coalescing domain refreshes a version. Early refresh moves work out of the request critical path and spreads it over time. Jitter alone is not sufficient: cell loss creates cold state regardless of TTL distribution, and a globally hot key can still have too many independent refreshers.

Serve stale within an explicit contract

The service retains the last known good product representation for up to 120 seconds during refresh error. Responses carry age and stale reason. A safety recall, legal restriction, inventory authority, or price policy may set a shorter window or prohibit stale service entirely. RFC 5861 defines HTTP stale-while-revalidate as permission to serve stale while validation proceeds and separately describes stale-on-error behavior; application-level caches must still define correctness for their data. See RFC 5861.

Stale service protects read availability and origin capacity. It does not make stale data current. The incident dashboard reports fresh, declared-stale, rejected, and erroneous outcomes separately so a low-latency stale response cannot hide product harm.

Admit origin work and prewarm in stages

The origin boundary admits at most 3,200 catalog refreshes/s for this workload class, below the observed knee. Excess refresh demand serves eligible stale data or fails fast with a bounded retry hint; it does not join an unbounded queue. Other origin workloads retain their own reserved partitions.

Prewarming restores hot keys and working-set slices in four stages: 10%, 25%, 50%, then 100%. Each stage holds until origin admission, queue age, goodput, error, cache memory, and freshness lag remain within limits. Prewarming is charged against the same origin budget as live misses. A batch job that ignores live demand can become another stampede.

Redis’s prefetch-cache documentation emphasizes that preloading moves work ahead of the first request but introduces fit, freshness, and synchronization obligations. The useful transfer is conditional: prewarming helps when the selected working set is known and the source can sustain the controlled load. See Redis prefetch-cache guidance.

Key-aware telemetry without cardinality collapse

The steady metrics path cannot label every request by raw product key. That would create unbounded cardinality, expose sensitive identifiers, and make the telemetry system a new bottleneck. Instead it reports:

• bounded popularity buckets such as top-1, top-10, top-100, and tail;
• sampled heavy-hitter sketches or a separately access-controlled top-key report;
• per-cell hit, miss, eviction, connection, CPU, memory, and queue signals;
• refresh leaders, coalesced followers, leader duration, and follower timeout;
• origin admitted, rejected, completed, cancelled, and late attempts;
• fresh, declared-stale, negative-cache, and error outcomes;
• cache age, soft-TTL distance, hard-TTL distance, and refresh lag;
• cold-key and warm-key load-test classes.

The key-risk indicator is not one score. Operators need both concentration and consequence: top-key fraction, miss cost, independent refresher count, and origin headroom. A key carrying 28% of traffic but served from immutable process memory may be safe; a key carrying 2% may be dangerous if each miss launches a multi-second analytical query.

Admission operates on refresh identity and workload class. User reads do not each acquire an origin slot. They join a leader, consume a valid local/distributed value, receive declared stale data, or fail at a bounded edge. This prevents a popular user-facing key from monopolizing the entire origin budget.

Deployment becomes a cache-state migration

The failed deployment treated application and cache cells as interchangeable stateless capacity. The redesigned rollout treats warm data, client routing, serialization versions, and origin reserve as migration state.

1. New code reads both current and previous cache formats and writes the new version.
2. A dark cell receives shadow lookups and prewarms the top popularity buckets within origin admission.
3. One percent of live reads enters the cell; fresh/stale correctness and key distribution are compared.
4. Exposure advances through 10%, 25%, 50%, and 100% only after hold periods.
5. Old cells drain after replacement hit rate and origin load remain safe; they are not emptied simultaneously.
6. Rollback preserves readable cache formats and the last-good stale snapshot.

A serializer mismatch must not respond by discarding the entire cache namespace at once. Unknown entries are isolated by version and refreshed within budget. Deployment automation has an abort condition on predicted origin demand, not only cache-node health.

The cold-region startup test is mandatory because a region can be compute-healthy and cache-empty. Routing expands only as its warm coverage and origin contribution satisfy the same stage gates. Capacity planners reserve origin work for one declared cell loss plus live steady misses; if that cannot be afforded, the availability design depends on stale service or a replicated warm snapshot and must say so.

The recovery runbook controls demand before warmth

The cache recovery runbook is executable in this order:

1. Declare the mode. Identify synchronized expiry, cell loss, invalidation storm, serializer rejection, or origin fault. Freeze nonessential deployments and bulk invalidations.
2. Protect the origin. Enforce the 3,200/s catalog-refresh admission partition, disable broad retries and bypass, and preserve capacity for correctness-critical origin work.
3. Preserve usable state. Extend only the approved stale window for the affected content class; do not alter price, recall, authorization, or inventory policy silently.
4. Constrain leaders. Verify coalescing identity and leader deadlines. Cap the number of refresh leaders per cell and region.
5. Rank warmth. Load the top-1, top-10, top-100, then broader working-set buckets. Deduct prewarm attempts from the origin budget.
6. Observe goodput and backlog. Hold until origin queue age falls, correct completions exceed new offered work, retry population drains, and cancellation releases resources.
7. Expand one stage. Increase routing or prewarm coverage only while admission, freshness, error, cache memory, and origin reserve stay inside bounds.
8. Restore policy deliberately. Reduce emergency stale windows, re-enable background jobs, and remove temporary controls one at a time.
9. Reconcile and retain evidence. Record key distribution, cache generations, invalidations, origin attempts, response semantics, and recovery time.

“Hit ratio returned” is not an exit criterion. The fixture’s uncontrolled recovery takes 11.8 minutes; the controlled path takes 2.6 minutes, an improvement of 9.2 minutes. Recovery completes only when backlog, retries, origin service demand, and stale population are inside steady-state bounds for two observation intervals.

Adverse-state validation

The test generator follows a fixed arrival schedule independent of completions so origin slowdown does not make offered load disappear. It reconciles logical reads, cache attempts, refresh leaders, origin admission, completion, stale service, rejection, and cancellation.

Correct fresh or declared stale combines two separately reported populations for compactness in this table. The test still fails if stale age exceeds policy, prohibited content is stale, or a response lacks an explicit marker. The percentages are fixed simulated observations, not confidence intervals.

Run the packet:

cd examples/performance-engineering-system-design-handbook/part-08/hot-key-cache-collapse
node analyze.mjs
node verify.mjs

It reproduces steady misses, hot-key demand, the 8× knee amplification, cell-loss demand, local and coalesced refresh counts, avoided origin attempts, overload goodput loss, recovery movement, and three adverse-state admission/correctness checks.

Failure modes after the repair

The controls do not eliminate cache complexity. They change an unbounded many-to-one fallthrough into a measured state machine whose maximum origin demand and degraded semantics are explicit.

Diagnostic drill

Thirty minutes after the repair, operators observe:

aggregate hit ratio: 99.2%
top-1 key fraction: 4% (was 28%)
top-100 fraction: 62% (was 51%)
origin admitted: 3,180/s
origin goodput: 3,090/s
refresh leader p99: 940 ms (was 120 ms)
coalesced follower timeout: 7.2%
declared stale: 0.4%
origin query service p99: 190 ms (was 54 ms)

Decide whether to increase the origin admission limit, lengthen follower deadlines, increase stale duration, or hold the rollout. Name three discriminating checks.

Diagnostic answer guide

The hot key is no longer dominant, but the top-100 concentration rose and refresh leaders became intrinsically or dependently slower. Origin service p99 also rose, so this is not only follower waiting. Admission is already close to 3,200/s; increasing it without a new goodput curve risks crossing the knee. Longer follower deadlines retain more waiting work and do not improve the leader. A broader stale window may be valid only for content classes whose age contract permits it.

Hold or restrict the rollout. Compare query plans and service demand by bounded key-cost class, inspect origin queue and dependency time, verify whether a data or index change affected the top 100, and separate leader queue time from service time. A targeted precompute or replication change may be appropriate for an expensive class. Any admission increase requires a controlled adverse-state test showing higher goodput and recovery reserve.

Design exercise: stale is forbidden

Assume the cached value now includes a safety-recall flag that must be current before every response. Stale service is prohibited. Redesign the launch and cell-loss behavior without lowering correctness. Specify capacity, admission, response semantics, deployment, and recovery evidence.

Answer guide

The answer must remove stale service for the recall-bearing value, not relabel it fresh. Options include separating immutable presentation data from a small strongly validated recall record, pushing recall invalidations with an authoritative version check, provisioning origin or an independently replicated authority for the full cold load, or rejecting reads beyond the safe admission boundary. Local caches and coalescing remain useful if invalidation and version validation are correct. Early refresh and prewarming reduce cold exposure but cannot be the only safety argument.

The availability objective may need to change because correctness forbids the previous degraded mode. The design should quantify maximum admitted reads, rejection behavior, launch ramp, cell-loss reserve, and recovery time; test a lost invalidation channel and a cold region; and prove that cache identity cannot mix recall versions.

Field review card

When a cache protects an expensive origin:

• Plot key popularity and miss cost, not only aggregate hit ratio.
• Count logical reads, cache attempts, refresh leaders, and origin attempts separately.
• Load-test the origin to a goodput curve and select admission below its collapse knee.
• Model synchronized expiry, cell loss, and cold startup.
• Replicate hot keys when cache-node concentration is the constraint.
• Use local caching only with explicit freshness and invalidation semantics.
• Coalesce by the full value identity and bound leader and follower lifetimes.
• Combine early refresh with bounded jitter; do not treat jitter as a complete repair.
• Serve stale only where age and correctness policy allow it, with visible markers.
• Keep origin admission active during cache failure and deployment.
• Charge prewarming to the same origin budget as live misses.
• Route to a cold region in stages, holding on warmth, goodput, queue, and correctness.
• Keep raw keys off the main metric label path; use bounded concentration views.
• End recovery when backlog and stale population normalize, not when hit ratio first rises.
• Preserve a runbook that removes emergency controls one at a time.

The collapse was produced by correlation: concentrated popularity, common expiry, common cell loss, and a shared origin. The repair works by breaking or bounding those correlations. The next case, a burst-tolerant event pipeline, replaces an immediate miss herd with durable backlog; its central question is not whether work queues, but whether the system can preserve live progress and eventually drain what it accepts.