Multi-tenancy, Isolation, and Fairness

The shared service reports a global p99 of 196 ms. Its 250 ms objective is green. The fleet has spare memory, no node is above 74% CPU, and useful goodput is steady.

Cedar’s export operation reports a p99 of 1,840 ms.

The two statements describe the same five-minute window. Cedar contributes only 1% of requests in the scoped latency sample, and its slowest 101 operations contribute about 0.01%. In the aggregate histogram, 985,000 of one million operations complete by 180 ms, another 9,000 by 196 ms, and the final 6,000 by 1,840 ms. The global 99th-percentile rank falls in the 196 ms bucket. Cedar’s tenant-scoped histogram contains 5,000 operations by 41 ms, another 4,500 by 128 ms, 399 by 196 ms, and 101 by 1,840 ms. Its 9,900th operation therefore falls in the final bucket, so Cedar’s p99 is 1,840 ms.

Nothing is wrong with the arithmetic. The measurement boundary is wrong for the promise.

The service promises performance to tenants but measures only the fleet. It admits work by API key, schedules from a shared queue, partitions by object ID, accounts CPU by process, and attributes cost after aggregation. A burst from Alder fills the queue and evicts Cedar’s cache lines before any tenant-level limit sees the demand. The global dashboard proves that the majority is healthy; it cannot prove that Cedar receives its contractual service.

The governing rule is: measure, schedule, and limit work at the same tenancy boundary at which promises and costs are assigned.

Start with the promise boundary

A tenant is the entity to which isolation, service, security, or cost is assigned. It might be an external customer, an internal team, a project, a namespace, an account hierarchy, a model, or a workload class. “Tenant” is not necessarily synonymous with user, database, process, or deployment.

One request may carry several relevant identities:

organization = Cedar
workspace    = cedar-eu
operation    = export
priority     = contracted-interactive
cost center  = finance-42
data class   = restricted

Choose the boundary separately for each promise. Cedar may receive an organization-wide monthly spend limit, workspace-level data residency, operation-level latency objectives, and workload-class admission priority. Flattening those dimensions into one tenant ID either loses control or produces a cardinality explosion.

Write a tenancy contract before choosing mechanisms:

The time window matters. A tenant can obey a per-minute request quota and still issue a microburst that fills a 200 ms queue. A scheduler can be fair over one hour while violating every interactive deadline. State the interval over which a minimum, weight, or limit is meaningful and the maximum burst admitted inside it.

Isolation has several dimensions

Isolation is not one switch. A design can be strong in one dimension and weak in another:

• performance: one tenant cannot consume another’s bounded service, queue, cache, or latency budget;
• failure: faults, overload, corrupt data, and recovery work have a contained blast radius;
• security: authorization, confidentiality, integrity, and administrative boundaries prevent prohibited access or influence;
• data: placement, encryption, backup, retention, and deletion follow the tenant’s policy;
• operational: deployment, configuration, incident response, and maintenance can be scoped;
• economic: consumption and shared overhead can be attributed well enough to govern price and capacity; and
• change: one tenant can move, upgrade, or receive a special policy without destabilizing the pool.

Performance isolation is not security isolation. A dedicated queue can protect latency while both tenants still share credentials. Separate virtual machines can strengthen some runtime boundaries while both depend on the same identity database, key service, control plane, or operator role. Never sell “dedicated” as a security claim until threat model and enforcement boundary say what is dedicated.

Choose a tenancy arrangement by the constraint

Four arrangements recur, with many hybrids between them.

Shared everything

Tenants use the same request path, worker pools, caches, partitions, and backing services. Pooling is efficient when demand is diverse and mechanisms account for cost accurately. It also creates the widest interference graph. A single global queue, connection pool, cache, metadata table, compaction budget, or retry policy can defeat tenant-aware admission at the edge.

Use shared-everything when workloads are sufficiently homogeneous, consequences are bounded, and all material constrained paths can enforce tenant context. Reject it when one tenant can monopolize an unschedulable resource or when audit/security requirements demand a stronger boundary.

Pooled resources with logical isolation

Tenants share capacity but receive separate queues, quotas, namespaces, partitions, cache budgets, or connection pools. This often provides the best first isolation step because spare capacity can remain work-conserving: idle reservations are borrowed under policy rather than stranded.

Logical isolation is only as complete as its accounting. Per-tenant request queues do not control a shared database’s storage I/O. Namespace CPU quotas do not bound an untagged object-store backfill. Trace identity through every expensive asynchronous effect.

Cells

A cell is a bounded copy of a serving stack that hosts a subset of tenants. It contains ordinary overload and failure without dedicating a full stack per tenant. Cell placement should consider demand, state, dependencies, geography, and correlated tenants. Hashing tenants evenly by count can place the three largest tenants together.

Cells reduce blast radius only when their dependencies and controls are also bounded. A cell that shares one metadata leader, token verifier, deployment controller, or global cache can still fail globally. Keep a declared global dependency budget and test cell loss plus tenant evacuation.

Dedicated arrangements

A tenant receives a dedicated process pool, cell, cluster, account, database, or hardware boundary. Dedicated capacity can simplify strong minimums, custom versions, and investigations. It reduces statistical multiplexing and increases idle reserve, fleet count, rollout variance, repair work, and cost.

Dedication should follow an identified constraint: extreme service demand, incompatible security policy, concentrated state, unusual availability requirement, regulatory placement, or a contract worth the operating cost. “Large customer” is insufficient without the resource and failure evidence.

The table is directional, not a score. A badly operated dedicated fleet can be less reliable than a mature pool, and a cell with one global dependency can have a global blast radius.

Account for service demand, not request count

Fairness needs a unit. Equal request counts are unfair when operation cost differs. Equal CPU time is insufficient when memory, storage, network, accelerator time, or a serialized metadata path constrains the system.

For each tenant and operation, estimate a resource vector:

demand(operation) = {
  cpu_ms,
  resident_byte_seconds,
  cache_insert_bytes,
  storage_read_bytes,
  storage_write_bytes,
  storage_iops,
  network_bytes,
  connections_seconds,
  metadata_ops
}

Measure successful, rejected, retried, failed, and recovery work. Rejected work still consumes parsing, authentication, logging, and response bandwidth. Retries must be charged to a logical operation and also visible as physical resource demand. Background repair, index build, export, deletion, and audit work need tenant attribution or an explicit shared-overhead pool.

The executable model in examples/performance-engineering-system-design-handbook/part-04/tenant-fairness/ uses one CPU-denominated bottleneck to keep the arithmetic inspectable. Alder offers 5,400 work/s at 1.0 CPU-ms/work, Birch 2,700 at 1.5, and Cedar 4,500 at 0.6:

Alder: 5,400 × 1.0 = 5,400 CPU-ms/s
Birch: 2,700 × 1.5 = 4,050 CPU-ms/s
Cedar: 4,500 × 0.6 = 2,700 CPU-ms/s
total                 12,150 CPU-ms/s
safe capacity          12,000 CPU-ms/s
deficit                   150 CPU-ms/s

There are 12,600 offered operations/s, yet their CPU demand is 12,150 CPU-ms/s. Allocating one third of capacity by request count would grant each tenant 4,000 requests/s: Alder receives 4,000 CPU-ms/s, Birch could demand 6,000, and Cedar only 2,400. The scheduler would promise equal counts while oversubscribing the expensive class.

Normalize only where calibration is valid. CPU-ms from different machine shapes or software versions may not be equivalent. Storage IOPS conceal request size and device behavior. A “cost unit” is a model with uncertainty, not a physical constant. Preserve raw resource measures beside the normalized unit and recalibrate after mix, code, or platform changes.

For multiple constrained resources, inspect each tenant’s dominant share rather than collapsing everything prematurely. Dominant Resource Fairness formalizes one allocation approach for users with different multi-resource needs. It is useful evidence that fairness across CPU and memory is not request round robin; it is not a drop-in guarantee for stateful, locality-sensitive, deadline-bearing production workloads.

Reservation, limit, weight, quota, and credit do different work

These terms should not be interchangeable:

• a reservation protects a minimum when the tenant has eligible demand;
• a limit caps consumption even when capacity is idle;
• a weight divides contested borrowable capacity among active tenants;
• a quota bounds consumption or object count over a declared scope and interval;
• a burst credit permits temporary use beyond a baseline and records how much permission remains; and
• a priority orders consequences when not all eligible work can run.

Ledgerline gives Alder, Birch, and Cedar CPU reservations of 3,600, 3,000, and 1,800 CPU-ms/s. The reservations total 8,400, leaving a 3,600 CPU-ms/s borrow pool. Active borrow weights are 3:2:1.

First-pass allocation is reservation plus weighted borrow, capped by offered demand:

Alder: 3,600 + 1,800 = 5,400 CPU-ms/s
Birch: 3,000 + 1,200 = 4,200; demand caps allocation at 4,050
Cedar: 1,800 +   600 = 2,400 CPU-ms/s

Birch leaves 150 CPU-ms/s unused. A work-conserving second pass lends it to Cedar, giving final allocations of 5,400, 4,050, and 2,550 CPU-ms/s. Cedar receives 4,250 work/s and carries the 150 CPU-ms/s shortage. All reservations survive; unused capacity is not stranded.

Borrowing needs a revocation contract. Capacity borrowed by a batch job may be tied up in a long transaction, GPU kernel, storage request, or non-preemptible compaction when the owner returns. Bound work units, use deadlines/checkpoints where safe, reserve preemption capacity, and measure reclaim time.

A burst-credit bucket can make borrowing time explicit. Cedar’s bucket contains 180,000 CPU-ms and permits 600 extra CPU-ms/s. A continuously full-rate burst lasts:

180,000 CPU-ms / 600 CPU-ms/s = 300 s

Specify refill rate, maximum bucket, eligible classes, price, expiry, debt behavior, and whether credits survive a failure or migration. Credits must not spend another tenant’s reservation or the fleet’s failure reserve. A credit is admission permission, not proof that every downstream resource has capacity.

Fair scheduling is a hierarchy

One flat tenant queue cannot express organization, workspace, operation, and priority promises. Use hierarchical scheduling:

fleet capacity
  -> failure reserve and system work
  -> cell
  -> tenant reservation and borrow
  -> interactive / batch / recovery class
  -> operation deadline and normalized cost

At each level define eligibility, unit, weight, maximum burst, unused-capacity rule, and starvation bound. Reserve system capacity for health, fencing, deletion, and recovery; do not let ordinary tenant traffic exhaust the mechanism needed to restore fairness.

Fair queueing approximates a fluid ideal with discrete jobs. The job size matters. A 2 GiB transfer at the head of one FIFO queue delays every small request behind it. Deficit-based schedulers can charge estimated cost and carry a balance; deadline-aware policies can reject work that can no longer meet its objective. Estimation error should be corrected from observed service demand without allowing a tenant to gain permanently by underdeclaring cost.

Priorities must have an aging or minimum-service rule. Strict priority can starve batch exports forever during sustained interactive load. Conversely, an unbounded batch reservation can consume recovery headroom during an incident. State the protected minimum for each class and the mode in which it can be reduced.

Scheduling happens at every queue: edge, application, connection pool, storage, network, broker, accelerator, and control plane. Tenant-aware ordering at the edge does not help if all admitted work enters one downstream FIFO. Carry tenant and class identity through RPC, messaging, logs, jobs, and derived work, with cardinality controls and privacy review.

Follow the noisy neighbor through every resource

CPU interference is only the visible case.

Memory and cache

A tenant can expand heap, page cache, metadata, socket buffers, or cache footprint. Memory is stateful: reclaim and eviction impose CPU and I/O after the allocation. Track resident bytes, allocation rate, reclaim stalls, fault rate, eviction victims, and cache hit rate by tenant and class. Separate hard safety limits from best-effort protection. A hard limit can turn interference into tenant-local OOM or retry storms; that is containment only if the failure semantics are safe.

Partition shared caches by admission, size, and eviction policy where reuse differs. A per-tenant maximum without a minimum still lets a burst evict everyone before the cap is observed. Admission filters, protected segments, object-size limits, and bounded fill rate can matter more than byte capacity.

Storage

Bytes are not IOPS, and IOPS are not service time. Small random reads, large sequential scans, sync writes, compaction, snapshots, and deletion compete differently. Attribute queueing and device time, limit background amplification, and schedule rebuild/repair separately from foreground work. Per-tenant logical write quotas must include replication, indexes, logs, and compaction amplification where those costs are causal.

Network and connections

One tenant can consume bandwidth, packets/s, sockets, connection-pool slots, TLS work, or downstream concurrency. Apply limits at ingress and egress. Long-lived idle connections may be cheap in bandwidth and expensive in memory or file descriptors. A byte-weighted scheduler can still be unfair to latency-sensitive small messages if large transfers are non-preemptible.

Metadata and control paths

Tenant creation, schema changes, list operations, policy evaluation, quota counters, placement directories, certificate issuance, and metrics labels often share a small serialized service. An ordinary data-plane quota may not touch them. Bound object counts and mutation rates, paginate/list safely, isolate operator traffic, and keep emergency control available during tenant overload.

Hot tenants and partitions

A tenant-aware partition key contains cross-tenant damage but does not split one hot tenant. Add a second dimension—workspace, bucket, entity range, or virtual shard—while retaining tenant identity for accounting. Large tenants may span partitions or cells; their global reservation then needs hierarchical leases that tolerate delayed counters and bound overshoot.

Detect hot tenant, hot partition, and hot node separately. Moving a tenant does not help if one entity key remains serialized. Splitting a key does not help if the tenant saturates a shared authority. Use service demand and queue age at each boundary to choose the repair.

Tenant SLOs require tenant evidence

Global objectives answer whether the service population is healthy. They do not imply fairness. Report at least:

• latency, goodput, rejection, and correctness by tenant tier and operation;
• worst and high-percentile tenant outcomes, not only request-weighted aggregate;
• tenant share of each constrained resource and queue;
• reservation delivered, borrowed capacity, reclaim delay, and limit throttling;
• hot-tenant and hot-partition concentration;
• cache and dependency behavior by tenant class;
• objective compliance by normal, overload, failure, recovery, and migration mode; and
• cost per objective-compliant operation with shared overhead shown separately.

Avoid one time series per tenant on the primary dashboard when cardinality is large. Use distributions across tenants, top-K by harm and consumption, tier cohorts, and drill-down indexes. Keep exact tenant IDs in controlled diagnostic paths rather than unbounded metric labels.

Two fairness views are both necessary:

1. work-weighted: what fraction of operations or business work met the objective?
2. tenant-weighted: what fraction of tenants met their assigned objective?

A request-heavy tenant should influence capacity planning. It should not make one hundred harmed small tenants disappear statistically.

Cost attribution is a governed estimate

Unit economics connect tenancy policy to architecture. Attribute direct demand—CPU, memory time, storage, transfer, external calls—and shared demand—reserve, control plane, replicas, observability, support, and idle fragmentation.

Do not pretend allocation rules reveal metaphysical truth. Equal split, usage-proportional, reservation-proportional, causal activity, and marginal-cost attribution answer different questions. Record the rule and uncertainty. A tenant that triggers a dedicated cell creates fragmentation even at low utilization; a tenant whose workload improves batching may reduce marginal cost.

Use cost as a design signal inside service constraints. A cheaper scheduler that violates Cedar’s minimum is not efficient. Track:

tenant cost per objective-compliant logical operation
= direct serving + amplification + reserved share + attributed shared overhead

Keep chargeback separate from admission when estimates are delayed or disputed. Never let a billing pipeline failure silently disable a safety limit; never let an approximate cost counter become the authority for a correctness-sensitive rejection.

Moving between tiers is a migration

Promoting Cedar from a pool to a dedicated cell changes routes, state, caches, quotas, keys, telemetry, failure ownership, and cost. Treat it as an online migration:

1. declare tenant scope, operations, authority, and target objectives;
2. prove target capacity for foreground, recovery, and uncertainty;
3. create target policy, identity, encryption, schema, and observability;
4. seed derived state under a movement budget;
5. catch up authoritative changes with stable identities;
6. shadow reads and compare correctness, freshness, latency, and cost;
7. stop or fence stale writers before authority-sensitive cutover;
8. canary one operation class and ramp within abort thresholds;
9. retain rollback while the old path is coherent;
10. drain queues, sessions, retries, and derived work; and
11. release old reservation only after recovery and deletion obligations pass.

The tenant ID must remain stable across tiers. Clients should not learn physical placement. Deduplication keys, versions, and audit identity must survive route changes. If migration changes semantics, version the contract rather than hiding the difference behind routing.

Demotion needs equal care. A dedicated tenant returning to a pool may have a larger cache, looser burst behavior, custom schema, or different failure assumptions. Prove that it fits pooled limits without harming existing tenants.

Isolation review record

Tenant model:
  identity hierarchy, lifecycle, cardinality, geography, data/security class

Promises:
  boundary, operation/class, unit, window, minimum, limit, burst, objective

Resource vectors:
  CPU, memory/reclaim, cache, storage/IOPS, network, connections,
  metadata/control, amplification, uncertainty, calibration version

Arrangement:
  shared / logical pool / cell / dedicated; included and shared dependencies

Scheduling:
  hierarchy, eligibility, cost estimate, reservations, weights, priorities,
  borrow/reclaim, starvation bound, non-preemptible work

Enforcement:
  admission and downstream gates; fail-open/closed behavior; counter delay

Evidence:
  per-tenant and aggregate outcomes, worst-tenant view, resource heat map,
  queue/service demand, cache/reclaim, failure and recovery mode

Economics:
  direct cost, shared rule, reserve, fragmentation, objective-compliant unit

Tier migration:
  state/authority, target capacity, shadow/canary, fence, drain, rollback

Security boundary:
  threat model and enforcement; claims not inferred from performance controls

Reject the design if a contractual minimum has no scheduler, a scheduler has no cost unit, a limit is enforced after the constrained resource, or tenant identity disappears before expensive work completes.

Fairness drills

Find the hidden tenant. Reproduce the coarse histogram’s global 196 ms p99 and Cedar 1,840 ms p99. Change Cedar’s sample share to 1.2%. Determine how the global result changes and explain why no aggregate threshold can replace a tenant objective.

Repair count fairness. Reproduce 12,150 CPU-ms/s offered demand against 12,000 capacity. Compare equal request allocation with CPU-demand allocation. Add storage bytes and memory-time vectors; identify each tenant’s dominant resource and the limits of one scalar unit.

Write the policy. Use reservations 3,600/3,000/1,800 CPU-ms/s, borrow weights 3:2:1, and Cedar’s 180,000 CPU-ms bucket. Specify refill, reclaim time, priority, missing-counter behavior, and the overload response when all three tenants exceed limits.

Trace six channels. For one export burst, follow CPU, heap/reclaim, cache eviction, storage read/compaction, network/connection, and metadata work. Mark the first boundary that loses tenant identity and repair it.

Choose a tier. Compare a tenant with unpredictable CPU demand, one with a unique encryption boundary, one with a hot entity key, and one with a custom release cadence. Choose shared, pooled, cell, or dedicated placement and state the decisive constraint.

Migrate Cedar. Move Cedar from a pooled cell to a dedicated cell while an export and deletion are active. Define state copy, authority, capacity, quota-credit treatment, shadow comparison, fencing, abort, drain, and rollback.

Separate security from performance. Red-team a dedicated worker pool that shares identity, keys, backups, operators, and a metadata database. State which performance claims survive and which security claims have no evidence.

Protect the control plane. A tenant creates one million objects while its data-plane rate stays below quota. Bound create/list/watch work, metric cardinality, audit output, operator access, and cleanup without blocking emergency isolation.

Durable rules for multi-tenancy

1. Define tenants by the boundary that receives the promise, not by the easiest available label.
2. Carry tenant and workload-class identity through every expensive synchronous and asynchronous effect.
3. Measure service demand and constrained resources; equal request counts are rarely equal work.
4. Keep reservations, limits, weights, quotas, credits, and priorities semantically distinct.
5. Make idle capacity work-conserving only through a bounded borrow and reclaim policy.
6. Schedule hierarchically across fleet, cell, tenant, class, and operation where promises exist.
7. Protect memory, cache, storage, network, connection, metadata, and recovery paths—not CPU alone.
8. Report request-weighted and tenant-weighted outcomes; aggregate p99 cannot prove tenant health.
9. Treat cells and dedicated tiers as scoped boundaries whose shared dependencies remain explicit.
10. Attribute cost with a declared rule and uncertainty; optimize only objective-compliant work.
11. Treat tier changes as stateful migrations with capacity, fencing, drain, and rollback.
12. Prove security isolation separately from performance and failure isolation.

Fair scheduling answers who may consume shared resources. It does not answer which copy is allowed to define truth. Once a tenant’s orders, cached reads, indexes, fraud features, exports, and deletion records cross cells and tiers, the next architectural obligation is state lineage: authority, freshness, rebuild, retention, and reconciliation for every copy.

Evidence and transfer limits

• The primary Dominant Resource Fairness paper develops a multi-resource allocation policy around dominant shares. Its model informs CPU/memory fairness; it does not include Ledgerline’s deadlines, locality, state transfer, pricing, or failure semantics.
• Parekh and Gallager’s primary Generalized Processor Sharing paper analyzes service guarantees for fluid sharing and a packetized approximation. Application jobs are less divisible, cost estimates are uncertain, and downstream queues can defeat an upstream scheduler.
• Current Linux kernel cgroup v2 documentation describes hierarchical controllers and distinct weight, limit, and protection models for resources such as CPU, memory, and I/O. It is an implementation mechanism, not a complete tenant contract or security boundary.
• Current Kubernetes resource-management documentation distinguishes scheduling requests from enforced limits and documents resource-specific behavior. Versioned platform semantics must be verified in the deployed release; namespace/container controls do not automatically cover application queues, caches, storage service demand, or external dependencies.
• The deterministic fixture in examples/performance-engineering-system-design-handbook/part-04/tenant-fairness/ reproduces the CPU-demand allocation, five-minute credit duration, and coarse p99 masking example. It deliberately omits multi-resource contention, variable job size, scheduler granularity, arrivals, retries, failures, and locality; use workload traces, shadow accounting, overload tests, and scoped production evidence before setting policy.