Object, File, Media, and CDN Systems

Vaultspan must accept a 12 GiB video from a client that may lose connectivity, make the completed object readable only after integrity and policy checks succeed, and serve a newly popular 8 MiB rendition through 180 edge locations. At the same time, one storage failure has placed 96 TiB of coded shards in a degraded state. The design question is not simply where to put the bytes. It is which work may share capacity and which work must retain an independent reserve.

Keep four ledgers from the first design review:

1. namespace and metadata work: creates, lookups, listings, version checks, authorization, and lifecycle transitions;
2. foreground content work: uploaded, downloaded, ranged, and transformed bytes;
3. cache-miss work: origin requests and origin bytes during cold starts, purges, key changes, and regional failover; and
4. durability work: audit reads, reconstruction reads, repair writes, rebalancing, and deletion proof.

One device count or aggregate bandwidth number cannot represent all four. A system can have abundant storage capacity but an exhausted metadata partition; a quiet origin can fail under simultaneous edge misses; a healthy foreground p99 can conceal a repair queue that lengthens the exposure to another failure.

The boundary begins when an authenticated namespace operation or content transfer is accepted. It ends only when the declared result is authoritative: a committed object version, a range response for a named representation, a durable deletion state, or a completed repair with evidence. Client codecs, media editing, and vendor selection are outside the boundary. Transcoding matters only where it changes object identity, cache keys, transfer shape, or capacity.

The namespace names; the content plane moves bytes

An object system usually presents a logical name such as (account, bucket, key, version). The metadata behind that name can include content length, checksum, media type, creation time, retention policy, encryption-key reference, access policy, placement class, manifest, and deletion state. The content plane stores and transfers chunks or coded fragments. The namespace is not the content, and a directory-like listing is not necessarily the authority for a direct read.

A useful request decomposition is:

metadata path
  authenticate -> authorize -> resolve namespace/version
               -> locate manifest/placement -> return read or upload capability

content path
  client/edge -> transfer endpoint -> chunk or coded-fragment owners
              -> verify bytes -> stream response or stage upload

The planes may scale independently, but their contract must meet at immutable identity. A content address, version ID, or generation prevents a metadata retry from silently pointing at different bytes. Signed transfer capabilities should bind operation, object/version, byte range or part, expiry, and tenant. Otherwise a fast content plane can bypass the correctness and isolation decisions made in metadata.

Small and large objects stress opposite ends of the design. Ten million 2 KiB objects contain only about 19.1 GiB of logical payload, yet may require ten million namespace records, allocation decisions, checksums, authorization checks, requests, and garbage-collection entries. Packing can reduce metadata and storage overhead, but a pack index becomes another derived access path and deletion may leave holes until compaction. One 100 GiB object needs few metadata operations but long-lived flows, resumability, range service, and failure-aware bandwidth scheduling.

Measure both distributions. Report requests per second by operation and object-size bucket; bytes per second by direction and cache outcome; active transfers and their ages; metadata amplification; chunk count; and request, storage, and egress cost per useful byte. “Average object size” mixes workloads whose bottlenecks differ.

A multipart upload is a commit protocol

Splitting the 12 GiB object into 64 MiB parts yields 192 parts. Parts may be transferred independently and retried without restarting successful work. In the teaching model, a monolithic transfer that fails after 93% has already sent 11,427.84 MiB. Retrying one failed 64 MiB part instead saves 11,363.84 MiB. That is modeled client-to-service traffic, not a product limit or measured saving.

Resumability requires durable session state:

initiate(account, key, expected policy)
  -> upload_id, generation, permitted part shape, expiry

upload_part(upload_id, part_number, bytes, checksum)
  -> durable part receipt {part_number, length, checksum, storage identity}

complete(upload_id, ordered receipts, object checksum/size)
  -> validate all parts and policy
  -> atomically publish immutable object version and manifest
  -> return committed version identity

An acknowledged part must survive the failure boundary promised by the API. The client persists the upload ID and acknowledged receipt set, lists authoritative parts after reconnect, and uploads only missing or mismatched parts. A client-side progress bar is not evidence. Duplicate part uploads need defined replacement or idempotency semantics; two clients completing the same upload need one terminal result; an expired session needs an explicit response rather than accidental resurrection.

Checksums answer different questions. A transport or part checksum detects corruption over a bounded transfer. A full-object checksum binds the ordered completed bytes. A cryptographic digest can also serve as content identity or deduplication evidence, but only when the trust and collision model permits it. Encryption changes which bytes are checksummed and whether equal plaintext should reveal equality. Record algorithm, scope, representation, and whether the value is supplied, verified, stored, or recomputed.

Completion must not publish metadata before the content and manifest meet the durability contract. A safe sequence stages parts, verifies length and checksums, durably records the manifest, publishes a new namespace generation, and makes cleanup of obsolete staging state asynchronous. If the coordinator fails after manifest durability but before replying, a repeated completion should return the same committed version. If policy validation fails, the object remains uncommitted even if every part exists.

Abandoned sessions consume bytes and metadata. Expiry, abort, and garbage collection must tolerate races with in-flight part writes and completion. Mark terminal state first, fence late writes by generation, then reclaim unreachable parts after a safety interval. Track staged bytes, oldest session, completion rate, abort rate, and reclaimed bytes by tenant. Quotas must include staged data or an attacker can avoid committed-storage limits.

Replication and erasure coding buy different recovery behavior

Replication stores complete copies. Three replicas have 3× raw-space overhead before metadata and reserve. Reads can choose among complete copies, small writes are straightforward to reason about, and repair can copy from one healthy replica. The cost is capacity and write traffic.

An erasure code with k data shards and m parity shards converts one stripe into k + m fragments and can reconstruct within its failure assumptions. A 10+4 model has (10 + 4) / 10 = 1.4× raw-space overhead. For 8 PiB logical, the simple comparison is 24 PiB under three-way replication versus 11.2 PiB under 10+4 coding. It excludes filesystem reserve, metadata, versions, staging, compaction, alignment, and free space needed for rebuild.

The capacity saving is not free. Reads may require several fragment owners; partial writes can create read-modify-write or full-stripe work; encode/decode consumes CPU; fragment placement must span real failure domains; and rebuild can read far more bytes than the missing shard contains. Small objects may be rounded, packed, or placed in a replicated tier because per-object coding and request amplification dominate.

Choose with a workload and failure matrix:

Do not infer durability from a nominal replica or parity count. Durability depends on correlated failure domains, latent corruption, detection time, repair start and completion, free capacity, control-plane availability, and whether a second failure arrives during degradation. An annual probability slogan is not a repair plan.

The fixture deliberately models a conservative 10+4 reconstruction in which rebuilding 96 TiB of missing fragments reads ten fragment-equivalents—960 TiB—and writes 96 TiB. The shared fabric supplies 80 GiB/s, but foreground work keeps 48 GiB/s and safety keeps 8 GiB/s. Repair receives 24 GiB/s, so 1,056 TiB of modeled shared-fabric traffic needs 45,056 seconds, about 12.52 hours. A local or regenerating code, cached source, bottlenecked destination, cross-rack topology, CPU limit, or several concurrent failures changes that calculation. The point is to expose every byte and reserve, not to recommend 10+4.

Track degraded logical bytes, missing fragments by failure domain, oldest degradation, audit coverage, reconstruction read/write bytes, repair useful throughput, throttling reason, remaining time range, and foreground impact. Rehearse repair from real placements. A checksum audit that detects corruption but cannot finish before the next likely fault is incomplete durability control.

Metadata skew can defeat uniform byte placement

Namespace partitioning might hash the full object identity, range-partition a tenant and prefix, or use a routing directory. Hashing spreads keys but makes ordered listing scatter. Range placement supports prefix listing but concentrates sequential names and large tenants. A directory can move hot ranges and enforce locality, at the price of another authoritative mapping and migration protocol.

Prefixes are semantic only if the implementation makes them so. Do not repeat inherited advice to randomize key prefixes without measuring the actual partition function and current service behavior. Instead, test exact operations: create, head, list, delete, multipart state, policy changes, and hot-object reads. Separate a hot object from a hot prefix and from a hot tenant; each needs a different control.

Listings need explicit snapshot and pagination semantics. If objects are created or deleted between pages, can the client see duplicates, omissions, or a mixed generation? Does a continuation token bind filter, order, tenant, and snapshot? A direct read of a committed version can be strongly consistent while an inventory or derived index is delayed. State each boundary separately:

• read-after-create for the exact key and version;
• overwrite visibility for named and latest versions;
• deletion/tombstone visibility;
• listing membership and ordering;
• metadata/tag/policy update visibility; and
• cross-region replication and failover visibility.

Metadata caches must be generation-aware and authorization-safe. Negative caching lowers repeated misses but can conceal a newly created key. Client retries can amplify a metadata incident, especially when every content transfer first asks for a fresh capability. Bound retry work, shed expensive listings, and retain an emergency direct-version path only if it preserves authorization and authority.

An edge cache is a distributed derived index

A cache key names the reusable representation. At minimum consider scheme, host, normalized path, relevant query parameters, selected request headers, encoding, rendition, authorization scope, and object generation. Omitting a dimension can serve the wrong bytes or cross a tenant boundary. Including every volatile dimension fragments reuse and turns ordinary demand into misses. Record why each dimension changes representation identity.

Prefer immutable versioned URLs for public assets. They permit long lifetimes because a new object gets a new identity. Mutable names require revalidation, short freshness, purge, or generation lookup. Purge is itself a distributed control operation with propagation, retry, and audit; it should not be the only correctness mechanism for sensitive content.

An origin shield or upper-tier cache narrows edge-to-origin fan-in:

client -> edge cache -> regional shield -> upper shield/origin
                    miss key | single in-flight fetch

Request collapsing matters as much as the hierarchy. Concurrent misses for one key should join one bounded fetch rather than each opening an origin transfer. Collapse identity must include the correct cache key and range policy. Followers need deadlines; a stuck leader cannot hold unlimited waiters. Negative responses and errors need deliberately short or disabled caching so an incident does not become durable.

The modeled miss event has 180 edges each receiving 280 requests/s for the same 8 MiB rendition for 30 seconds. With no cache reuse, shield, or collapse, that is 50,400 origin requests/s and 393.75 GiB/s, totaling 1,512,000 requests and about 11.54 TiB. If six regional shields each collapse the wave into one complete origin fetch, the origin sees six requests and 48 MiB: a 252,000× request reduction in this intentionally favorable model. Real keys, ranges, arrival timing, eviction, shield failure, and cacheability make the result worse.

Protect the origin with a miss budget, not only a client rate limit. Bound concurrent fills and fill bytes per key, tenant, region, and origin. Serve stale content only where correctness permits it. Pre-position predictable launches; ramp a purge; isolate personalized or uncacheable traffic; and reserve origin capacity for control and correctness reads. When a shield fails, lower tiers must not all bypass it simultaneously. Use alternate shields, admission, jittered retry, and explicit degraded behavior.

Observe hit ratio by bytes as well as requests. A 99% request hit ratio can still miss the largest objects. Track request/byte hits, revalidations, fills, collapsed waiters, eviction reason, key cardinality, object age, range behavior, origin concurrency, origin egress, and status by cache layer. Global ratios hide the one region or rendition exhausting origin.

Ranges and adaptive media change the unit of reuse

HTTP byte ranges let a client request one or more portions of a selected representation. A valid range response identifies both returned offsets and complete representation length. Conditional range requests need a validator so bytes from two object generations are not spliced together. Reject pathological multi-range requests or normalize them into bounded work; many tiny ranges can cause more metadata, seeks, and response framing than one sequential transfer.

Chunk boundaries, encryption blocks, compression, coding stripes, CDN cache units, and media segments need not align. Misalignment creates read amplification: a 1 MiB client range may fetch several coded fragments, decrypt a larger block, and fill an 8 MiB cache segment. Measure useful returned bytes versus storage reads and inter-tier bytes by request shape.

Adaptive media normally selects among separately encoded renditions and time segments. The manifest is a small, freshness-sensitive object; segments are larger and often immutable. Cache identity must include rendition and version. Player startup, seek, and bitrate switching create bursts and overlapping ranges. Capacity tests should replay session behavior, not issue uniform full-object downloads. Geographic placement should minimize user transfer latency without weakening residency, encryption-key, deletion, or source-authority rules.

Upload acceleration can terminate near the client, retain verified parts, and transfer them over a managed backbone to the durable region. It helps when the public path is lossy or high-latency, but it adds staging authority and cleanup. Define whether an edge acknowledgment means “received nearby” or “durable at the destination.” Download placement can use cached derived copies; upload placement changes the write path and therefore requires stronger language.

Lifecycle work competes with foreground work

Objects move through hot, warm, cold, archived, deleted, and reclaimed states according to access evidence, retention, compliance, and economics. A transition may copy and recode bytes, update a manifest, and delete the old placement. It is a migration protocol: establish destination durability, atomically change authority, then reclaim the source after rollback and reader leases expire.

Packing small immutable objects reduces request and space amplification, but deletes become tombstones until compaction rewrites live entries. Compaction needs temporary space and read/write bandwidth; a crash must not lose either the old or new index. Track dead-space ratio, pack age, compaction debt, rewrite amplification, and deletion deadline. Do not compact solely to recover pennies if it steals the bandwidth required for repair.

Deletion has at least three moments: no longer visible, no longer recoverable through ordinary product paths, and physically reclaimed from every governed copy. Retention locks, snapshots, replicas, caches, multipart staging, packs, derived renditions, and audit logs can each delay the third. State legal and product semantics separately. A tombstone must be durable and ordered against late replication so deleted bytes do not reappear during recovery.

Garbage collection should prove unreachability from a stable namespace generation, then wait through a safety window before reclaim. A raw scan that deletes “unknown” chunks while metadata is partially unavailable can turn a control-plane incident into data loss. Mark-and-sweep, reference counts, or manifests each require a failure-safe authority and repair story.

The object-system capacity ledger

Use a worksheet that keeps means, tails, peaks, and degraded modes separate:

Workload and identity
  operation/object-size distributions, tenant/prefix skew, versions,
  range/session patterns, cache-key dimensions, geographic modes

Metadata plane
  create/head/list/delete/part RPS, records and bytes per object,
  partition heat, cache behavior, listing snapshot/pagination boundary

Content plane
  foreground upload/download/range GiB/s, active flows, useful-byte ratio,
  chunk/stripe/request amplification, staging and incomplete-upload bytes

Durability
  replication or k+m code, real placement domains, audit rate,
  degraded bytes, reconstruction read/write/CPU, repair completion target

CDN and origin
  request and byte hit ratios by layer, fill concurrency/bytes, collapse,
  purge/key-change/failover miss envelope, stale/error behavior

Lifecycle
  tier transition bytes, pack/compaction debt, tombstones, reclaim proof,
  temporary free-space and bandwidth reserve

Capacity decision
  normal/peak/degraded/recovery envelopes, foreground reserve,
  miss reserve, repair reserve, admission rules, rollback triggers

Evidence
  observed/modeled fields, date/version, raw telemetry or fixture,
  uncertainty, failure rehearsal, cost boundary, transfer limit

Capacity by stored bytes alone misses metadata, request, and egress ceilings. Capacity by normal egress alone misses the regional cold-start. Capacity by raw device bandwidth misses read/write amplification and topology. Use the maximum requirement at each independent bottleneck, then verify their combinations: foreground peak plus repair, purge plus launch, compaction plus device failure, and regional failover plus cold caches.

Applied work

Design the unreliable 12 GiB upload. Choose part size from retry cost, parallelism, request overhead, client memory, checksum cost, and service limits—not from the fixture alone. Specify upload identity and expiry; part receipt durability; duplicate-part semantics; client persistence and resume; full-object verification; atomic version publication; ambiguous completion retry; quota for staged bytes; abort and garbage collection; observability; and failure tests. Rehearse loss after part acknowledgment, during completion, and after commit before the response.

Protect origin during a regional miss wave. Inventory every event that changes cache identity or residency: launch, purge, deployment, certificate/host change, rendition change, failover, and shield reassignment. Model unique keys and bytes, not requests alone. Set per-origin fill concurrency and bandwidth; collapse identical fills; stage predictable objects; constrain range fragmentation; define stale eligibility; isolate uncacheable work; and test loss of one shield. The acceptance result is bounded origin work and a declared client degradation mode, not merely a recovered hit ratio.

Audit durability under foreground peak. Select a real failure domain, compute fragments and bytes to reconstruct, identify source/destination hot spots, reserve CPU/network/storage work, and inject a representative foreground peak. Verify repair completion range, foreground objective, audit coverage, free-space floor, and behavior if another domain fails. If the system must choose, document which traffic is shed to shorten exposure.

Sources and transfer limits

• Amazon S3 multipart-upload documentation describes independent parts, completion, and checksum behavior for current S3 APIs. It supports the protocol discussion but does not define limits or durability semantics for another implementation.
• RFC 9110, HTTP Semantics defines current HTTP range and validator semantics. It does not prescribe storage chunking, CDN cache policy, or protection against application-specific expensive ranges.
• Ceph erasure-code documentation documents the (k+m)/k overhead relationship and warns about recovery, backfill, and small-object trade-offs in Ceph. The chapter’s 10+4 reconstruction is a separate conservative model, not a Ceph benchmark or default recommendation.
• Cloudflare cache-key documentation and tiered-cache documentation show how one current CDN defines cache identity and narrows origin access through tiers. Product behavior and defaults are version-specific; the transferable principle is explicit identity plus bounded miss fan-in.

The numeric examples are reproduced by examples/performance-engineering-system-design-handbook/part-05/object-cdn/. They are deterministic models, not observed service results. A real decision needs measured size and operation distributions, cache-key cardinality, placement, code implementation, repair topology, foreground objectives, hardware/network, cost, and failure evidence.

Decision rule

Separate metadata scaling from content transfer. Give cache-miss recovery and durability repair their own admission and bandwidth reserves, then test them concurrently with foreground peak. Choose replication, coding, placement, caching, ranges, and lifecycle rules only after their authority, amplification, failure domains, and reclaim semantics are explicit.

Large-object systems amplify bytes across parts, fragments, tiers, and repair. Real-time collaboration changes the multiplier: one small event can become work for hundreds of thousands of recipients, each with its own session, buffer, and recovery position. Chapter 46 follows that fan-out all the way to the slowest client.