Storage Media and I/O Paths

A storage design is incomplete until it can finish this sentence: the operation is acknowledged after which bytes, in which state, have crossed which failure boundary?

“The write returned” does not answer it. A return may mean that an application buffer was copied, a kernel page became dirty, a file-system journal accepted metadata, a device cache accepted a command, a replicated block service accepted a quorum, or stable media completed the write. Those events can differ by milliseconds and by the failures they survive. Comparing their latency as if they were the same operation produces a fast chart and an undefined system.

Follow one logical operation from the application call to the declared stable-storage boundary. The decision is not which storage product wins, but which access path, concurrency policy, durability boundary, integrity work, and evidence can satisfy the application’s objective. Replication semantics above a block service are a separate design problem, as are query and index choices. Here the machinery is bytes, requests, queues, ordering, and completion.

Five quantities describe the offered I/O

Storage work must be described as a distribution of operations, not as “disk traffic.” For each operation class, record:

• request size, in bytes, and whether it crosses alignment or allocation boundaries;
• access pattern, including sequentiality, random offsets, hot ranges, and read/write mix;
• arrival process, including bursts and correlations with checkpoints or compaction;
• queue depth, meaning outstanding requests at the measured boundary;
• completion requirement, such as data copied, read satisfied, write ordered, or bytes durable.

From these inputs come three different rates. Input/output operations per second (IOPS) counts completions. Throughput counts bytes completed per unit time. Latency measures elapsed time for one declared population. For a fixed request size (b), their first-order relation is:

[ X = I \times b ]

where (X) is throughput in bytes/s and (I) is IOPS. This identity is accounting, not a performance prediction. An implementation can merge adjacent requests, split a large request, retry, journal, compress, encrypt, or relocate data. Host-visible completions and physical-media operations need not have a one-to-one relationship.

Queue depth is similarly contextual. At an asynchronous submission ring it counts issued operations not yet reaped. At a device it may count commands after file-system merging. At a remote service it may omit network and client-pool waiting. Little’s Law can connect average outstanding work, completion rate, and average residence time for a stable population, but it cannot recover a missing queue or explain a tail by itself. State the boundary for every depth.

The useful comparison is therefore an I/O envelope: operation mix × request-size distribution × locality × offered concurrency × durability mode × data state. A single maximum IOPS value is one point in that envelope.

Media preserve different costs

A hard disk drive must position a mechanical head and wait for a rotating surface. Sequential transfers can amortize positioning and exploit streaming bandwidth; small random operations repeatedly pay movement and rotational delay. A queue can reorder independent requests to reduce movement, but that changes completion order and may harm deadlines or fairness.

Flash removes mechanical seek, not internal work. A solid-state drive typically maps host logical block addresses through a controller and a flash translation layer. NAND flash is read and programmed in page-like units but erased in larger blocks. Updating a small logical range can require allocating new physical pages, invalidating old mappings, later copying still-live pages, and erasing a block. Controller parallelism can spread work across channels and dies, while garbage collection and wear management can create background demand.

Persistent-memory technologies occupy design points between volatile memory and block devices, but “byte addressable” or “persistent” is not an application commit protocol. CPU caches, memory-controller queues, ordering instructions, power-failure protection, and recovery metadata still define when a data structure is recoverable. Treat the term as a media capability and ask which software and hardware boundary provides persistence.

Remote block storage adds a transport, service scheduler, shared fleet, replication or erasure coding, and control-plane behavior beneath a local-looking device. The client still sees block commands, but its latency distribution can include network paths, remote queues, failover, throttles, and noisy neighbors. A local device result does not transfer merely because both targets expose the same block size.

These media differ in mechanism, but the decision variables remain stable: request shape, parallelism, locality, steady versus burst state, read/write proportion, durability, endurance, failure scope, and recovery work.

Request size and depth trade latency for parallel work

Larger requests reduce per-command overhead per byte and often expose sequential bandwidth. They can also read unused bytes, occupy a queue longer, consume larger buffers, and delay small urgent operations behind bulk work. Higher queue depth gives a device or service more work to schedule across parallel resources. Past the useful parallelism, it mostly creates residence time.

The simulated Vaultspan fixture illustrates the shape; it is not a device specification:

Every row satisfies IOPS × request size = MiB/s. That consistency does not make the values empirical. It makes the claimed populations inspectable. The table shows why “fewer IOPS” can carry more data and why a throughput win can coexist with a latency loss. It also makes an application question unavoidable: does Vaultspan need one 4 KiB record soon, or a 1 MiB segment eventually?

A depth sweep should plot offered depth, achieved depth, completion rate, and latency distribution together. If throughput stops rising at depth 8 while p99 continues rising through depth 64, the extra 56 outstanding commands are backlog, not capacity. If achieved depth stays at one, inspect the I/O engine and buffering path before attributing the flat curve to media. The fio 3.42 documentation explicitly warns that synchronous engines do not gain effective depth from a larger iodepth, and that some asynchronous/buffered combinations do not achieve the requested depth.

Reads may stop at memory or descend to media

A buffered file read commonly asks the operating system’s page cache first. A hit can copy or map already resident data without issuing media I/O. A miss allocates or selects cache pages, submits lower-layer work, waits or arranges completion, and may trigger read-ahead. That is useful application behavior, but it means a repeated small-file test can become a memory benchmark.

Buffered writes usually dirty cached pages and return before writeback. The kernel can merge and schedule later writes, smoothing bursts and improving locality. It also creates dirty-memory state, writeback thresholds, and a later burst that competes with foreground reads or flushes. The application must distinguish write-call latency from durable-acknowledgment latency.

Direct I/O asks to reduce or bypass page-cache participation, subject to operating-system, file-system, device, and alignment rules. It can prevent double caching and give an application more explicit buffer ownership. It does not remove all caches, controller work, copying, or alignment cost. Linux’s open(2) documentation notes that O_DIRECT semantics and restrictions vary and warns against mixing direct I/O with overlapping mmap access. “Direct” is a path choice, not a synonym for durable or zero-copy.

Memory mapping exposes file-backed pages through a process address space. It can make random access and shared pages convenient; page faults then become implicit I/O submissions, and writeback or durability still requires explicit reasoning. Fault latency may arrive at an arbitrary load instruction. Mapping a file does not make failed storage disappear from the critical path.

Synchronous and asynchronous describe how a caller waits and how operations are issued, not whether bytes are durable. A synchronous read can hit memory. An asynchronous write can later require a durability fence. Asynchrony permits overlap and controlled queue depth; it also creates cancellation, buffer-lifetime, completion-order, and error-drain obligations. Chapter 11’s scheduling rule carries through: make waiting and ownership visible.

Durability is an ordering claim

On Linux, a successful fsync() requests transfer of a file’s modified in-core data and associated metadata needed for retrieval to the storage device. The fsync(2) manual also cautions that synchronizing the file does not necessarily synchronize the directory entry; applications that require the directory operation to survive need to synchronize the directory separately. That distinction is a recovery invariant, not trivia.

Beneath the file system, volatile write-back caches complicate completion. Linux block-layer documentation describes cache flushes and Force Unit Access as mechanisms by which file systems order data-integrity operations against device caches. A device or service must honor the declared semantics for the guarantee to hold. Disabling flushes because a benchmark is slow changes the operation being measured.

Ordering matters whenever recovery reads multiple related writes. Consider a log record and a pointer that makes it visible. If the pointer reaches stable storage first, a crash can expose an incomplete record. Journaling, copy-on-write metadata, write-ahead logging, and application commit protocols arrange data and metadata so recovery can distinguish complete from incomplete transitions. The file system cannot infer an application’s multi-file invariant; the application cannot assume a write-return ordered all lower layers.

Group commit amortizes durability work. Vaultspan accepts 8,000 logical 4 KiB record appends/s, or 31.25 MiB/s. With 16 writes per durability group, it performs 500 group flushes/s rather than 8,000 flushes/s. The throughput benefit is purchased with a batching wait and a shared failure fate. A 12.8 ms p99 acknowledgment can be acceptable for an archival segment and unacceptable for a 5 ms transactional deadline. The group needs a maximum age as well as a maximum count so light traffic does not wait indefinitely.

Trace the acknowledgment in order. When the application copy completes, the caller may be free to reuse its buffer; the kernel or device need not have accepted a byte. When a buffered write returns, dirty data may exist only in the page cache. An ordinary device completion advances the boundary to the device or service’s completion point, but says nothing about draining a volatile cache unless the interface promises it. A flush- or FUA-ordered completion can carry prior writes across the declared nonvolatile boundary; it still cannot make application metadata recoverable if the commit protocol ordered that metadata incorrectly. Only the transaction or group-commit completion makes the protocol’s declared record set recoverable, and even then replicas outside that protocol boundary may not have applied it.

Choose which of those events the application calls success. Then inject a failure immediately after it and ask recovery to find every acknowledged record and reject every incomplete transition. A latency result without that recovery oracle has measured a return path, not the claimed durability operation.

Logical bytes can multiply below the application

Write amplification is a ratio, and every ratio needs a boundary. Define:

[ WA_{media} = \frac{\text{physical bytes programmed to media}}{\text{logical application bytes accepted}} ]

Vaultspan’s simulated 64 GiB logical write interval produces 80 GiB after file-system data and journal work, 96 GiB of host writes visible to the device, and 240 GiB of physical media programming. The three useful ratios are 1.25× at the file-system boundary, 1.5× at the device-host boundary, and 3.75× at the media boundary. Calling all three “write amplification” without a subscript conceals the responsible layer.

Sources of multiplication include small overwrites, journal or copy-on-write metadata, parity, checksums, encryption block handling, compaction, flash relocation, and retries. Compression can reduce bytes at one boundary while increasing CPU service demand. Deduplication can reduce media bytes while adding index I/O and a new correctness boundary. Measure the numerator you intend to improve.

Amplification consumes bandwidth and endurance. Flash endurance is a finite write-work budget expressed by a device under specified conditions; application lifetime estimates must include amplification, spare capacity, failure replacements, rebuild traffic, workload drift, and uncertainty. A high-level 100 GiB/day estimate is not safe if the media sees 375 GiB/day and recovery periodically doubles that work.

Background work creates the severe tail

Read-ahead, delayed writeback, journal checkpoints, snapshots, scrubbing, encryption, garbage collection, log cleaning, compaction, repair, and rebuild all use the same finite paths as foreground work. An average over idle and busy minutes can look healthy while the foreground objective fails predictably during one phase.

Vaultspan’s ten-minute packet reports a 1.4 ms average write completion and a 48 ms p99, a 34.3× ratio. During minutes four through six, background compaction drives 610 MiB/s while foreground logical writes contribute only 31.25 MiB/s. Peak device queue depth reaches 93, and checksum failures remain zero. The fast average and severe p99 are compatible: most writes arrive outside the interference interval, while those inside wait behind background work or its downstream consequences.

The discriminating evidence is time aligned:

1. application submission, acknowledgment, timeout, and cancellation populations;
2. file-system dirty pages, writeback, journal/checkpoint, and compaction state;
3. block-layer issued/completed bytes, achieved depth, latency, merges, and errors;
4. device or service utilization, internal background work where exposed, throttling, and media errors;
5. remote-path latency, retransmission, service limits, and failover events;
6. CPU demand for checksums, encryption, copying, completion handling, and interrupts.

A full device can have idle-looking moments while a hidden queue exists above it. A busy device can still meet an objective if work is large, ordered, and deadline tolerant. Utilization alone does not rank these states; queue age and useful completion outcomes do.

The first controlled change should alter one causal term: cap compaction bandwidth, reserve foreground queue slots, shift the checkpoint, or isolate the test file. If p99 moves with the controlled term at equal workload, the inference strengthens. Increasing global queue depth during the same test adds backlog and destroys attribution.

Remote storage makes the path a distributed system

Network-attached storage needs two extra ledgers. The latency ledger separates client queue, transport, service queue, media, replication, and response. The durability ledger states which replicas or coded fragments must accept data, what “stable” means across power and zone failure, and what happens during degraded writes.

Variability can come from shared bandwidth, noisy tenants, service-side admission, burst credits, maintenance, placement, failover, rebuild, or a longer network route. Some limits are IOPS-shaped, some byte-rate-shaped, some capacity-shaped, and some per-volume or per-instance. Record the exact service class, size, attachment topology, limits, encryption mode, region/zone, and date for a product-specific result. Do not turn one dated configuration into a media law.

Local storage has different failure and migration costs. It can remove a network hop and shared service queue while binding state to a host, changing evacuation, rebuild, and replacement. A lower nominal latency is not automatically a lower recovery objective. Compare normal, failed, and rebuilding states.

Integrity work belongs inside the budget

Checksums detect some forms of corruption; they do not repair data by themselves. End-to-end protection requires defined coverage, stored expected values, verification points, error propagation, and a trusted repair source. A checksum calculated and checked on the same corrupted path can miss faults outside its boundary. Metadata needs protection as much as payload.

Encryption adds CPU or accelerator work, key access, block or record framing, and sometimes read-modify-write behavior. Modern processors can make its steady-state throughput cost small for some workloads, but that is an environment-specific measurement. Tail effects can instead appear through key rotation, cold key caches, fallback paths, alignment, or shared accelerator contention. Benchmark with the production integrity modes enabled.

Integrity validation must survive performance pressure. A load generator that sets verify=0, ignores short writes, or never reads data back can report impressive corrupted goodput. fio supports content verification and records per-block metadata for appropriate workloads; its documentation also notes combinations that can invalidate verification assumptions. Application invariants still require an application oracle above block correctness.

Separate the layers with a benchmark ladder

One benchmark cannot simultaneously isolate media and represent the application. Use a ladder whose rungs answer different claims:

For each rung, preserve the machine-readable parameter sheet from the fixture. At minimum it names claim; target; kernel, file system, mount, device/service, topology, encryption, and date; engine and achieved depth; buffering; access patterns; request sizes; data-set-to-memory relation; preconditioning and free-space state; warm-up, duration, repetitions, order, and cooldown; durability modes; outcome populations; correctness; raw output; and transfer limits.

Design the Vaultspan experiment in three contrasts. First, warm the test range and measure buffered reads to bound the cache-hit path. Second, use a data set larger than assigned memory and verified cache-miss controls to expose the file-system/device path. Third, compare write-return, group-fsync, and fsync-per-write modes on the same preconditioned target. A separate direct-I/O phase can help isolate page-cache effects where the target supports it. Do not run all phases against the same steadily filling device without reset or randomized order; free-space and background state would become treatments.

The retained artifact lives at examples/performance-engineering-system-design-handbook/part-02/storage-io-paths/. It deliberately computes modeled arithmetic rather than exercising this host’s storage. Running it is safe:

node examples/performance-engineering-system-design-handbook/part-02/storage-io-paths/run.mjs
node examples/performance-engineering-system-design-handbook/part-02/storage-io-paths/verify.mjs

Choose by boundary, not headline

Field diagnostic: explain the 48 ms p99

Use the fixture’s 1.4 ms average, 48 ms p99, queue depth 93, 610 MiB/s compaction, 31.25 MiB/s foreground demand, and zero checksum failures. Rank at least three hypotheses. Specify one measurement that distinguishes background interference from client batching, remote-service throttling, and application lock contention. A strong answer aligns operation, file-system, block, device/service, and CPU timelines; it does not infer media failure from p99 alone.

Principal design drill: benchmark an acknowledged append

Vaultspan must acknowledge 8,000 4 KiB logical records/s with a 15 ms p99 durability objective, survive the declared host-power boundary, and recover without an acknowledged-record gap. Design the benchmark ladder and group-commit policy. Include request size, offered arrival process, queue control, data-set state, preconditioning, integrity oracle, flush semantics, local-versus-remote variants, background compaction, repetitions, raw-data custody, and abort criteria. Then assume the remote block path enters rebuild and its write p99 triples. Revise admission, group age, recovery headroom, and the claim you are still willing to make.

State the durability boundary and access pattern before comparing storage numbers. Once the path is explicit, storage stops being a mysterious box: bytes are copied, queued, ordered, amplified, checked, transported, persisted, and sometimes repaired. When that path becomes remote, propagation, congestion, retransmission, and protocol queues join the same ledger.

Sources and evidence scope

• Linux fsync(2) manual, man-pages 6.18 defines current Linux file and metadata synchronization semantics, including the separate directory-entry caveat. Other operating systems and remote file systems may differ.
• Linux open(2) manual, man-pages 6.18 documents current Linux O_DIRECT, synchronization flags, alignment discovery, and path-specific limitations. It does not make direct I/O universally faster or durable.
• Linux kernel volatile write-back cache control explains block-layer flush and Force Unit Access ordering against volatile device caches. Device and virtualized-service compliance remains part of the tested boundary.
• Linux virtual file-system documentation documents page-cache address-space, dirty, and writeback concepts for current Linux kernels.
• fio 3.42 documentation defines engines, depth, workload, steady-state, reporting, and verification controls used by the benchmark parameter sheet. fio generates evidence only when the job and environment match the claim.
• All Vaultspan request-matrix, durability, amplification, queue, compaction, and latency values are simulated teaching evidence in examples/performance-engineering-system-design-handbook/part-02/storage-io-paths/; they are not observations of this host, a storage product, or a production system.