Networks, Protocols, and the Cost of Distance

A retransmission is an event, not a diagnosis. It says useful bytes were sent again; it does not identify whether queueing, loss, reordering, receiver delay, or descheduling withheld them.

The packet trace looks incriminating. Mercury API’s p99 rose from 86 ms to 174 ms while retransmitted bytes reached 0.8%. Yet the remote service’s processing span moved only from 11.8 ms to 12.1 ms, and its application-pool wait remained 4.4 ms. The non-congested round-trip time was 36 ms; during the event it reached 78.7 ms. A bottleneck interface recorded 37 queue drops. The retransmissions are real, but “the network lost packets” is still not a sufficient diagnosis.

The useful question is where elapsed time entered the request. The trace supplies several discriminators: round-trip inflation began before retransmissions, server time stayed nearly flat, the receiving window never closed, and drops appeared at the bottleneck. The best current inference is queueing followed by loss, not slow application work. A controlled load reduction or queue-policy change can test it. The same retransmission counter with flat round-trip time and a closed receive window would support a different mechanism.

Network performance becomes tractable when it is treated as a ledger. Distance contributes propagation. Bytes consume transmission time. Protocol state introduces round trips. Every layer may queue. Endpoints spend CPU on framing, copying, encryption, checksums, routing, interrupts, and application work. Loss and reordering change when useful bytes become available. The reader’s decision is which of those terms to remove, bound, or measure—not which networking feature sounds fastest.

A request pays five kinds of delay

For a path with several links and processing stages, one-way latency can be modeled as:

[ T_{one\text{-}way} = T_{prop} + T_{tx} + T_{ser} + T_{queue} + T_{proc} ]

where every term is a distribution for a declared path and traffic class:

• Propagation delay is time for a signal to traverse the physical route. Geography is a lower bound, not the route: fiber paths bend, equipment adds hops, and return routing may differ.
• Transmission delay is the time to place a frame or packet’s bits onto a link. For a payload of (L) bits and link rate (C) bits/s, one-link transmission time is (L/C), before framing and contention.
• Serialization and framing here cover application encoding, compression, encryption records, RPC envelopes, and packetization. Do not confuse application serialization with the physical act of clocking bits onto a link.
• Queueing delay occurs wherever arrivals wait: connection pools, executor queues, socket buffers, transport flow-control credit, queueing disciplines, NIC rings, switches, middleboxes, and the peer’s admission queue.
• Processing delay includes packet and protocol handling plus application service demand. It may occur on CPUs, NICs, accelerators, proxies, or remote services.

Round-trip time (RTT) is not automatically twice one-way delay. It includes forward and return paths, their independent queues, and endpoint response behavior. Clock-synchronized one-way measurements can expose asymmetry; a client-observed RTT cannot assign its two halves without additional evidence.

For Mercury’s warm successful-read population, the deterministic model allocates a 95 ms p99 objective as follows. Every value is simulated teaching evidence, not a production measurement:

The table is an accounting model, not a promise that p99 terms add exactly. Component percentiles are generally drawn from different requests and must not be added as if they occurred together. Here they are explicitly allocated budget terms for one modeled tail path. In a trace-based analysis, sum per-request spans or model dependence; do not sum fleet p99 dashboards.

Distance turns bandwidth into in-flight state

A high-rate path must hold enough unacknowledged data to keep transmitting while feedback travels. The bandwidth-delay product is:

[ BDP = C \times RTT ]

where (C) is the bottleneck rate in bytes/s and (RTT) is seconds, producing bytes in flight. For a 10 Gbit/s path with a 36 ms RTT:

[ BDP = \frac{10 \times 10^9\ bit/s}{8} \times 0.036\ s = 45{,}000{,}000\ bytes ]

About 45 MB must be in flight to fill the path under the model. Sender congestion state, receiver flow-control window, socket buffers, and application production/consumption must permit at least that much useful outstanding data. A 4 MB effective window bounds steady transfer near 4 MB / 0.036 s, about 111 MB/s, even though the link is nominally 1.25 GB/s.

BDP is not a command to enlarge every buffer. Large queues can hide bursts while converting overload into latency. The required in-flight window carries data across propagation; excess queued data waits without increasing the bottleneck’s service rate. Measure delivered goodput, RTT under load, flow-control blocking, congestion window, and queue occupancy together.

The counterexample is a small RPC. Mercury’s 256 KiB payload needs only about 0.210 ms of ideal transmission time on a 10 Gbit/s bottleneck, excluding headers. Saving 10% of that time is immaterial beside a 36 ms remote RTT. Removing one sequential remote round trip can save roughly the propagation budget of the entire call. For a 5 GiB stream, by contrast, in-flight windows and sustained goodput dominate. The same path supports different optimizations because the unit of work changed.

Loss changes delivery; congestion control changes sending

Packets may be dropped, corrupted, duplicated, or reordered. A reliable transport detects missing data and retransmits according to acknowledgment and timer evidence. Because TCP presents an ordered byte stream, later bytes are not delivered to the application past a gap until the missing range arrives. This is transport-level head-of-line blocking.

Retransmission does not identify the cause of the missing acknowledgment. Candidates include a congested bottleneck drop, physical loss, path change, reordering that made loss detection fire, receiver delay, capture loss, or a sender timer expiring while the endpoint was descheduled. Packet evidence must be tied to capture location and clocks. A capture on the sender before a NIC offload can look unlike one on the wire; a busy capture host can itself drop packets.

Congestion control limits how aggressively a sender injects data based on path feedback. TCP’s exact algorithm is implementation- and configuration-specific, but the durable model is a sender-controlled congestion window, receiver-advertised flow-control window, acknowledgments, loss or congestion signals, and pacing. Loss recovery replaces missing data; congestion control protects a shared path. They interact but answer different questions.

QUIC also provides recovery and congestion control, but its reliability is scoped to streams carried over UDP. Loss can delay stream data that was in a lost packet without forcing unrelated streams to wait for the missing byte offset. Connection-wide congestion control and connection flow control still couple the streams, and a blocked receiving application can withhold stream or connection credit. “No head-of-line blocking” is therefore too broad; the claim must name the layer.

Reordering deserves its own column in a diagnosis. A changed path or parallel links can deliver packets out of order without losing them. Modern loss detectors use timing and packet-threshold evidence rather than treating every gap as immediate loss. Capture duplicate acknowledgments, selective acknowledgment ranges, retransmission timing, original and retransmitted sequence ranges, and whether both copies arrive. A retransmission whose original later arrives suggests reordering or a premature loss decision, not necessarily a drop.

Connections spend round trips before useful work

A cold call may need name resolution, address selection, transport establishment, cryptographic negotiation, authentication, and then the application exchange. Some of these can overlap, resume, or be cached; none should be assumed absent merely because the application issued one function call.

In the fixture’s deliberately simple TCP plus TLS 1.3 model, remote DNS adds 9 ms, the TCP handshake adds one 36 ms RTT, and a full TLS handshake adds another 36 ms RTT before ordinary application data. The cold premium is 81 ms. This is a modeled sequence, not a universal TLS claim: session resumption, TLS early data, TCP Fast Open, QUIC, address racing, and implementation scheduling change the timeline and the security/replay contract.

Pooling amortizes establishment only when the pool is healthy. Record:

• checkout wait and configured maximum, not only socket count;
• active, idle, connecting, draining, expired, and failed connections;
• connection age and reuse count;
• destination, protocol, credential, and policy identity that defines safe reuse;
• per-connection stream or request concurrency;
• peer admission, idle timeout, maximum age, and drain signals;
• failure behavior when a destination or address changes.

An unlimited pool can overwhelm the peer and create more handshake work during failure. A tiny pool can serialize unrelated calls. A large multiplexed connection reduces handshakes but concentrates loss, flow-control state, and failure. A useful policy bounds both connections and work per connection, queues excess demand with a deadline, warms capacity before a launch, and drains old destinations without synchronized reconnection.

QUIC integrates transport and TLS establishment and can permit application data sooner than a fresh TCP-plus-TLS sequence. Zero-RTT data trades latency for replay and freshness constraints; it is appropriate only for application operations whose semantics and server policy tolerate them. Connection migration can preserve a connection across some address changes, but path validation and changed RTT/congestion state still matter. Protocol names do not remove application invariants.

Choose a service, not a transport label

The application needs semantics before it needs a protocol.

TCP does not preserve writes as records: one write may be read in pieces, and several writes may be coalesced. RPC framing therefore needs an explicit length or delimiter, maximum frame size, validation, and behavior for truncation. A receiver that reads a length before enforcing a maximum can allocate unbounded memory. A sender that buffers an entire 400 MB response before the first write defeats streaming even if the API type says “stream.”

Datagrams avoid ordered-stream delivery but do not make overload disappear. UDP itself does not provide the application with reliable delivery or general congestion control. A datagram application must decide which messages may be lost, how duplicates are detected, whether ordering matters, how send rate responds to congestion, and how payload size stays within the path’s usable limit.

Multiplexing moves head-of-line boundaries. HTTP/2 can multiplex logical streams while its TCP connection still delivers one ordered byte stream; packet loss can delay bytes for all streams behind the missing range. HTTP/3 maps streams onto QUIC, avoiding cross-stream transport HOL, while individual streams retain order and the connection still shares congestion and resource limits. At the application layer, one global executor, lock, output buffer, or priority scheduler can recreate HOL above either protocol.

MTU and request shape determine packet work

The maximum transmission unit (MTU) limits the network-layer packet size on a link. The end-to-end usable packet size is constrained by the path, tunnel and encryption overhead, and protocol headers—not by a universal 1,500-byte assumption. A 256 KiB payload with a 1,500-byte path MTU and a simplified 40-byte IPv4/TCP header model requires at least 180 packets. Options, TLS records, encapsulation, acknowledgments, and response framing add work.

IP fragmentation is a recovery mechanism with operational costs, not a sizing strategy. Losing one fragment can make the whole original datagram unusable. Firewalls and tunnels may treat fragments differently. IPv6 routers do not fragment packets in transit; the source responds to Packet Too Big information or uses a suitable discovery method. Packetization-layer path MTU discovery can probe around paths that mishandle control messages, but applications still need conservative datagram sizing and error handling.

Segmentation offload allows the host stack to hand a large logical segment to a NIC, which splits it for the wire. Receive offloads may combine packets before upper-stack processing. They reduce per-packet CPU work, but they also make host captures and counters boundary-dependent. A capture showing 64 KiB “packets” above an offload boundary does not prove those frames were transmitted on the wire. Record interface, direction, capture hook, offload settings, snap length, dropped-capture count, and timestamp source.

Request sizing is an end-to-end decision. Larger messages amortize per-message framing and syscall work; they also increase serialization time, buffering, retransmitted bytes, and HOL exposure. Batching 100 unrelated requests into one frame may increase throughput and worsen the earliest deadline. Prefer bounds: maximum frame, maximum buffered bytes, maximum batch age, per-stream credit, and a cancellation rule that releases downstream work.

Packet processing is a scheduled resource path

On a conventional Linux receive path, a NIC places descriptors in receive queues and signals work. NAPI combines interrupt-driven scheduling with bounded polling so the kernel can process bursts without taking an interrupt for every packet. Receive-side scaling (RSS) can distribute hardware queues across CPUs; receive packet steering (RPS) and receive flow steering (RFS) add software or application-locality choices. The transmit side has socket buffers, queueing disciplines, device queues, steering, and completion work.

Every technique trades something:

• interrupt moderation reduces interrupt rate but can add waiting for sparse latency-sensitive traffic;
• polling can lower wake-up delay under sustained traffic while consuming CPU when idle;
• more queues expose parallelism but can worsen cache/NUMA locality and per-flow imbalance;
• offloads reduce per-packet CPU demand while making burst shape and observability less literal;
• busy polling or user-space networking can avoid some scheduling and kernel work while requiring dedicated cores, explicit memory ownership, driver/device support, and a complete isolation and failure model.

Kernel bypass is justified when packet-processing service demand is measured on the critical path and architectural terms are already controlled. It cannot shorten geographic propagation, remove an application round trip, repair an oversized response, or make an overloaded peer consume faster. Before changing the path, align per-CPU softirq or polling time, NIC queue drops, qdisc backlog, socket memory, packet rate, bytes, flow distribution, application CPU, NUMA placement, and end-to-end useful completions.

One hot flow may remain bound to one receive queue even when aggregate host CPU looks idle. Conversely, spreading packets from the same flow without maintaining order can induce reordering and extra transport work. Optimize for the flow distribution and application ownership, not host-average utilization.

RPC flow control must reach admission

Flow control protects a receiver from a sender that produces bytes faster than they are consumed. Backpressure is the larger application policy that makes upstream work slow, wait within a bound, shed, or fail before buffers grow without limit. A transport window alone does not stop an application from filling an unbounded pending-call queue.

For every RPC path, follow credit backward:

1. the consumer releases message or byte credit only after it has made bounded progress;
2. the transport stops the sender at per-stream and connection limits;
3. the RPC runtime exposes blocked sends and buffered bytes;
4. the producer stops reading or generating upstream work;
5. admission rejects work whose deadline or memory bound cannot be met;
6. cancellation propagates and releases pool, buffer, and executor capacity.

Streaming is effective when these stages are incremental. If a proxy buffers the complete request for inspection, a codec requires the whole object, or a client library prefetches without a bound, the nominal stream becomes a batch at that layer. Inspect first-byte latency, bytes buffered per hop, window-stall duration, consumer rate, cancelled bytes, and time from cancellation to resource release.

Retries multiply the network path and belong to Chapter 25’s full treatment. Here the rule is enough: never interpret wire throughput as useful goodput without separating original, retransmitted, duplicated, failed, cancelled, and late bytes. A 20% traffic increase during a tail event may be retry work caused by the event, not original demand.

Infrastructure hops are part of the critical path

Name resolution can include local caches, stub resolvers, recursive resolvers, multiple address families, retries, and policy. Report cache-hit and miss populations, answer age, resolver latency, failures, address selection, and connection outcome. A low median DNS time can coexist with a cold-start tail dominated by misses or failed first addresses.

Proxies, service meshes, gateways, and load balancers may terminate security, parse protocols, enforce policy, select destinations, retry, buffer, transform, rate-limit, and emit telemetry. Each function consumes CPU and memory and can add a queue or connection pool. Their value may be correctness or operability rather than speed; the performance requirement is to make the added work and failure boundary visible.

Cross-zone and cross-region calls add distance, failure domains, transfer work, and often cost. More replicas can reduce reader distance while increasing write, invalidation, repair, and consistency work. The first decision should be whether the synchronous dependency must be remote. Move computation to state, cache immutable or safely stale data, precompute, parallelize independent calls, or change the user-visible contract before tuning packet processing.

A multi-region budget must include normal, degraded, and recovery routes. When a region fails, traffic may travel farther, surviving proxies and pools receive a step increase, connections re-establish, caches cool, and replication or rebuild consumes bandwidth. A design that meets 95 ms only on the normal route has described a preference, not an objective.

Read a capture as a hypothesis test

Packet captures show events at one observation point. They do not directly show application intent, hidden queues, the remote capture, or why a scheduler delayed a process. Use them with traces and host/path counters.

The fixture preserves three contrasting cases in packet-cases.json: queue-loss, receiving-application stall, and isolated path loss/reordering. Run its dimensional and classification checks with:

node examples/performance-engineering-system-design-handbook/part-02/network-path/run.mjs
node examples/performance-engineering-system-design-handbook/part-02/network-path/verify.mjs

The packet guide should always record five facts before interpretation: clocks and capture points, five-tuple/connection identity, direction and sequence space, offload/capture-loss conditions, and the application operation population. Without them, a precise packet timestamp can still belong to the wrong request or boundary.

Reduce architectural terms before specializing the stack

Begin with the dependency graph. If protocol or state placement permits one remote round trip to disappear, the saving is bounded by distance rather than by a microbenchmark. The per-hop trace should show that the call was sequential and material to the objective; the design review must then account for any larger message, stale state, or changed correctness boundary. Keeping work region-local makes the same trade at a larger scale. It shortens both RTT and the synchronous failure path, while creating replication, invalidation, and placement work that must survive the failed-route budget.

Next remove repeated establishment. Reuse bounded connections when destination, credential, protocol, and policy identities are stable enough to share them. Cold-versus-warm timing should show the handshake saving, while checkout wait, connection age, and failure rate expose the queueing and synchronized reconnect that pooling can introduce. Multiplexing can remove still more connections for independent operations, but only if stream-stall, connection-credit, and loss-correlation evidence show that shared congestion and flow control do not create a worse common fate.

Change the service contract only when the application can use it. Datagrams fit information whose freshness is worth more than reliable ordered delivery; their case rests on useful outcomes under loss, bounded size, pacing, and receiver overload—not on avoiding TCP by name. They move duplication, replay, loss response, and congestion duties into the application.

Only then descend into packet processing. Tune kernel queues, steering, and offloads when per-queue drops, per-CPU work, packet rate, and useful completions place host networking on the critical path. Consider user-space networking after a controlled end-to-end test shows that bypass changes the objective and the design can afford dedicated cores, explicit memory ownership, isolation, driver lifecycle, security, and portability. A faster packet path is valuable only after distance, round trips, bytes, and avoidable queues have stopped dominating it.

Field diagnostic: retransmissions beside a 174 ms p99

Use the fixture’s 36 ms baseline RTT, 78.7 ms loaded RTT, 0.8% retransmitted bytes, 37 bottleneck drops, 11.8→12.1 ms server processing, zero receive-window closures, and 174 ms end-to-end p99. Rank queueing, path loss/reordering, and application stall. State what is observed, what is inferred, and one controlled test that could falsify the leading hypothesis. Then repeat the ranking after replacing the bottleneck drops with twelve zero-window events and returning loaded RTT to 37.4 ms.

Principal design drill: fit a remote dependency into 95 ms

Mercury must return a 256 KiB response under a 95 ms p99 objective. The remote dependency has a 36 ms baseline RTT, may fail over to a 58 ms RTT path, and must preserve read-your-write semantics for authenticated updates. Design normal, cold, overloaded, failed, and recovering paths. Specify the latency ledger, connection and stream bounds, framing and maximum sizes, flow-control/backpressure chain, MTU discovery behavior, retry/idempotency rule, capture plan, and admission response. After the first design, assume the proxy buffers full responses for inspection and the peer allows only 128 concurrent streams per connection. Revise the architecture and the claim it can support.

Field checklist

• Is the latency population defined by operation, outcome, path, state, and interval?
• Have one-way distance, bytes, sequential round trips, queues, and endpoint work been separated?
• Is the BDP compared with actual congestion, receive, socket, and application windows?
• Are cold, warm, resumed, failed, and draining connections measured separately?
• Do transport and RPC semantics match ordering, loss, cancellation, and replay requirements?
• Are message, buffer, stream, pool, and queue sizes bounded with deadline-aware admission?
• Is path MTU handled without assuming a universal frame size or relying on fragmentation?
• Are capture point, clocks, offloads, and capture drops recorded?
• Are retries and retransmissions separated from correct useful goodput?
• Does the failure route retain enough latency and capacity budget for reconnect and recovery?
• Has the team removed avoidable distance, round trips, bytes, and queueing before lower-level tuning?

Distance cannot be optimized away after it enters a synchronous dependency. Protocols decide how many times that distance is paid, how much state remains in flight, and which loss or queue blocks useful work. Make those terms explicit first. Even with topology and protocol unchanged, the next uncertainty is whether a host, resource controller, placement, or managed-service boundary withholds progress along the packet-processing path.

Sources and evidence scope

• RFC 9293, Transmission Control Protocol defines TCP’s reliable ordered byte-stream service and current protocol requirements. Congestion-control and implementation choices require their own specifications and environment evidence.
• RFC 5681, TCP Congestion Control specifies classic slow-start, congestion-avoidance, fast-retransmit, and fast-recovery behavior. A deployed stack may use later algorithms and extensions.
• RFC 6349, Framework for TCP Throughput Testing connects bottleneck bandwidth, RTT, BDP, receive window, path MTU, throughput, retransmissions, and buffer delay for managed-path testing. Its methodology is not a universal Internet guarantee.
• RFC 8446, TLS 1.3 defines the full and resumed handshake protocols and early-data security constraints. Application timing depends on transport, implementation, and reuse state.
• RFC 9000, QUIC defines QUIC connections, ordered streams, connection and stream flow control, establishment, and migration. RFC 9002 specifies QUIC loss detection and an example congestion controller.
• RFC 9114, HTTP/3 explains HTTP semantics over QUIC streams and the head-of-line boundary relative to HTTP/2 over TCP.
• RFC 8200, IPv6 defines IPv6 packet sizing and source-only fragmentation; RFC 8899 defines datagram Packetization Layer Path MTU Discovery.
• Linux kernel NAPI documentation and network-stack scaling documentation describe current Linux interrupt/poll scheduling, RSS, RPS, RFS, and XPS concepts. Exact device, driver, kernel, NUMA, and configuration behavior must be measured.
• All Mercury latency, BDP, packet-count, connection, retransmission, and capture-case values are simulated teaching evidence in examples/performance-engineering-system-design-handbook/part-02/network-path/; they are not measurements of this host, a network provider, or production traffic.