Networking and the Web

The checkout that timed out—and charged twice

A customer taps Pay on a mobile connection. After eight hundred milliseconds the client gives up. It shows an error and sends the checkout again. Two charges appear.

“The network timed out” describes an observation, not a cause. The first request may have failed before it reached the service. It may have been rejected by a gateway. The payment provider may have committed the charge after the application stopped waiting. The response may simply have been lost on its way back. From the client’s timeout alone, these histories are indistinguishable.

This ambiguity is the useful center of networking. A request crosses several boundaries, and each boundary makes a limited promise:

client -> DNS -> connection and TLS -> HTTP -> edge -> application
       -> payment provider -> response -> client decision

At every arrow, ask four questions:

• What has been established so far?
• How long may this stage consume?
• What evidence survives if it fails?
• Is it safe for the caller to try again?

Those questions turn a protocol inventory into a way to design and debug real systems.

Before bytes move: name, address, route

The checkout begins with a name such as api.example.com. DNS may answer from the operating system, a local network cache, a recursive resolver, or an authoritative server. The answer can contain several addresses or vary by region. Its time to live permits caches to reuse it for a period.

That indirection enables traffic movement, but it does not make movement instant. A low TTL can help clients discover a new destination sooner, at the cost of more lookups. It cannot recall answers already cached, and some software keeps connections longer than it keeps DNS answers. A high TTL reduces lookup work but can keep clients aimed at an old destination during a failover. Failed lookups may also be negatively cached.

DNS success proves only that the client obtained an address. It says nothing about whether a route exists, a socket can be opened, TLS will accept the hostname, or the service is healthy. Conversely, the absence of an application log does not prove that the client never sent anything. The failure may lie before the application: resolution, routing, handshake, edge policy, or load-balancer health.

For the checkout, record the hostname, returned addresses, lookup duration, TTL, and region. These facts are far more useful than “DNS looks fine.”

A connection carries more history than one request

With HTTPS over TCP, a cold request normally pays for a TCP handshake and a TLS handshake before HTTP data moves. TCP supplies an ordered, reliable byte stream and congestion control. If packets are lost, TCP recovers them before presenting later bytes in order. It does not preserve HTTP message boundaries; the application protocol does that.

UDP supplies datagrams without TCP’s built-in delivery, ordering, or stream semantics. This makes it a useful substrate when a protocol wants to define those properties differently. QUIC, which carries HTTP/3, runs over UDP but adds encrypted connections, reliable streams, congestion control, and loss recovery. “UDP is faster” is therefore the wrong explanation. The meaningful question is which layer owns streams, ordering, recovery, and connection state.

Connection reuse removes much of the cold-start cost. Browsers and service clients keep pools so later requests can use an established connection. A pool, however, is both an optimization and a finite queue. When all connections are busy, new work waits. When old connections have been closed by a peer or intermediary, the next borrower may discover the failure. Long- lived connections can also remain attached to a retiring upstream after DNS or load-balancer membership changes.

The HTTP versions alter how work shares connections:

• HTTP/1.1 commonly uses several reusable connections because requests on one connection otherwise have limited concurrency.
• HTTP/2 multiplexes many HTTP streams over one TCP connection and compresses headers. A lost TCP packet can still delay progress across those streams because TCP delivers one ordered byte stream.
• HTTP/3 maps HTTP onto QUIC streams. Loss on one stream need not delay data already available on another, and connection migration can preserve a session when a mobile client’s network path changes. The operational cost is another transport to observe, tune, and support through the network path.

None is universally fastest. A small number of ordinary API calls may gain little from a protocol change. A page with many resources, users on lossy links, or clients that change networks may gain more. Measure connection setup, reuse, pool wait, bytes transferred, and tail latency before assigning credit to the protocol label.

TLS establishes a peer—and a trust boundary

TLS protects confidentiality and integrity and lets the client authenticate the server named in the certificate. Certificate expiry, an incomplete trust chain, or a hostname mismatch can stop the request before HTTP exists.

In production, TLS often terminates at a CDN, load balancer, ingress proxy, or service-mesh sidecar rather than in the application process. The next hop may use another protected connection or may travel in plaintext inside a trusted network. The architecture must say where the trust boundary is and which identity crosses it.

Forwarded headers illustrate the danger. A proxy can set X-Forwarded-For or X-Forwarded-Proto to preserve facts about the original connection. The application should trust those values only when they came through known infrastructure that removes or sanitizes client-supplied versions. Otherwise an arbitrary client can claim a different address or scheme.

Our checkout trace should therefore name the TLS termination point, the authenticated hostname, the protocol on the next hop, and the component that constructs trusted forwarding metadata.

HTTP gives the request semantics

Once the connection exists, HTTP describes an application action with a method, target, headers, body, and response status. These are not decorative labels. They tell clients, caches, proxies, and operators how the message may be handled.

GET retrieves a representation and is expected to be safe: repeating it should not request a state change. PUT places a representation at a known resource identifier and is defined to be idempotent: repeating the same request has the same intended effect. POST submits data to be processed, often as a command or subordinate creation. It is not inherently safe to repeat. PATCH may or may not be idempotent depending on what the patch means.

Idempotent does not mean “the responses are byte-for-byte identical” or “no logging occurs.” It means repeated identical requests have the same intended effect on server state as one request. A payment command sent by POST needs an application mechanism if repetition must be safe.

Status codes make the outcome legible. A 202 Accepted response says that work has been accepted but is not complete and should provide a way to observe it. A 409 Conflict can distinguish a state conflict from a server fault. A 429 Too Many Requests can communicate admission control, often with retry guidance. A 5xx response says the server could not fulfill an otherwise request, but it does not by itself prove that no side effect occurred. That last point matters most when a proxy or application fails after a dependency has committed.

Intermediaries are participants

The edge between client and application may contain a CDN, web application firewall, reverse proxy, API gateway, load balancer, ingress controller, or service mesh. Each can terminate TLS, reject a body, enforce authentication or rate limits, buffer a stream, cache a response, rewrite headers, choose an upstream, or retry a failure.

A load balancer does not simply “spread traffic.” Its choice may occur per connection or per request, depending on the protocol and configuration. Health checks decide which targets are eligible, but a target can pass a shallow check while its critical dependency is unusable. Session affinity can preserve useful local state while making distribution uneven. A least- connections policy may behave differently from round robin when requests have very different durations.

Proxies also impose limits that become part of the effective API. A large upload may be rejected before application logs appear. Response buffering may defeat streaming. An edge timeout may close the client connection while the application continues working. An automatic gateway retry can turn one checkout into two payment attempts.

For any mutation, make retry ownership explicit. If the mobile client retries, the gateway should not secretly add another independent retry budget unless the end-to-end command contract tolerates the multiplication.

One deadline, divided along the path

Suppose the checkout has an 800 ms client deadline. The gateway allows 1,000 ms, the application waits 1,500 ms for its handler, and the payment client waits 3,000 ms. Every inner component is willing to work after its caller has left. A user retry can overlap the original attempt, consuming more capacity and increasing the chance of duplicate work.

A coherent path starts with the caller’s remaining budget and spends it inward. It reserves time for the response to travel back and for cleanup. The exact allocation depends on measured behavior, but its ordering should look like this:

client deadline          800 ms
edge deadline            720 ms
application deadline     650 ms
payment call deadline    450 ms of the time still remaining

A deadline is stronger than copying the same timeout to every hop because it shrinks as time is spent. Separate connect and response limits may also be useful: waiting 400 ms to establish a connection is a different failure from receiving the first response byte and then stalling.

When a deadline expires, ask whether cancellation actually reaches the callee. Closing a client socket does not necessarily roll back application or payment work. Work that cannot be canceled must be safe to finish once and must leave a result the caller can recover later.

Latency is the sum of waits and work across the whole path:

DNS + handshake + upload + edge queue + app queue + handler
    + dependency waits + download + client parse and render

Server compute is only one term. If the median is healthy while p95 rises, look for queueing, pool exhaustion, cold connections, packet loss, cache misses, retry amplification, or a small group of expensive requests. An average erases precisely the slow histories the trace must explain.

A safe retry needs a durable identity

A retry is justified only when the failure may be transient, another attempt fits inside the remaining budget, and repetition is safe. Backoff and jitter can keep background clients from returning in lockstep. Interactive requests usually have room for fewer attempts; a prompt degraded response can be better than invisible retry delay.

For the checkout, give the command an identity before sending it:

POST /checkouts
Idempotency-Key: 8f2c...client-generated...

server:
  insert or find command by idempotency key
  if complete, return the recorded result
  if in progress, return or wait according to the API contract
  otherwise execute payment with the same provider-side identity
  record the outcome

A uniqueness guard prevents two handlers from becoming first. The stored record lets a later request recover the original result. The provider-side identity extends the protection across the next boundary. Retention must be long enough to cover realistic retries, and the key should be scoped so one customer cannot collide with another accidentally or maliciously. Store a fingerprint of the original command as well; reuse of the same key with a different amount or cart should conflict rather than replay an unrelated result.

Now the lost response is survivable. The client still cannot infer success from silence, but it can ask about command 8f2c... or safely repeat it. The network remains capable of loss; the application contract removes the duplicate effect.

Retries also need a load budget. Two retries in a client, two in a gateway, and three in a dependency library can expand one action into many attempts during the very outage that reduced capacity. Prefer one owner, cap attempts, record attempt number, and stop when the deadline or system health says more work will do harm.

Caches can return a correct response from the wrong history

A response may be cached in a browser, service worker, CDN, reverse proxy, or application client. Correctness depends on three facts: the cache key includes everything that changes the representation, the freshness rule matches the product, and there is a way to expire or bypass an unsafe entry.

HTTP provides Cache-Control, validators such as ETag, and Vary to express some of this contract. no-cache permits storage but requires validation before reuse; no-store asks caches not to store the response. A CDN key that omits tenant, authorization state, language, or a relevant query parameter can serve a valid response to the wrong audience. A stale-if-error policy can keep public documentation available during an origin failure but would be a dangerous default for a just-completed checkout.

Caches also explain deploys that fail only for some users. Old HTML may name an asset that has been removed. A service worker may retain an obsolete API route. Long-lived connections may still reach old application instances even after DNS has changed. “Clear your cache” is not a diagnosis; identify the specific cache, key, stored version, freshness instruction, and invalidation path.

The browser has a policy engine of its own

A backend test can succeed while the browser rejects the same interaction. The browser enforces origin, cookie, mixed-content, redirect, and content- security rules around the HTTP exchange.

For a cross-origin request, the browser may send an OPTIONS preflight to ask whether the target allows the method, headers, origin, and credential mode. CORS controls whether browser code may read the response; it is not a general server authentication mechanism. A wildcard allowed origin cannot be used with credentialed requests, and reflecting arbitrary origins can expose protected data.

Cookies add another boundary. Secure, HttpOnly, SameSite, domain, and path attributes affect whether the browser stores and sends them. Because a browser may attach credentials automatically, state-changing endpoints also need a deliberate defense against cross-site request forgery. An HTTPS page may refuse active content loaded over plain HTTP before application code gets a useful response.

When a failure is browser-only, collect the exact origin, preflight and redirect chain, request and response headers, cookie attributes, console message, network waterfall, and service-worker state. Reproducing the URL with curl answers a different question.

Reconstruct the request from evidence

Return to the duplicate checkout. A useful investigation does not begin with a list of every networking component. It establishes the last boundary with evidence and follows the command identity:

1. From the client, obtain the command ID, timestamps, network type, observed timeout, and whether it retried.
2. At DNS and the edge, confirm the address, TLS result, request ID, routing decision, policy outcome, queue time, and any retry.
3. In the application, separate queue delay from handler time and record the deadline that arrived, not only the configured default.
4. At the payment boundary, find each attempt made with the command identity and its provider result.
5. In the response path, determine whether the application produced a result, whether the edge received it, and whether the client connection was still open.

This sequence separates several superficially similar failures. No edge log suggests a problem before or at the edge. An edge log with no application log points toward routing, health, limits, or proxy policy. A client-closed signal with later application success shows that work outlived the caller. Two provider attempts with one command ID reveal broken idempotency or retry ownership. One provider attempt and two client errors point elsewhere.

Correlation IDs help join these records, but they are not enough on their own. A request ID identifies an attempt. An idempotency key identifies the logical command across attempts. Keep both.

Trace a harder variation

Suppose the payment provider now sometimes takes 900 ms, the checkout deadline remains 800 ms, and product wants the screen to respond promptly rather than wait longer.

Design the next version before choosing a protocol or queue product. Decide:

• whether checkout can become a durable asynchronous command;
• what POST /checkouts returns while work is pending;
• how the client observes completion without creating duplicate commands;
• which component owns retries and how long it may keep trying;
• what happens when payment succeeds after the user leaves;
• which status is safe to cache, and for whom;
• which attempt ID and command ID appear at every boundary.

One credible shape returns 202 Accepted with a command location, processes the payment under a durable owner, and lets the client poll or receive a carefully designed update. That design adds storage, queueing, expiry, and recovery obligations. It is better only if those costs serve the product’s actual promise.

The durable networking habit is to follow the history rather than name the stack. Resolve the name, establish the peer, understand the HTTP action, expose what intermediaries do, spend one deadline inward, and preserve enough identity to survive an uncertain outcome. The next chapter begins where this one leaves off: with the durable record that must make “execute this command once” true after connections, processes, and machines fail.