Distributed Failure Debugging

One Checkout, Four Plausible Stories

At 10:12 UTC, support reports that several customers cannot see orders they just placed. The search team sees indexing lag. The database team sees a successful failover. The checkout team sees timeouts followed by retries. Each account is true, and none explains the incident.

The first affected checkout began fourteen minutes earlier. A storage node developed intermittent slow I/O. Primary commit latency rose from 40 milliseconds to 900. The API gave up after 700 milliseconds, but some transactions committed after their callers had stopped waiting. Mobile clients retried without stable idempotency keys. The outbox consumer, reading from the same pressured database, fell behind. Follow-up status reads went to a replica that was now 90 seconds behind. A failover improved write latency while changing which read routes were fresh.

No component needed to fail completely. The incident lived in the intervals between local truths: a client timeout and a later commit; a primary commit and replica visibility; an order row and its outbox delivery; one request and the retry that looked like a new purchase.

Distributed failure debugging begins by refusing to choose a culprit too early. Reconstruct what crossed each boundary and when. Then form an explanation precise enough to predict evidence.

Build the Timeline Before the Story

Start with one affected operation: a request, tenant, key, partition, or trace. An average can show when the system bent, but it cannot tell you which order committed after its timeout or which replica answered the next read.

For the checkout, the initial timeline might contain these observations:

• 09:58:12 UTC — storage latency begins rising on the primary; the metric identifies the node, not an order.
• 10:01:04 — order A731 enters the API with request ID r-18; trace sampling retains it.
• 10:01:04.700 — the client timeout expires. The client knows only that it received no answer.
• 10:01:04.912 — the primary commits order A731; the database transaction record supplies the time.
• 10:01:05.140 — the client submits attempt two under request ID r-21; no stable operation key links it to r-18.
• 10:01:06.003 — a second order row commits.
• 10:01:06.180 — the status read reaches replica replica-3, whose applied position precedes both commits.
• 10:01:08 — the outbox checkpoint stops advancing while queue depth grows.

This sequence turns “checkout is flaky” into a tractable contradiction. The first request timed out and committed. The retry became independent work. The read path then observed a version of the world older than either write.

Use absolute time and record the timezone. Preserve event time, commit time, replication time, processing time, observation time, and report time as different fields. A record created at 10:00, committed at 10:00:02, replicated at 10:01, indexed at 10:04, and reported at 10:20 did not take twenty minutes to commit.

Do not force clocks into agreement on paper. Record which clock produced each timestamp and check synchronization health before using small differences to infer causality. When trustworthy time is unavailable, causal markers can still order events: transaction ID, log position, broker offset, generation number, deployment version, or a trace edge.

Uncertainty is part of the evidence. A timeout means the outcome is unknown unless the protocol proves otherwise. A consumer crash leaves a last durable checkpoint that may not match its last external side effect. Mark those gaps explicitly; a neat timeline that converts unknowns into failures will direct the repair toward the wrong facts.

Follow the Fact Across Boundaries

Once events are ordered, trace one fact from the user’s action to every place that can represent it. For this checkout, the fact crosses the API, database transaction, outbox, broker, consumer, search index, replica, and status page. Each crossing weakens or changes some guarantee.

The transaction boundary answers what committed together. The order row and its outbox row may be atomic; the search update and payment provider are not. If the system uses a dual write instead of an outbox, a successful row says nothing about whether a message exists.

The message boundary answers what delivery means. A broker acknowledgment, a consumer checkpoint, and completion of an external side effect are separate events. At-least-once delivery permits duplicates. An exactly-once claim may cover broker state while leaving emails, payments, or shipments outside its protection.

The replica or region boundary answers what a successful write makes visible elsewhere. A primary can acknowledge a durable commit while another region or replica remains behind. A failover may restore writes and simultaneously alter routing, leadership epoch, cache warmth, and read freshness.

The ownership boundary answers who can see and change the path. During the checkout incident, the database team can confirm commit positions, the application team can disable unsafe retries, and the search team can measure consumer lag. If no one owns the end-to-end promise, every team can report a healthy component while the user remains stuck between them.

The consistency boundary answers what the reader was entitled to observe. Analytics may tolerate a minute of lag. An order confirmation immediately after checkout usually cannot. “Eventually consistent” does not settle the question; it needs a freshness bound, a route policy, an owner, and a repair path.

When an investigation stalls, annotate each boundary with three things: the promise made there, the evidence that would prove it, and the outcome currently unknown. The likely failure often appears as the first place where a user-facing promise is stronger than the mechanism carrying it.

A Timeout Creates a Branch, Not an Answer

The checkout timeout did not cancel the database transaction. It split the investigation into three possible histories: the write failed, the write committed before the timeout but its response was lost, or the write committed after the caller gave up. Retrying is safe only if those histories converge on one logical result.

Inspect retry behavior as load generation and as a correctness policy. Count the original attempt, maximum attempts, wall-clock budget, backoff, jitter, caller fan-out, nested downstream retries, and side effects per attempt. Five downstream calls with three attempts each can turn one user action into far more than three operations, especially when dependencies retry one another.

Stable idempotency identity is what lets repeated transport attempts represent one business operation. It must survive process restarts and reach the enforcement point: a uniqueness constraint, version check, deduplication record, provider key, or idempotent state transition. A fresh request ID on every attempt aids tracing but does not prevent duplicate orders.

Backoff without jitter can synchronize callers into recovery waves. A circuit breaker that opens after pools are exhausted protects little; one that opens across unrelated workflows converts a partial failure into a broad outage. Retry budgets, concurrency limits, admission control, and breakers should preserve the most important work while the dependency recovers.

In the running incident, the useful immediate mitigations follow from this analysis: stop retries that lack operation identity, reduce pressure on the primary, route post-write reads to a path that can meet the confirmation promise, and keep the outbox from competing without limit for the same constrained resource. None requires pretending that “the database failed” explains everything.

Recognize the Amplifiers

Failure labels are useful only when they sharpen the search for evidence.

A partial network failure produces mixed outcomes: success, delay, timeout, and completion after timeout. Look for route, packet, and attempt differences rather than a clean outage interval.

A slow dependency consumes queues, threads, connections, and deadlines before it emits obvious errors. The loudest alert often fires in a caller whose pool is exhausted, not at the initiating bottleneck.

Node loss changes more than capacity. Leadership, partition ownership, client routes, connection pools, cache warmth, and replica freshness can all change. Put recovery and failover events on the timeline beside the original fault.

Backpressure appears as lag, memory growth, checkpoint delay, stale derived data, and increasingly old work. Lag is a correctness risk when the derived view participates in a user promise.

Clock skew undermines leases, expiry, last-write-wins conflict resolution, event-time windows, and any incident story inferred solely from timestamps. Leadership epochs, fencing tokens, and sequence numbers often provide stronger evidence than wall time.

Split brain is an authority failure. Ask which quorum, lease, generation, epoch, or fence should have prevented two actors from accepting writes or owning the same partition. “The wrong node acted” is less useful than locating the absent or ignored authority check.

These shapes compose. Slow I/O produces timeouts; retries increase pressure; consumer lag grows; stale reads spread; failover changes routes. A strong explanation names the trigger, the amplifiers, and the assumption that allowed user harm.

Ask the Hypothesis to Predict Evidence

Suppose the team proposes: slow primary commits created ambiguous checkout outcomes; retries without stable idempotency created duplicate orders; shared database pressure delayed outbox processing; lagging replicas served the follow-up reads.

That explanation predicts specific observations:

• affected first attempts should time out before their transaction commits;
• duplicate rows should correspond to multiple attempts for one logical purchase without a shared operation key;
• outbox rows should exist before their delayed broker offsets and consumer checkpoints;
• stale reads should concentrate on replicas whose applied positions trail the relevant primary commits;
• recovery should follow reduced primary pressure, safer retry behavior, replica catch-up, and consumer progress.

Now look for disproof. If duplicate orders share an enforced idempotency key, the retry explanation is incomplete. If stale reads came from the primary, replica lag cannot explain them. If outbox checkpoints advanced normally, the missing status may belong to a later sink. A hypothesis that cannot lose is a label, not an investigation.

Reproduction establishes plausibility, not history. In a staging or shadow environment, introduce the production-shaped delay, concurrency, routing, and retry policy. Delay the primary past the client timeout, allow the write to finish, repeat the request, and route the read to a lagging replica. The test should observe both business facts and system state: number of logical orders, attempt count, commit positions, applied positions, outbox IDs, offsets, checkpoints, and user-visible status.

Fault injection needs an expected outcome and stop conditions before the fault begins. Pausing a consumer, delaying a dependency, dropping a network path, forcing leadership change, or exhausting connections is useful only when the team knows whether the system should reject work, buffer it, degrade a feature, serve bounded stale data, or fail over. Recovery evidence belongs in the same drill.

For concurrency and consistency bugs, record operation histories and check business invariants. Jepsen-inspired thinking does not require a full formal harness: concurrent inputs, observed outputs, ordering metadata, and a model of allowed outcomes can expose histories the architecture claims are impossible.

The production timeline must still match the reproduced mechanism at the affected boundary and time. A convincing laboratory failure is not proof that production failed the same way.

Repair the Facts, Then Preserve the Lesson

Restoring latency does not repair duplicate orders, missing index entries, repeated notifications, or contradictory customer records. Containment, service recovery, and data repair are related but distinct tracks.

Define the affected set before changing data. Preserve request IDs, operation identities, transaction records, outbox rows, offsets, replica positions, and external-provider references. Choose an authority for each fact. Compensate irreversible side effects rather than deleting the evidence that they occurred. Reconcile source records against replicas and derived stores after replay or rebuild.

The incident should leave executable memory. The checkout failure can become a test that delays commit beyond the caller timeout, retries under the same operation identity, verifies one logical order, reads through every supported route, pauses and resumes the outbox consumer, and proves derived state converges. Monitoring should expose ambiguous outcomes, retry amplification, applied-position gaps, and the age of the oldest unprocessed work. The runbook should name who may disable retries, change routes, fence a writer, pause replay, and approve compensation.

This is the bridge to the next chapter. An explanation becomes durable confidence only when its important claim can be tested under the condition that broke it.

Distributed Failure Timeline

Keep one working record during the incident. Add rows as evidence arrives; do not rewrite uncertainty into certainty for appearance.

A usable timeline explains the order of symptoms, identifies ambiguous outcomes, and points to the next observation. It also records recovery and repair, not only onset.

Investigation Drills

1. An API returns success, but the next page refresh shows old state. Trace the primary commit, replica replay, route decision, cache lookup, and UI response. State the promised read behavior and identify the first boundary unable to support it.
2. A dependency’s p95 rises from 80 to 600 milliseconds for six minutes, then its callers saturate. Draw the fan-out and calculate the maximum attempts generated by one user action, including nested retries. Choose a mitigation that protects the dependency while preserving the highest-value workflow.
3. A consumer is restarted from an older offset to repair a derived store, and users receive duplicate notifications. Identify the checkpoint, external side-effect boundary, operation identity, compensation path, and verification that should precede another replay.

The investigation is complete when the explanation predicts the evidence, accounts for user impact, directs safe mitigation, and tells the repair team which facts to trust. Local success is partial evidence. A timeout is an ambiguous outcome. A retry is new work. The boundary between those truths is where the incident becomes understandable.