Reliability and Failure Engineering

The payment provider has not failed

Return to the ticket checkout from the previous chapter. An order is in pending_payment, and the payment provider normally answers in 250 milliseconds. This afternoon its latency rises to eight seconds. It still returns success to some requests. Nothing is cleanly down.

Meanwhile, buyers see spinners and press Pay again. Mobile clients retry. The API gateway retries. The checkout service retries. Each attempt occupies a worker and a connection while inventory holds expire behind it. Soon tax calls and ordinary order-status reads are waiting for checkout capacity too.

This is the reliability problem in its instructive form. A slow dependency can consume the healthy system around it, and the mechanisms intended to help can multiply the damage. The design has to answer four questions in order:

1. What promise must remain trustworthy?
2. How long and how much capacity may one attempt consume?
3. What work will the system refuse or degrade to protect that promise?
4. What evidence will show that service and data are safe again?

The checkout promise is not “every component is available.” It is narrower and more useful: a buyer receives one durable order identity, is not double-charged, can learn whether payment is pending or complete, and is never shown a successful purchase that the system cannot later honor.

Spend one deadline, not several unrelated timeouts

Begin with the useful lifetime of the user request. Suppose the edge gives this checkout 1.2 seconds. That is a design value for the modeled service, not a universal constant. Each hop receives a portion of the remaining time:

edge deadline                 1,200 ms
  request admission and auth   100 ms
  order and hold decision       250 ms
  payment authorization         600 ms
  response and safety margin    250 ms

The downstream timeout must expire before its caller’s deadline. Otherwise the caller gives up while work continues invisibly below it. The budget also needs propagation: a service receiving only 90 milliseconds of remaining time should not begin a call whose normal completion takes 300.

A timeout bounds waiting; it does not identify the outcome. If payment times out after 600 milliseconds, the provider may still have accepted the charge. The order therefore moves to the durable payment_unknown state designed in Chapter 81. The buyer gets the same order ID and a stable status page. A fresh checkout would turn uncertainty into a duplicate-charge risk.

This makes the timeout useful at two levels. Operationally, it releases scarce capacity. Semantically, it sends the workflow to an honest state from which a callback or reconciliation can finish the decision.

Retries spend a shared budget

A retry is another request against the component already showing distress. It is justified only when the operation is safe to repeat, enough deadline remains, and the extra attempt has a credible chance of succeeding.

Let the checkout service own payment retries. The client may repeat the high-level checkout under the same idempotency key, but the gateway and SDK do not independently retry the provider call. Otherwise three layers making three attempts each can turn one user action into as many as 27 downstream attempts.

The policy might allow one quick retry for a connection failure before any response bytes arrive, but no retry after the checkout deadline is nearly spent. Attempts reuse the same provider idempotency key. Randomized backoff prevents a recovering provider from receiving a synchronized wave, and a service-wide retry budget caps retries as a fraction of normal traffic. When that budget is gone, new work fails or becomes pending instead of borrowing capacity from healthy requests.

The interviewer should be able to hear the stop condition:

Retry only the payment client, under the original idempotency key, while the
request deadline and the shared retry budget both have room. Otherwise preserve
payment_unknown and let reconciliation resolve it outside the request path.

Backoff without a budget merely spaces out amplification.

Stop calling before the checkout fleet is consumed

As timeout rate rises, a circuit breaker opens for this provider endpoint in this region. New calls receive the degraded behavior immediately rather than waiting six hundred milliseconds to discover the same failure. The breaker protects checkout workers and connection pools; it does not make payment correct.

Its scope matters. A single global breaker could disable healthy provider regions because one route is bad. A breaker per process can let thousands of processes keep probing. A practical scope follows the failure boundary—perhaps provider, operation, and region—with shared health evidence and a bounded number of probes.

After a cool-down, the half-open breaker admits a small probe flow. Success rate, latency, and provider semantics decide whether traffic grows gradually or the breaker opens again. An operator needs to see breaker state, rejected calls, probe results, and the count and age of orders awaiting reconciliation.

Payment capacity also gets a bulkhead. Its calls use a pool that tax, inventory, status reads, and receipt delivery cannot consume. Separate queues and worker pools keep payment reconciliation from starving interactive status reads; tenant quotas keep one flash sale from occupying the entire pool. These boundaries reserve some useful service when the dependency is slow.

Bulkheads are not free capacity. A pool can be too small during normal peaks or leave resources idle while another pool queues. Size and admission policy come from the promise: interactive order status may deserve reserved capacity while analytics is shed, and scarce-inventory checkout may deserve a stricter queue limit than browsing.

Degradation is a product state

With the breaker open, the system cannot honestly call checkout “available” in the ordinary sense. It can still decide precisely what remains available:

• existing orders and payment status remain readable;
• carts and inventory browsing continue from their normal read path;
• a payment attempt already accepted locally remains visibly pending;
• email and analytics stay off the critical path and drain later;
• new scarce-item orders stop before promising inventory if their holds cannot survive the delay;
• fulfillment never treats payment_unknown as paid.

The degraded response tells the truth: pending, read-only, delayed, partial, or unavailable. It also leaves a repair path. An order accepted as pending has an expiry policy, a reconciliation job, and a final transition to paid, declined, refunded, or manual review.

Some work may fail open; recommendations can disappear and analytics events can be sampled or dropped. Authorization, payment correctness, inventory ownership, and legally required tax decisions cannot be waved through because a fallback sounds more available. The next chapter takes up those trust boundaries in detail.

Redundancy only helps when failures are independent

The provider recovers, but now suppose the checkout region disappears. The service has instances in a second region and the order database has a replica there. That inventory is evidence of redundancy, not yet a recovery plan.

Ask what the second region still shares with the first:

• Does it depend on one global identity or configuration control plane?
• Can it reach the same payment provider through an independent route and account boundary?
• Does the inventory owner live only in the failed region?
• How far behind is the order replica, and which acknowledged writes may be absent?
• Who fences the former primary if its network returns?
• Can DNS or the traffic manager change while the usual control plane is down?

For ticket inventory, automatic promotion of a lagging replica can violate the no-oversell promise. The system may keep account pages and completed-order reads available in the surviving region while pausing new reservations until it has one fenced inventory owner. That is a deliberate reduction in availability to preserve correctness.

Active-active service instances do not remove this decision. They move it into data ownership, conflict handling, quota placement, and dependency independence. An active-passive design may recover more slowly but be easier to reason about. Choose between them from the promised recovery behavior, not from the prestige of the topology.

Recovery has a time, a point, and a proof

Two objectives make the regional promise testable:

• RTO, the recovery time objective, bounds how long the service may be down or materially impaired.
• RPO, the recovery point objective, bounds how much committed history may be lost when recovery uses a replica or backup.

An RPO of zero for acknowledged orders requires a replication and acknowledgment path capable of defending it. A daily backup cannot supply it. An RTO of fifteen minutes requires capacity, credentials, routing, automation, and practiced decisions that can actually restore service in that time.

Backups defend against failures that replicas faithfully copy: accidental deletion, corruption, bad migrations, or destructive credentials. A backup plan names the covered data, schema, configuration, indexes, encryption keys, and object storage; its frequency and retention; and the environment in which a restore is rehearsed. A backup that has never been restored is an untested input, not a recovery capability.

Failover is only the middle of the incident. For the checkout system, recovery proof includes:

1. fence the failed writer and establish one authoritative owner;
2. compare acknowledged orders with the recovered order log and payment provider records;
3. reconcile payment_unknown orders and deduplicate callbacks under their stable identities;
4. verify inventory holds, paid orders, tickets, and ledger postings agree;
5. drain outboxes and queues without letting old work starve current traffic;
6. restore traffic gradually while watching correctness as well as latency and error rate.

Only then can the system declare the data safe. Green health checks prove that processes answer; they do not prove that a paid order survived, that a customer was charged once, or that the promoted inventory owner is unique.

Make the interview answer follow the failure

A compact answer can preserve the causal chain:

The promise is one durable order, no duplicate charge, and a truthful status.
I give checkout an end-to-end deadline and let only the payment client spend a
small shared retry budget under one idempotency key. Rising timeouts open a
provider-and-region breaker before payment consumes the checkout fleet. Payment
has isolated capacity; status reads and inventory do not share its exhausted
pool. During the outage, existing orders remain readable and uncertain payments
stay pending rather than becoming guessed failures or successes.

For regional loss, I separate redundant instances from recoverable state. I
name shared dependencies, fence the old owner, and decide which operations stop
until ownership is safe. RTO and RPO determine replication, backup, and failover
choices. Recovery ends after orders, provider records, inventory, tickets, and
ledger entries reconcile—not when the new region merely answers health checks.

That answer gives the interviewer useful pressure points. Change the provider from optional enrichment to authorization. Make the second region’s replica lag. Remove the global control plane. Ask whether pending orders outlive their inventory holds. Reliability judgment appears in the changed behavior, not in the number of mechanisms named.

Practice the cascade

Choose a design you have already practiced and identify one dependency whose latency rises tenfold without failing completely. Before proposing a pattern, write the core promise and trace what the slowness consumes: request time, threads, connections, queue space, locks, memory, or an external quota.

Then design:

1. an end-to-end deadline and per-hop budgets;
2. one owner and one stop condition for retries;
3. breaker scope and half-open probe behavior;
4. the capacity boundary that prevents starvation;
5. the user-visible degraded state and its repair path;
6. the shared dependency that could defeat regional redundancy;
7. RTO, RPO, restore method, and data-level recovery proof.

For the variation, let the dependency recover just as the primary region fails. Decide which queued attempts may replay in the second region and how their identities prevent duplicate effects. If the design can only say “fail over,” the unresolved state has been hidden rather than handled.

Field reference

PROMISE
  Define what users must still be able to trust.

DEADLINE
  Bound the whole request, then give each hop less than the remaining time.

RETRY
  Choose one owner, require safe repetition, add jitter, and spend a shared
  retry budget with an explicit stop condition.

CONTAIN
  Scope circuit breakers to the failure boundary. Isolate pools, queues,
  tenants, and regions so slow work cannot consume every path.

DEGRADE
  Say what remains useful, what stops, what users see, and how accepted work
  will be repaired. Never guess success across an unknown outcome.

RECOVER
  Name independent capacity, shared dependencies, fencing, RTO, RPO, backups,
  restore tests, replay, and reconciliation.

PROVE
  Check business and data invariants before declaring the incident over.

Reliability design is proportion, not maximal redundancy. The system spends time and capacity according to the value of the promise, lets lower-value work fail first, and preserves enough history to recover without guessing. Once that failure boundary is explicit, the next question is who may cross it and what must remain protected while the system is degraded. Security, Abuse, and Privacy takes up that boundary.