Observability and Operability

Every service is healthy. Checkout is broken.

A buyer submits a valid cart. The payment provider authorizes the card, the client waits, and the request times out. No order appears. A retry returns the same generic error.

The service dashboard is reassuring: the gateway is answering requests, CPU is low, the database is available, and the payment adapter reports mostly 2xx responses. Each component is alive. The product promise is not.

This is the useful test of observability in a system design. Can the system recognize that a buyer may have paid without receiving an order? Can an operator follow that one attempt across services and queued work, distinguish an authorization from a completed order, and choose a mitigation that will not charge the buyer twice? Observability supplies the evidence. Operability is the ability to use that evidence for a safe decision.

Reliability mechanisms still have to do the work described earlier in the design: idempotency, timeouts, retries, durable state, isolation, and reconciliation. Telemetry cannot repair an unsafe transaction model. It can reveal where the model has entered an ambiguous state and tell an operator which recovery path the architecture permits.

Turn the promise into something measurable

For checkout, “the API is up” is not the promise. A better statement is: a valid checkout attempt reaches one durable outcome within the buyer’s waiting window. Depending on the product, that outcome may be a committed order or a clear rejection with no lingering payment authorization or inventory hold.

That statement yields two kinds of evidence. The journey evidence asks whether valid attempts complete successfully and how long they take. The invariant evidence asks whether the states agree: no captured payment without an order, no committed order without its fulfillment event, and no accepted attempt left indefinitely unresolved.

An SLI measures one of those properties. An SLO gives the measurement a target and a window. For example:

SLI: completed valid checkout attempts / all valid checkout attempts
SLO: the agreed fraction completes within the latency bound over 28 days

invariant signal:
  payment authorized
  AND no order committed or authorization released within 5 minutes

The exact fraction and time bounds come from product risk, traffic, dependency behavior, and the team’s ability to operate the design. Repeating “four nines” without those constraints is decoration. A billing authorization path, a fraud-review queue, and an internal merchandising tool need not share a target.

The SLO’s error budget makes unreliability finite rather than aspirational. A team can spend some of that budget on change, but a fast burn says that the current failure rate will consume the allowance too quickly. Burn-rate alerts are often more useful than a single instantaneous threshold because they can distinguish a sharp incident from ordinary variation while still catching a slow, sustained regression.

Component measurements remain valuable, but they play a different role. CPU, connection-pool use, queue depth, replica lag, provider latency, and cache evictions help explain a breach or predict one. They should not be allowed to outvote the user’s experience. In the opening failure, green components are clues, not acquittals.

Design the path from symptom to state

The checkout attempt crosses a gateway, checkout service, inventory service, payment adapter, order store, outbox, and fulfillment worker. A request ID from the edge will not necessarily survive a delayed retry or queue redelivery. The architecture needs durable identifiers for the business transitions:

gateway request_id
  -> checkout_attempt_id
  -> idempotency_key
  -> inventory_reservation_id
  -> payment_authorization_id
  -> order_id
  -> outbox_event_id
  -> fulfillment_job_id

These identifiers do more than make log search convenient. They define which records can be reconciled and which repeated operations must collapse to the same effect. The checkout attempt should retain the provider authorization ID even if the client connection disappears. The order commit should be discoverable from the idempotency key. The outbox event should identify the order whose downstream work it represents.

Metrics, traces, and logs then answer different questions. The SLI metric says that valid checkout completion fell in one region. A dashboard places that symptom beside traffic, latency, saturation, dependency errors, queue age, deploy markers, and feature-flag state so the operator can see scope and change. A trace follows one attempt through synchronous calls and preserves links to later spans created by queued work. Structured logs explain local decisions: the adapter received a new response shape; the retry policy classified it as transient; the order commit never began.

No one signal needs to contain the whole story. In fact, forcing every detail into metric labels makes the telemetry expensive, slow, and dangerous. User IDs, order IDs, document IDs, and raw URLs create unbounded cardinality and may copy sensitive data into systems with broad access and long retention. Keep aggregate dimensions bounded. Put high-cardinality identifiers in access- controlled traces or sampled structured logs when the investigation requires them.

The same restraint applies to contents. Logs and traces should not capture raw tokens, card data, private messages, document bodies, or complete request payloads by default. Record safe error classes, stable identifiers, versions, durations, and state transitions. If diagnosis genuinely needs sensitive context, narrow access and retention rather than quietly turning the observability platform into a second production database.

A page must imply an action

Suppose valid checkout completion begins burning its error budget after a canary deployment. The page should identify the affected promise, severity, scope, owner, and current burn. It should link the dashboard and runbook, group related symptoms, and carry enough deployment context that the responder does not have to rediscover what changed.

The dashboard first answers four questions:

1. Which buyers, regions, versions, or tenants are affected?
2. When did the change begin?
3. Which dependency or resource now differs from the healthy population?
4. Which mitigation is safe for this failure state?

In this incident, the canary cohort has elevated payment-adapter latency and schema errors. Traces show retries continuing until the client times out. Payment authorizations exist for some attempts, but order commits do not. That last fact changes the response: blindly retrying checkout is unsafe.

The runbook therefore separates service restoration from data repair. To stop new harm, the responder can roll back the adapter, disable the new provider route, or shift traffic to a known-good integration. To resolve existing attempts, a reconciliation job compares checkout, provider, inventory, and order state using the durable identifiers. It completes an idempotent commit when that is safe or releases the authorization and reservation when it is not. It records unresolved cases for review instead of inventing an outcome.

A useful runbook names confirmation queries, permitted mitigations, rollback conditions, actions requiring approval, escalation targets, and checks for replay or repair. “Look at the logs” is not a runbook. Neither is a sequence that depends on one database expert manually editing rows from memory. Awkward recovery instructions are architectural evidence: they point to missing idempotency, controls, ownership, or repair paths.

Not every abnormal measurement should wake someone. A full disk, rising queue, or expiring certificate deserves a page when it threatens the promise soon or risks irreversible harm. Otherwise it may belong on a dashboard, a ticket, an autoscaling policy, or a capacity review. Paging without a plausible action trains responders to distrust the system that is asking for attention.

Background work has user-facing truth

The HTTP request can succeed while the product quietly fails later. An order may commit while its outbox stalls, leaving fulfillment unaware. A video upload may return 202 Accepted while transcoding never makes it playable. A search write may succeed while index lag leaves the document undiscoverable.

For queued work, backlog size alone is ambiguous. Ten thousand jobs may be healthy at one throughput and disastrous at another. Queue age, age by priority, retry rate, dead-letter volume, and completion latency reveal whether accepted work is still meeting its freshness promise. The durable workflow ID must cross queue messages, retries, workers, and derived stores so an operator can distinguish delay from loss and safe replay from duplicate effect.

Capacity evidence should follow the resource that can actually saturate. A worker pool exposes utilization and queue age. A database exposes query latency, lock waits, connection pressure, storage growth, and replication lag. A partitioned service exposes hot keys and skew, not only fleet averages. An external dependency exposes quota use, throttling, timeout rate, and the retry load the system is adding to the problem.

The useful question is always what decision the signal supports. Rising queue age may justify adding workers, shedding low-priority work, or pausing a backfill—but only if the database and provider can absorb the extra concurrency. Replication lag may delay a failover rather than trigger one. Cache eviction may call for a larger cache, a better key design, or removal of a layer that was hiding an unbounded access pattern. A capacity graph without the corresponding constraint can recommend the wrong cure.

Migrations, backfills, re-indexes, and repair jobs need their own promises. Track progress, lag, error samples, skipped records, version boundaries, idempotent retries, and pause or rollback behavior. The serving fleet can look perfectly healthy while a migration corrupts derived state one batch at a time.

Make the deployment prove itself

Observability changes release design. Before shifting traffic, name the user SLIs and invariants the change could harm. Compare the canary with a healthy cohort, mark versions and feature-flag state in traces and dashboards, and set rollback criteria while the decision is still calm. A global average can hide a severe canary failure; the comparison must preserve version, region, or tenant scope without creating uncontrolled cardinality.

The checkout release needs more than HTTP error rate. It watches valid checkout completion, latency, payment technical failures, unresolved authorizations, order-commit reconciliation, and outbox age. If the schema change affects only one provider region, that slice must remain visible. If rollback cannot undo a schema migration or external side effect, the plan needs a compatible forward path and a repair procedure before rollout.

After the incident, the control loop closes. The team may add a contract test for the provider response, stop classifying schema errors as retryable, make the unresolved-payment invariant a first-class signal, or automate the safe feature-flag action. The purpose is not to collect a larger postmortem. It is to make the next breach easier to detect, explain, and recover from—or prevent it entirely.

Give the interview one complete control loop

A compact answer can stay attached to the design:

For checkout, I would measure valid attempts that reach a durable outcome, not
just gateway availability. I would set the target from the buyer promise and
page on fast or sustained error-budget burn. The health view would put that
SLI beside region, version, deploy markers, payment errors, order commits,
outbox age, and saturation.

Each attempt carries an idempotency key and durable IDs through payment,
inventory, order commit, outbox, and fulfillment. Traces connect the path;
structured logs record safe state transitions and error classes without card
data. The runbook can roll back or disable a provider route, then reconcile
authorized-but-uncommitted attempts without double charging.

For capacity, I would watch queue age, hot partitions, connection pressure,
provider quotas, retries, and storage growth. A canary has explicit rollback
criteria for the user SLI and the payment-order invariant. That gives the team
a path from broken promise to cause and safe action.

This answer leaves room for useful pressure. Ask what counts as a valid attempt. Make the payment provider slow rather than unavailable. Lose the trace sampling for one customer. Let the outbox backlog grow while API latency stays flat. Require a rollback after a non-reversible migration. The answer remains coherent because each variation changes evidence or action in the same control loop.

Practice the breach

Choose one design you know and write a single promise in user terms: a message arrives, an upload becomes playable, a feed becomes fresh, an entitlement is correct, or a workflow reaches a durable outcome. Define one journey SLI and one invariant signal. Then trace a failed instance through every service, queue, job, store, and derived read path.

Decide what pages a person, what remains a dashboard or ticket, and what can be handled automatically. Draft the first safe mitigation and the repair step that follows it. Add a canary release that breaks only one region or tenant cohort, then make a background job fall behind without changing HTTP success. If your evidence cannot distinguish those failures, the design is not yet operable.

For a harder variation, remove the main trace for the failed request. Explain which metrics reveal scope, which durable business identifiers reconstruct the path, which logs remain safe to query, and where the investigation must admit uncertainty rather than infer a state change that was never recorded.

Field reference

PROMISE
  Name the critical journey and the invariant that must survive partial
  failure. Define the population, measurement, target, and window.

DETECT
  Page on user harm, error-budget burn, or imminent irreversible failure.
  Give the alert an owner, scope, evidence, and plausible action.

EXPLAIN
  Put traffic, errors, latency, saturation, dependencies, deploys, and affected
  cohorts together. Carry durable identifiers across services and queued work.

RECOVER
  Separate immediate mitigation from reconciliation and repair. Make rollback,
  traffic shift, degraded mode, replay, and escalation conditions explicit.

PROTECT
  Bound metric cardinality. Keep secrets and contents out of telemetry; narrow
  access and retention when sensitive diagnostic context is unavoidable.

ANTICIPATE
  Track queue age, hot partitions, storage growth, quotas, retries, dead
  letters, migrations, backfills, canary health, and rollback readiness.

LEARN
  Turn incidents and awkward runbooks into safer state models, automation,
  tests, signals, and recovery controls.

An operable design does not promise that every failure will be obvious. It makes the important promises measurable, preserves the state needed to investigate them, and limits the actions an operator must invent. Those choices also consume storage, bandwidth, vendor capacity, and human attention. Cost and Architectural Economics asks whether that operating model is economically credible.