Senior Engineering Interview Handbook / Chapter 103
Incident Response
A production-engineering chapter that follows one payment-routing incident from a segmented alert through declaration, command, mitigation, recovery, and handoff.
Page tools
The alert is clear; the safe action is not
At 14:05 UTC, card authorization failures for US checkout traffic rise from a 0.2 percent baseline to 8 percent. Wallet payments remain healthy. The alert arrives because the operational-readiness work from the previous chapter kept provider and region slices visible instead of averaging them into a green global number.
There are two plausible changes. A checkout release reached 20 percent of US traffic half an hour ago, and a payment-routing flag moved part of that cohort to provider B two minutes before the first failures. Logs show timeouts, but a timeout does not say whether the provider rejected an authorization or accepted it and lost the response. A blind retry could turn an availability incident into duplicate financial side effects.
The on-call engineer has enough evidence to know that customers are being harmed and not enough to know why. That is the moment to declare an incident. Normal debugging is optimized for finding a defect. Incident response must simultaneously limit harm, coordinate people, make reversible decisions, communicate what is known, and preserve a path to recovery.
Declaration changes how the team works
“Investigating checkout errors” describes activity but leaves the operating mode ambiguous. The on-call engineer posts a declaration:
14:08 — Declaring SEV-2 for elevated US card-authorization failures.
Observed impact: failures rose from 0.2% to 8% at 14:05; wallet payments
appear healthy. Two recent changes are under investigation: checkout release
2026.07.15.1335 and payment route B enabled at 14:03.
Priya: incident command
Mateo: checkout operations lead
Lena: payments subject expert
Omar: communications
Ren: timeline and decision log
Immediate actions: freeze rollout, preserve failed-request samples and change
state, compare route disablement with release rollback. Next update 14:20.
The severity is provisional. Its purpose is to route urgency, authority, and communication, not to award the incident a permanent label. The team will raise it if impact expands or data integrity becomes likely, and lower it if the affected cohort proves small and contained.
The role split is equally practical. Priya holds the high-level state and decides what question the room must answer next. Mateo directs changes to the system. Lena supplies provider and payment-state knowledge without becoming a second commander. Omar keeps support and leadership informed. Ren records facts, hypotheses, decisions, and results. In a smaller incident one person might hold several jobs, but the jobs still need names.
This division protects attention. The engineers changing production should not also answer every status request. Executives who need visibility receive the update stream instead of joining the diagnostic path. Additional people enter for a specific expertise or authority gap, not as spectators.
Frame impact before choosing a cause
The first command question is not “Which change is guilty?” It is “What harm is occurring, where, and how is it changing?” At 14:11, the incident record contains this snapshot:
Affected: US card attempts routed to provider B; mobile and web both present
Not observed: wallet failures, non-US failures, completed-order regression
Trend: provider-B authorization errors still rising; checkout latency rising
Uncertain: whether timed-out requests were accepted; duplicate risk unknown
Recent changes: checkout release at 13:35, route-B flag at 14:03
Customer evidence: 11 support contacts, 4 request IDs linked to failed orders
Scope is expressed as claims with evidence, including the limits of that evidence. “Wallet payments are not observed to be affected” is more honest than “wallets are fine” when the team has checked only a few minutes of data. The recent release is suspicious but not yet a cause. Keeping both changes in view prevents the room from fitting every new fact to its first theory.
Three workstreams now run in parallel. Operations tries to stop new harm. Diagnosis distinguishes a provider-route failure from a release defect and tests the ambiguous-success path. Coordination maintains the shared state, decision cadence, and communications. Diagnosis serves mitigation; it does not get to hold customers hostage until root cause is certain.
Preserve what the next decision needs
Before changing traffic, Ren captures the routing flag version, release ID, provider response codes, four representative traces, the authorization-rate dashboard by provider and region, queue age, and the first support examples. Mateo stops the rollout at its current exposure. These actions take less than a minute and preserve the comparison the team will need after mitigation.
Evidence has a live operational purpose. A before-and-after trace can show whether disabling route B changed the failure. Provider references can separate declined requests from accepted requests whose response was lost. The decision log explains why an action was reasonable with the information available at the time.
Collection must remain proportional. Customers should not wait through a perfect archive while the failure grows. Sensitive customer and payment data stays in approved systems with access limited to the responders who need it. If the signal suggests intrusion, fraud, or regulated-data exposure, the team invokes the organization’s security-response and evidence procedures rather than treating an ordinary incident channel as a forensic repository.
At 14:13, the decision log reads:
Decision: disable new traffic to provider B; pause automatic retry of ambiguous
authorization results; keep wallet checkout available.
Why: route enablement is the closest correlated change and is independently
reversible. A full release rollback would also change checkout code and would
not resolve authorizations already accepted without a response. Blind retries
could create duplicate side effects.
Risks checked: provider A has current headroom; flag propagation is observable;
ambiguous requests enter a bounded reconciliation queue; no schema rollback is
required.
Success signal: provider-B exposure reaches zero, new authorization success
returns toward baseline, ambiguous queue stops growing. Reassess at 14:20.
This is mitigation under uncertainty. It does not pretend to be the final fix. It chooses the smallest action that attacks the strongest correlation while containing the dangerous unknown.
Communication is part of containment
At 14:14, support and leadership need different depth but the same facts. Neither audience needs the room’s running theories.
We are seeing elevated card-authorization failures for some US checkout
traffic beginning at 14:05 UTC. Wallet payments are not currently affected.
We have stopped new traffic to the affected payment route and are reviewing
requests whose result is uncertain before retrying them. Next update at 14:25.
The update states observed impact, scope, current action, and cadence. It does not blame the release, promise a recovery time, or use “resolved” while customer orders remain uncertain. Support can add approved guidance for customers; leadership can decide whether more authority or external review is needed. Omar absorbs incoming questions so Priya can keep command and Mateo can operate the system.
Good communication changes system behavior. It prevents another team from re-enabling the route, stops support from advising customers to retry blindly, and keeps side conversations from producing conflicting instructions. During an incident, a factual update is an engineering control.
A green rate is only the start of recovery
By 14:20, new authorization success has returned to baseline. Disabling route B worked, but the incident is not resolved. The reconciliation queue contains 642 requests with an uncertain provider outcome. Some customers may retry on their own. Support volume is still rising. Provider A is carrying the shifted load.
The team defines recovery in terms of the harmed journey:
- new card attempts remain at baseline by region, app version, and provider;
- provider A retains safe capacity under the shifted traffic;
- every ambiguous request is classified before any retry or customer action;
- duplicate authorization and capture checks remain clean;
- completed orders reconcile with payment state;
- the uncertainty queue drains and support contacts begin to decline;
- the route-B flag remains owned, documented, and disabled through the watch period.
Where a provider reference exists, the reconciliation worker queries the recorded outcome. Requests without enough evidence go to controlled review rather than being guessed into success, failure, or retry. The repair path is slower than a queue purge, but it preserves the financial invariant: the team must not trade a lower backlog number for duplicate or missing payment state.
At 14:37, Lena confirms that provider B changed timeout behavior during a regional degradation. Mateo also finds that the new release classified one ambiguous response as retryable. The provider event explains the initial concentration; the release enlarged the duplicate risk. The room does not need to choose one story while both conditions have operational consequences.
At 15:05, authorization success has remained healthy for 45 minutes, provider A has headroom, the ambiguous queue is empty, order and payment reconciliation is clean, and support contacts are falling. Priya moves the incident from active response to watch. The service owner takes the remaining tasks:
Watch until 16:00: authorization success, provider-A saturation, duplicate
checks, support contacts. Route B remains disabled; owner Lena.
Open risks: timeout classification is unsafe for ambiguous provider outcomes;
route-B behavior needs vendor confirmation; customer follow-up remains with
support.
Handoff artifacts: timeline, change IDs, traces, decision log, reconciliation
result, communications, and unresolved questions.
Resolution is a decision with criteria, not the moment a dashboard first turns green. The watch owner must know what would reopen the incident, and every temporary mitigation must have an owner and an exit condition.
What the incident model transfers
The details change with the failure. During regional saturation, moving traffic may overload the destination, so capacity and retry amplification constrain failover. During suspected data corruption, a write pause and auditable repair may matter more than availability. A narrow internal-tool failure may need only one owner and a short record. The response should scale with harm, uncertainty, recurrence risk, and the authority required.
The underlying questions remain compact:
- What customer, data, or trust harm is occurring, and how is it changing?
- Who owns command, system changes, communication, and the decision record?
- Which bounded action reduces harm without creating a worse recovery path?
- What evidence is worth preserving before that action?
- What facts should each audience receive, and when will they hear again?
- Which user, system, data, backlog, and support signals prove recovery?
- Who owns the remaining risk and temporary state after active response?
In a production-engineering interview, work those questions through the specific failure instead of reciting an incident checklist. In a system design discussion, show where the architecture exposes user harm, supports bounded mitigation, and preserves state for reconciliation. In a project or behavioral discussion, describe the choices available at the time, including the uncertainty and authority boundaries, rather than polishing the event into a hero story.
To practice, change one fact in the payment incident: provider A lacks enough capacity for the traffic shift, or the ambiguous requests concern account balances rather than authorizations. Write the declaration, the next decision log entry, and the recovery criteria. A copied sequence will not solve the variation; the safe mitigation must follow the new constraint.
The incident record now passes to the learning review. That review may explain why the provider event, retry policy, rollout controls, and reconciliation path combined as they did. During the live response, the team’s accomplishment was more immediate: it made uncertainty visible, limited new harm, restored the customer journey without corrupting payment state, and left the next decision enough evidence to be honest.