Incident Response Template

Open One Incident File

A founder learns at 8:35 a.m. that scheduled client reports have not arrived. The queue is growing, two customers have written, and a release went out the previous evening. This is exactly when memory becomes a poor operating system. The founder is investigating, answering customers, changing production, and deciding what to postpone with the same pair of hands.

Open one incident file as soon as a failure threatens customer work, money, access, data, privacy, a promised deadline, or trust. Use it as the shared truth even when the company consists of one person. It should answer five questions throughout the incident: who is affected, what is still at risk, what has been done, what customers have been told, and what evidence would justify closing.

Do not wait for a confirmed cause. Detection begins an incident; diagnosis can follow.

INCIDENT HEADER
Incident ID and opened at:
Owner:
Short description of the broken customer promise:
How it was detected:
First known failure and most recent failure:
Affected or possibly affected customers, records, jobs, or regions:
Customer consequence—work / money / access / data / privacy / deadline:
Current state—investigating / contained / recovering / monitoring / closed:
Next update due:
Links to logs, jobs, deploys, support cases, and status communication:

Write the short description in customer terms. “Reports due this morning have not been delivered” is more useful than “worker errors.” The second statement may describe a mechanism; the first tells the founder what must be restored.

Set Urgency from Consequence

Choose a response level that changes what you do. Use critical when harm is active or can compound quickly—for example, cross-account data exposure, incorrect charges, loss of access to essential work, destructive corruption, or a security compromise. Stop risky activity, protect evidence, and get specialist help when the incident exceeds your competence or authority.

Use urgent when a core paid workflow is blocked, a material deadline is at risk, or several customers are affected but the damage is bounded. Defer normal work and communicate on a short clock. Use routine when the fault is narrow, work remains recoverable, and a delayed response will not increase harm. It still needs an owner and a closure condition; it does not need to consume the whole company.

If the scope is unknown, classify from the worst credible consequence until evidence narrows it. Low traffic does not make exposed private data a minor incident, and a loud customer does not by itself make a reversible cosmetic defect urgent.

IMPACT AND RESPONSE
Response level and reason:
Confirmed impact:
Possible impact still being bounded:
What could make harm grow:
What has been paused or disabled:
Normal work displaced until:
Outside help needed, from whom, and why:

Stabilize Before You Explain

The first useful action often reduces the rate of harm rather than repairs the root cause. Pause a worker that may duplicate reports. Disable a newly broken path. Revoke a compromised credential. Preserve the database, logs, queue state, and provider records before a cleanup destroys evidence.

Choose the smallest containment that protects customers without disabling unrelated value. For the report incident, the founder pauses report delivery but leaves viewing and editing available. They confirm that reports were generated correctly and that only notification is failing. This is narrower and safer than rolling back every service because the release time looks suspicious.

Before retrying, restoring, or replaying anything, ask what the first attempt may already have changed. If you cannot tell whether an email was sent, a charge was made, or data was overwritten, another attempt can become a second incident.

Keep a chronological decision log. It need not capture every query. Record observations that change scope, actions that change production, and decisions another tired version of you might otherwise repeat.

TIMELINE AND DECISION LOG
[time] Observation:
Evidence:
Decision or action:
Expected effect:
How and when it will be checked:
Result:

Separate observations from hypotheses. “No delivery records exist after 07:52” is evidence. “The release broke the mail worker” is a hypothesis until the mechanism is established. This distinction keeps an attractive theory from outrunning customer recovery.

Communicate on a Clock

Customers should not have to rediscover the incident or ask whether their data is safe. Send an initial note once you can state the affected promise and the known risk honestly. If investigation continues, name the next update time and send it even when there is no resolution.

CUSTOMER UPDATE
What customers are experiencing:
What is and is not affected:
What has been done so far:
What customers should do—or why no action is needed:
When the next update will arrive:
Where questions should go:

Prefer a bounded truth to premature reassurance. “Scheduled reports are delayed; report data remains available in each workspace; please do not reschedule delivery; the next update is at 10:00” gives customers something safe to do. “We are aware of intermittent issues and working hard” gives them almost nothing.

Use the narrowest channel that reaches everyone affected. A direct message may be right for two known accounts. A status page or in-product notice becomes useful when the affected set is larger or still changing. Keep support replies consistent with the incident file so one hurried answer does not make a promise the system cannot keep.

Recover the Customer Outcome

Recovery is not the absence of new errors. Define it before acting. For the report incident, the founder needs every affected report delivered once, the workspace to show the correct delivery state, and the oldest queued report to remain inside its promised window. Restarting the worker proves none of those things by itself.

RECOVERY PLAN
Known safe state:
Recovery action and guardrail against duplicate or destructive effects:
Customer outcome that must be restored:
Stored state to verify:
Customer-visible result to verify independently:
Accounts or samples checked:
Monitoring window and regression signal:
Rollback, forward repair, or stop condition:

Verify from both sides of the boundary. Check internal records, then inspect the result available to the customer or obtain confirmation appropriate to the promise. A provider accepting an email is not the same as a customer receiving it. A restored database row is not necessarily a restored workflow. Sample checks are reasonable only when you can explain why the sample covers the failure and what signal would reveal a missed case.

Move the incident to monitoring when the customer outcome is restored and no known harm is growing. Close it only after the monitoring window passes, the final customer update is sent, and every temporary bypass has an explicit owner or has been removed.

Carry Forward One Better Obligation

An incident review is not a search for blame or a licence to build a reliability platform. Reconstruct the sequence from the log and find the earliest practical point where a different control would have reduced customer harm or founder effort. The answer might be earlier detection, a safe retry, a restore test, a narrower product promise, a provider fallback, or removal of a brittle feature.

CLOSURE AND FOLLOW-UP
Closed at and by:
Customer impact, including duration:
What failed and why—mark unknowns honestly:
Why existing detection or safeguards did not contain it sooner:
Recovery evidence:
Final customer communication:
Temporary changes still in place:
One committed follow-up—owner, due date, and expected risk reduction:
Deferred ideas and the evidence that would justify them:
Support cases, regression tests, runbooks, or product decisions linked:

Some incidents deserve several actions, especially those involving security, privacy, money, or data integrity. Even then, distinguish the risk-reducing commitments from the cleanup list. “Improve monitoring” is not a commitment. “Alert when the oldest undelivered paid report is ten minutes from its promised window” names the failure, signal, threshold, and customer consequence.

The incident file is complete when a later reader can reconstruct what the customer experienced, why each consequential action was safe, how recovery was proved, and which new obligation the founder accepted. That record feeds the next support and hardening decisions. It should leave the product with a more honest promise, not merely a longer backlog.