One-Person Ops Runbook

Write It Before the Failure

A runbook is a promise made by the calm version of you to the tired version. It covers one failure that can interrupt the product’s customer promise: a paid account has no access, a scheduled export is late, a background job is stuck, a release must be rolled back, or important data must be restored.

Write one runbook for one failure path. “Database problems” is too broad. “Primary database unavailable” and “one customer record deleted” have different evidence, containment, recovery, and customer consequences. If the page tries to cover both, it will become a list of tools rather than a path to safety.

The runbook is not the live incident record. It describes the prepared route. During an incident, record what actually happened, which route you took, and where reality departed from the page.

Open on a Customer-Visible Trigger

Name the customer-visible promise before naming infrastructure. “Weekly reports arrive by 08:00” provides a sharper boundary than “the worker is healthy.” Then state the signal that opens the runbook: a missed-delivery alert, an impossible product state, a support report with confirming evidence, or a failed restore check.

Name the Stop Condition

Also state when the founder must stop. A useful runbook does not turn uncertainty into permission. Stop the prepared action when you cannot identify the affected account, cannot tell whether an earlier attempt produced side effects, suspect unauthorized access or data exposure, lack a usable recovery point, or would exceed a legal, contractual, or vendor boundary. Preserve evidence, contain further harm, and move to the appropriate security, privacy, legal, or vendor escalation instead.

The stop condition protects against the most tempting operational mistake: repeating an action merely because its first result is unclear.

Copyable Runbook

Store links and opaque identifiers here, not credentials, customer exports, access tokens, or copied private data. Commands should mark account and object IDs as values the operator must supply. Any destructive command needs a dry-run, preview, backup, or explicit recovery step appropriate to its harm.

# Runbook: [one observable failure]

Owner:
Last reviewed:
Last rehearsed:
Customer promise or workflow at risk:

[OPEN THIS RUNBOOK WHEN]
Trigger or alert:
Signal source and link:
Expected normal state:
Known false-positive condition:

[STOP AND ESCALATE WHEN]
Unknown side effects that make repetition unsafe:
Security, privacy, money, access, or data boundary:
Evidence that must be preserved:
Vendor or specialist contact:
Customer support or incident channel:

[ESTABLISH SCOPE]
Customer, workspace, job, invoice, release, or dataset identifier:
Authoritative state to inspect:
Related event, operation, and provider identifiers:
Most recent relevant change:
How to find every affected customer or object:
Customer-visible impact:
Possible impact to money, access, data, deadlines, or trust:

[STABILIZE]
Work to pause, disable, rate-limit, or isolate:
Safe state to preserve:
How to confirm that harm is no longer increasing:

[DIAGNOSE]
Check 1 — observation and meaning:
Check 2 — observation and meaning:
Check 3 — observation and meaning:
If the observations do not match a known path:

[CHOOSE ONE RECOVERY PATH]
Path A — preconditions:
Path A — bounded action or command:
Path A — duplicate and partial-failure guard:
Path A — rollback, compensation, or fallback:

Path B — preconditions:
Path B — bounded action or command:
Path B — duplicate and partial-failure guard:
Path B — rollback, compensation, or fallback:

[VERIFY INDEPENDENTLY]
Stored product state to confirm:
Customer-visible outcome to confirm:
Data, billing, access, or notification side effects to confirm absent or correct:
Signal that must remain healthy, and for how long:

[COMMUNICATE]
Affected-customer source:
Initial message location or template:
Known facts to include:
Customer action, if any:
Next-update time when work remains:
Recovery message and delivery check:

[CLOSE AND IMPROVE]
Incident record location:
Operator, times, evidence, action, and result recorded:
Temporary controls removed or retained with an expiry:
Follow-up decision: repair | instrument | automate | narrow promise | keep manual
One owner and review date:
Runbook correction discovered during use:

Worked Runbook: Report Exists, Delivery Does Not

A reporting product promises that each workspace’s weekly report will be available and emailed by 08:00. At 08:05, an alert says that a report exists for workspace ws_184, but no successful delivery event exists. The runbook opens on that exact disagreement; it does not open for every email-provider warning.

The founder first establishes scope. The product database is authoritative for report generation and the intended recipient reference. The delivery ledger is authoritative for the product’s send attempts. The email provider can show whether it accepted or rejected a particular message. Searching the same delivery window finds two affected workspaces, not one. The reports exist, and no accepted provider message or customer-visible “sent” state exists for either.

The stop conditions are explicit. Do not resend when an earlier attempt has an unknown provider result, when the recipient cannot be resolved from current authorized product state, or when the report itself is incomplete. Do not attach the report to ordinary email as an improvised fallback if its contents require authenticated access.

Stabilization means pausing the scheduled sender for this delivery window, not disabling report generation or the whole application. The founder records the two delivery IDs and confirms that the backlog is no longer growing. A recent credential rotation now supplies the likely cause: the provider rejects new requests as unauthorized.

The prepared recovery has two paths. If the provider credential can be restored through the normal secret-management path, update it, send one non-customer test, and replay each failed delivery by its existing delivery ID. The replay operation checks for an accepted provider message before sending and records one new attempt, so a browser retry cannot create a second email. If the provider remains unavailable, publish an in-product notice that the reports are ready and send a plain notification through the approved fallback channel; the report itself remains behind normal product access.

Verification crosses the boundaries where the failure occurred. For each workspace, the founder confirms the intended report is still the current completed report, the delivery ledger has exactly one successful outcome, the provider accepted the matching message ID, and the authenticated product link opens the correct report. A provider success alone would not prove that the customer received the right result. A healthy application page would not prove that the notification left the provider.

The customer note says that the weekly report was ready on time but its notification was delayed, that the report is available in the product, whether a later message was sent, and whether the customer must act. If delivery remains unresolved, it names the next update time. The incident record keeps the affected workspace references, timing, provider result, action, and outcome without copying report contents or addresses into the runbook.

The follow-up is smaller than “build a new mail system.” Keep the alert on the promise-level disagreement, add a credential-expiry check before the delivery window, and rehearse the fallback. If provider failures become frequent, the evidence may justify a second delivery path. One incident does not.

Rehearse From the Signal

Read-through is not rehearsal. Trigger the safest representative failure in a test or isolated environment and begin where an alert or support message would place you. Follow the page without filling gaps from memory. Use the identities and permissions intended for an incident, exercise a duplicate attempt, and verify the customer-visible result separately from the internal state.

Record every hidden dependency: an expired login, an unlinked dashboard, a command that assumes the current directory, a credential only one machine possesses, or a manual choice the page never explains. Correct the runbook or the product before calling the path usable.

Review the page after a related product change, vendor change, incident, or rehearsal. Retire it when the failure path disappears. A runbook that preserves an obsolete workaround is operational debt with instructions attached.

Keep Only the Page You Can Use

The runbook is ready when a future, interrupted founder can recognize the trigger, establish scope, stop unsafe repetition, choose a bounded recovery, verify the customer outcome, communicate without invention, and leave an honest record. If following it still requires blind production edits, private memory, or a lucky guess about side effects, the product does not yet have a recovery path; it has a familiar improvisation.