Reliability Without Overengineering

The Packet That Did Not Arrive

At 4:10 on Thursday afternoon, a founder notices that one customer’s weekly compliance packet has no output. It is due Friday morning. The source evidence is present, the generation job says running, and the worker that claimed it no longer exists.

The product is young. Six companies use it, and several parts of the workflow are still manual. Multi-region infrastructure would be absurd. So would telling the customer on Friday that early products sometimes break.

The useful question is not how reliable the entire product ought to be. It is what this particular promise deserves.

The marketing site’s decorative preview can fail with little consequence. The first evidence upload must work well enough to reveal whether a new user can reach value. Billing must preserve the truth about money and access. The weekly packet carries a deadline and the customer’s reputation. One product therefore needs several reliability postures at once.

This is reliability before product-market fit: protect the workflows whose failure would corrupt learning, damage trust, create intolerable support work, or prevent honest use. Leave the rest simple enough to change.

Price the Failure Before Buying Machinery

Reliability work begins with a failure, not a tool. For the packet, the founder names the worst credible outcome:

A customer believes the packet will be ready for an audit-preparation meeting, discovers too late that it is missing, and cannot tell whether the uploaded evidence is safe or must be submitted again.

That sentence reveals four costs. A deadline may be missed. Customer data may be placed at risk by an unnecessary second upload. The founder may spend the evening reconstructing state. Later retention data may blame weak product value for what was actually broken delivery.

Now compare a theme preference that fails to save. The error is visible, the underlying work is intact, and the customer can continue. It needs an error message and an ordinary defect fix, not a pager and a recovery system.

Make the comparison for each important workflow. Ask:

• What does the user reasonably believe the product promised?
• What can be lost or made untrustworthy: work, money, access, privacy, a deadline, or a decision?
• Would the founder know about failure before the customer reports it?
• Can the failure be repaired safely, and how much founder time would repair consume?
• Is the workflow experimental, recurring, paid, or already embedded in real work?

Criticality is local to the workflow. Stage changes how much machinery is justified, but it does not erase the promise already made. A pilot may need a tested restore for customer evidence while needing no automated failover. A disposable prototype may need almost nothing beyond visible failure. A payment flow can be low volume and still deserve exact records and safe correction.

The ladder is not a release sequence for the whole architecture. It is a way to resist two comforting mistakes: hardening everything because infrastructure feels controllable, and excusing dangerous gaps because the company is early.

Establish the Smallest Reliable State Machine

The stuck packet cannot be repaired safely while running is the only useful fact. That label hides the mechanism the founder must reason about. Did the worker merely claim the job? Did it read every source file? Did it write a partial packet? Did it mark the packet current? Did it notify the customer?

The founder gives the workflow a small explicit state model:

1. scheduled: a packet is expected for an account and delivery window;
2. claimed: one attempt has acquired the work;
3. assembling: source evidence is being read and validated;
4. ready: one immutable output has been committed;
5. delivered: the intended recipient can reach that output;
6. failed: the attempt stopped with a recorded reason and last safe state.

This is not ceremony. State makes recovery decisions possible. A lease or timeout can return abandoned claimed work to an inspectable state. An attempt identifier distinguishes the dead worker from a later retry. Committing the output before changing the current-packet pointer prevents a partial file from becoming the customer’s packet. Recording delivery separately prevents “generated” from being confused with “received.”

The exact states will differ by product. The durable principle is that an important asynchronous workflow must reveal where it stopped, what durable effects occurred, and which transition may happen next.

Queues can help absorb work and retry transient failures, but a queue does not supply that truth by itself. An invisible job can still be duplicated, stuck forever, or acknowledged before its result is safe. Likewise, a retry policy is not reliability if the operation can charge twice, send two customer messages, overwrite newer work, or create conflicting records.

For the packet job, retry is keyed to the account and reporting period. Before committing, an attempt checks whether a current packet already exists for that period. Notification has its own delivery record. Running the attempt twice may waste computation, but it cannot publish two competing packets or send two “ready” messages. Where duplicate side effects cannot be prevented, they must at least be detected and reconciled.

Make Failure Observable at the Promise

The product was technically “up” while the packet was missing. Uptime at the web server therefore says little about this customer’s experience. The useful reliability signal lives at the promise: packet ready before its delivery window.

The founder records the account, reporting period, attempt, source version, current state, timestamps, output identifier, delivery state, and a bounded failure reason. These records avoid secrets and unnecessary customer content; they preserve enough context to answer who was affected and what happened.

Only one condition interrupts the founder immediately: a trust-bearing packet is likely to miss its promised window or has failed in a way that risks source data. The alert includes the account, job and attempt, last safe state, time remaining, and link to the recovery action. Lower-priority errors go into a scheduled review.

An interruption earns its place when waiting increases harm. A transient error that recovered automatically may need a record, not a notification. A theme-setting defect can wait. A rise in third-party latency may deserve observation until it threatens an actual promise. If the founder repeatedly receives an alert and takes no action, the signal is badly placed, too sensitive, or not an alert at all.

Rate limits belong to the same reasoning. They are useful when a dependency can be overwhelmed, a customer can accidentally create expensive work, or abusive traffic can deny real usage. A generic rate-limiting layer copied across every endpoint can impose more failure states than it prevents. Limit the costly or dangerous operation, return a clear response, preserve the work where appropriate, and tell the founder enough to distinguish protection from a product defect.

Recover Before You Engineer for Continuous Survival

At six customers, the packet service does not need to survive every machine failure without interruption. It does need to return the customer to a known, safe state.

The founder marks the abandoned attempt failed, verifies that no output was committed and no notification was sent, then starts a new attempt from the stored source version. The new packet is compared with its source manifest, committed, exposed through the customer’s workspace, and only then announced. The product records the repair. The customer receives the packet before the deadline and never has to guess whether to upload again.

Recovery is often the best early reliability investment because it remains useful while the architecture changes. Depending on the workflow, it may mean:

• retrying from a known checkpoint without duplicating side effects;
• rolling back application code while explicitly accounting for migrations and background work;
• restoring a deleted record or file into an isolated place before overwriting current state;
• correcting money or entitlement state against the provider’s authoritative record;
• disabling one risky capability for one account;
• completing a promised artifact manually from preserved inputs.

Each action needs a stop condition. If the founder cannot tell whether a charge occurred, an email was delivered, or newer data will be overwritten, “try again” is not recovery. Preserve the evidence and choose an action that cannot multiply harm.

Backups deserve the same skepticism. A successful backup job proves that bytes were copied somewhere. It does not prove that the expected data is present, that credentials are available during an incident, that the copy matches the current software, or that restoration preserves newer work.

For the evidence product, the founder restores a snapshot into an isolated environment, opens one customer’s source manifest, rebuilds one known packet, and records the recovery point and elapsed effort. The test is modest, but it changes an unknown into an operating fact. As the amount and importance of customer data grow, the acceptable recovery point and recovery time can become explicit service decisions. Before that, “we test the restore we would actually perform” is more valuable than an elaborate backup dashboard.

Let the Product Fail Without Lying

Some failures cannot be hidden or repaired before the customer notices. Graceful failure preserves a truthful next step.

If an evidence source is unavailable, the packet should not quietly omit it and present itself as complete. It can remain delayed, identify the blocked source, preserve the validated evidence, and name the next update time. If only two files are malformed, the import can accept the sound files and report the rejected ones without claiming full success. If an AI-assisted classification step lacks sufficient evidence, it can leave the item unclassified instead of fabricating confidence.

The best fallback narrows the promise while keeping the state honest. For Friday’s packet, the founder can assemble the output manually from the stored source manifest if the generator remains unavailable. That path is legitimate because it uses preserved inputs, records the resulting artifact, and produces the same visible completion state. It also records founder time, so the business cannot mistake manual fulfillment for scalable software.

A fallback becomes dishonest when customers cannot tell what they are buying, founder labor makes product metrics look healthier than they are, or the hidden work creates a commitment one person cannot sustain. Manual work may protect a young product. Invisible labor must not protect the founder from learning what the product costs to deliver.

Status communication follows the same rule. Tell affected customers what failed, what was and was not affected, what has been recovered, whether they must act, and when the next update will arrive. Do not promise a permanent fix before the evidence supports one. Reliability includes removing uncertainty the system created.

Stop at the Rung the Workflow Has Earned

After the incident, the founder could add a second region, automated failover, synthetic packet generation, a new observability platform, and a complex queue cluster. None addresses the weakness that caused this failure as directly as explicit job state, an abandoned-work timeout, a safe retry, and a delivery-window alert.

Those controls are the next rung. They make the current promise visible and recoverable without committing the product to infrastructure its usage has not justified.

Hardening should move ahead of feature work when failure blocks activation or repeat value, involves money, access, privacy, important data, or deadlines, repeatedly creates manual recovery, or prevents the founder from knowing whether customers value the product. Stronger isolation, redundancy, automated failover, stricter release controls, and synthetic checks become reasonable as the same workflow gains dependence, volume, tighter recovery expectations, or costly repeated failures.

Wait when demand remains unproven, failure is visible and cheaply reversible, a rare manual fallback is safe and informative, the workflow is likely to be redesigned, or the new infrastructure would create more operational burden than the failure it contains. A system the founder cannot understand, test, or repair is not made reliable by having more components.

The incident review should produce one decision, not a catalogue of possible improvements. Reconstruct the sequence from recorded state. Identify the earliest point where a different control would have reduced customer harm or founder effort. Choose that control, assign an owner—still the founder—and decide how its effect will be observed. Repeated incidents may justify climbing again. A one-off low-impact failure may justify only a defect fix.

Write the Reliability Brief

Before inviting more users into an important workflow, capture one page:

• Promise: what the user expects, including any cadence or deadline.
• Worst credible failure: the concrete loss of work, money, access, privacy, time, reputation, trust, or product evidence.
• Safe states: the few states needed to know what happened and what may happen next.
• Visibility: how failure is found before avoidable harm grows, and which conditions truly interrupt the founder.
• Recovery: the retry, restore, rollback, correction, disablement, or manual completion path, with its stop condition.
• Data protection: what is backed up and the last restoration that proved it usable.
• Customer truth: what users see during failure, whether they must act, and when they hear an update.
• Current rung: the controls justified now.
• Deferred machinery: what is deliberately not being built, and what evidence would change that decision.

Write this brief for the product’s most trust-sensitive workflow, then rehearse one failure. Start with the state the product actually records, not the state the founder hopes it records. If the rehearsal ends in an ambiguous retry, an untested restore, a silent delay, or a private production edit, add the smallest control that removes that uncertainty.

The work stops when the promise is observable, recoverable, and explainable at its present risk—not when the infrastructure looks complete. That boundary protects more than uptime. It protects the evidence the founder is about to use to decide whether the product has found a market at all.