SLOs and Operational Readiness

The service is healthy; the password reset is not

A team is preparing a notifications platform for a broad launch. It sends password resets, security alerts, purchase receipts, and marketing messages over email, SMS, and push. The canary is technically quiet. Instances are healthy, queues are mostly empty, and the dashboard reports 99.95 percent delivery success.

Yet support has three reports of password-reset codes arriving too late to use. All three traveled through the same SMS provider route. Marketing traffic is far more numerous than password resets, so its successful deliveries make the aggregate look excellent.

The release system has done its job: a bounded cohort exposed a condition that tests and staging missed. The next decision belongs to the service’s operating model. Does the team ramp, pause, or change the scope? “The dashboard is green” cannot answer because the dashboard has averaged away the promise that matters.

Operational readiness begins when a team can name that promise, measure it from the user’s side, and let the result change what happens next.

Separate the promises before choosing the metric

“Notifications are reliable” hides several different obligations. A password reset must arrive before its code expires. A security alert must not disappear silently. A purchase receipt may tolerate more delay, but a duplicate can alarm the customer and burden support. A marketing message has a weaker delivery deadline, while suppression and unsubscribe correctness are strict: reliability sometimes means not sending.

One availability percentage cannot express those differences. The team first names the critical journeys and the harm attached to failure. That act is not metric design yet. It is product judgment.

For the canary, password-reset delivery is the promise under pressure. Its SLI must define a population and an outcome, not merely name a chart:

eligible events: valid password-reset requests accepted by the platform
good events:     reset messages delivered within the usable-code window
slices:          channel, provider route, region, and app version
diagnostics:     queue age, provider response, retry count, template failure

The distinction between the SLI and its diagnostics is important. Queue depth, CPU, worker count, and provider status can explain a failure, but none proves that a customer received a usable reset. Conversely, a delivery receipt may be the closest available observation rather than perfect end-to-end truth. The team should say what the measurement can see, how late or incomplete its receipts are, and which customer or support signals help close that gap.

Good-event accounting also forces awkward questions early. Does a message count as delivered when the provider accepts it or when the handset reports delivery? How are duplicate reset requests treated? What happens when the provider never returns a terminal status? Which synthetic and abusive requests are removed from the population?

Exclusions deserve particular suspicion. A downstream provider outage still hurts the user, so excluding it from the user-facing objective can turn an honest dependency failure into a flattering number. Track provider-caused failures separately for diagnosis and contracts, but keep them in the user view unless the service promise genuinely says otherwise. Any exclusion should be narrow, auditable, and incapable of appearing only when the number looks bad.

The original aggregate remains useful as a broad system signal. It simply no longer gets to overrule the password-reset slice.

Set an objective that governs behavior

An SLO gives the SLI a target and a time window. Suppose observed customer needs and historical performance justify the following modeled objective:

Over a rolling 30-day window, 99.9% of eligible password-reset messages are
delivered within two minutes, measured separately by channel and reviewed by
provider route and region.

The numbers are not universal recommendations. A team should derive them from the time a user can reasonably wait, the expiry and retry behavior of the product, contractual commitments, traffic volume, and performance the system can sustain. Writing 99.99 because it looks serious is no more disciplined than choosing 95 because it is easy to meet.

The wording does useful work. It identifies the event, threshold, population, window, and slices. It also makes the remaining ambiguity visible. If a code expires in ten minutes, a two-minute delivery target preserves time for the customer to retrieve and use it; the product team can argue about that choice with evidence rather than debating a generic word such as “available.”

A 100 percent target is rarely an honest operating objective. It leaves no allowance for controlled change or unavoidable failure, and teams often respond by inventing exceptions or ceasing to trust the objective. The target should protect users while leaving an explicit, small tolerance for failure. That tolerance is the error budget.

For an event-based 99.9 percent objective, the allowed bad-event fraction is 0.1 percent. If one million eligible reset messages occur in the window, the budget permits one thousand bad events. At low traffic, however, a handful of events can swing the percentage sharply. The team may need minimum sample rules, longer decision windows, direct treatment of every critical failure, or a combination. Arithmetic does not remove judgment; it reveals where judgment is still required.

Turn the budget into a policy before it burns

The error budget becomes valuable only when people know how behavior changes. The notifications team agrees on a policy before broad launch:

• With ample budget and stable slices, normal release controls apply and the team continues planned reliability work.
• When failures consume the budget unusually quickly, the on-call engineer mitigates current harm and the release owner pauses exposure.
• As the remaining budget becomes small, the service owner reduces risky change, prioritizes the failure modes responsible for the burn, and reviews launch commitments with product and support.
• If the budget is exhausted, non-essential risky launches stop until the team restores confidence or an accountable owner explicitly accepts a bounded exception.

This is an agreement about risk, not a punishment system. A budget policy allows product, engineering, support, and on-call owners to make the same trade-off from the same evidence. It also prevents a monthly number from becoming an excuse to spend every remaining failure. Reliability is not a coupon that must be redeemed.

Burn rate supplies urgency. It compares the observed rate of bad events with the rate the objective allows. Under a 99.9 percent objective, a sustained one percent bad-event rate consumes budget at roughly ten times the permitted rate. A fast burn can justify immediate action long before the 30-day budget is empty. A slow, persistent burn may need investigation and planned repair without waking someone at 03:00.

For the canary, the global notification objective is healthy while the password-reset SMS slice burns rapidly. The slice is not a second-class chart; it reveals concentrated user harm. The team pauses the affected route without waiting for the aggregate to fail.

Page on a decision, not on curiosity

An alert spends a person’s attention. It should do so only when the recipient can take a useful action with appropriate urgency. Dashboards can retain trends, exploratory signals, and diagnostic detail without demanding an immediate interruption.

The password-reset page should tell the on-call engineer what is happening and where to begin:

Password-reset delivery is consuming its SLO budget rapidly for SMS traffic
on provider route B. Queue age is rising; email delivery remains within its
objective. The current release is at 20% exposure.

First actions: pause ramp, inspect provider quota and retry age, preserve the
affected message IDs, and follow the provider-degradation runbook.
Escalate to: notifications owner and provider contact.

The page is based on the user-visible symptom. Queue age and quota state help diagnose it. A separate CPU alert would add noise unless CPU saturation itself requires an immediate, owned action.

A useful dashboard answers the next questions rather than trying to display everything the platform emits. For this launch it shows delivery success and age by message class and channel, the affected provider and region slices, queue and retry age, template errors, provider quotas, duplicate suppression, deploy and configuration markers, error-budget burn, and support reports. Together these views support three decisions: whether to ramp, whether to return new messages to the legacy path, and whether degraded service is recovering.

Alerts also need deduplication and an end condition. Repeated pages for one unresolved condition train responders to ignore the system. A recovery signal should not resolve the event merely because one short window turned green while old messages remain queued.

A runbook is a rehearsed recovery path

The alert links to a runbook, but the existence of a document is weak evidence. Readiness depends on whether someone other than its author can use it under pressure and whether the proposed action is still safe.

When provider route B throttles requests, the runbook should help the responder answer concrete questions. Can critical messages move to another provider without violating regional, privacy, or sender-identity constraints? Does the alternate route have quota and capacity for both fresh traffic and retries? Will retrying create duplicate codes or messages? Can marketing traffic be paused so that password resets and security alerts retain capacity? How is a backlog drained without creating a second surge? Which message IDs and state transitions must be preserved for reconciliation and support?

The team rehearses the path with side effects controlled. The first exercise reveals that failover changes a provider identifier used by duplicate suppression. That is exactly the kind of discovery a readiness review should produce. The team repairs the identifier mapping, repeats the exercise, and measures how the alternate route behaves under retry load.

Capacity evidence belongs here, close to the failure path. A load test at steady expected traffic says little about recovery from a provider outage. The dangerous moment may arrive when the provider returns and fresh traffic, retries, and backlog drain compete at once. The team therefore measures quota headroom, saturation, maximum queue age, retry amplification, and drain time. It defines which lower-priority messages can be delayed or shed to protect the critical journey.

Backup and failover claims require the same honesty. An alternate route is not capacity until it has been exercised. A rollback is not safe until state and side effects have been considered. A restore procedure is not recovery evidence until a restore has been tested.

Let readiness change the launch

The readiness review returns to the question that opened the chapter. The platform runs, but should it take all notification traffic next week?

The review can now make an evidence-based decision. Ownership and escalation are clear. Password-reset and security-alert promises have distinct SLIs and budget policies. The dashboard exposes message class, channel, provider, and region rather than only a global average. The fast-burn page reaches an owner and names a first action. Provider failover has been exercised, including duplicate suppression. Retry and backlog behavior have been measured under the alternate route.

One weakness remains: route B’s quota has too little headroom for peak reset traffic plus backlog recovery, and the provider has not approved an increase. The right outcome is neither “everything is blocked” nor “the checklist is mostly green.” The team keeps password-reset SMS on the proven route, pauses marketing SMS during peaks, launches email receipts on the new platform, and holds broader SMS exposure until quota and recovery evidence meet the stop condition. Product and support receive the scope, reason, owner, and next review date.

That is operational readiness doing work. It did not promise that the service would never fail. It found where failure would concentrate, established how the team would see and contain it, and reduced launch scope before customers paid for an unsupported assumption.

For another service, the evidence will differ, but the questions persist:

• Who owns the service and has authority to pause or reduce scope?
• Which user journeys define health, and what measurements can actually see their outcomes?
• What target and window produce sensible product and engineering behavior?
• Which slices prevent a harmed cohort from disappearing in an average?
• What budget state changes release, reliability, and on-call decisions?
• Which conditions page a person, and what can that person do?
• Have rollback, degradation, failover, data repair, and escalation paths been exercised where they apply?
• What happens at peak traffic, during dependency failure, and while a backlog is recovering?
• Can support explain the impact and route affected customers?
• What evidence can stop, stage, or narrow the launch?

The questions form a launch gate because a missing answer changes the decision. They are not a ceremonial demand for ten documents.

Practice the decision, not the vocabulary

In a system design interview, attach SLO reasoning to the riskiest user journey rather than reciting availability terms at the end. In a production engineering discussion, connect the objective to alerting, on-call ownership, capacity, and the policy that pauses a rollout. In a project deep dive, show the evidence that changed a launch or reliability priority. The useful signal is the decision you can defend, not the number of operational artifacts you can name.

Try the notifications case with one fact changed. The provider quota is now ample, but delivery receipts arrive six hours late. Or failover works, but it cannot preserve regional data boundaries. Or reset delivery is healthy while duplicate security alerts rise. For the chosen variation, write only five things: the user promise, the observable SLI and its blind spot, the policy trigger, the safest immediate action, and the evidence required for the next launch decision.

The exercise should make one limitation uncomfortable: measurement cannot substitute for ownership. Someone must decide which promise deserves protection, which uncertainty is tolerable, and what scope the evidence can support. If the launch proceeds and the service later fails, those same promises, alerts, owners, and runbooks become the starting state for incident response—not a collection of documents the responders discover during a bad hour.

Related links

• The Software Delivery Lifecycle
• CI/CD and Release Engineering
• Incident Response
• Performance Engineering
• Reliability and Failure Engineering
• Observability and Operability