SLOs, SLIs, and Error Budgets for Data Systems

Uptime Is Not the Promise

A database can be healthy while the data system is breaking the promise users actually rely on. The primary accepts connections, but checkout writes time out because the service pool is exhausted. Reads return quickly, but from a replica that is minutes behind. The event log is available, but the consumer that grants entitlements has stopped. The search cluster answers queries, but newly published products are invisible. The backup job reports success, but no one has restored the dataset since the last schema migration.

Those failures are not edge cases around reliability. They are reliability failures. The user did not ask whether a process was running. They asked whether an order could be placed, a balance could be trusted, a report reflected recent facts, a delete had propagated, or a team could recover before the business was damaged.

Service-level indicators, service-level objectives, and error budgets are useful only when they express that kind of promise. An SLI measures evidence of the promise. An SLO states how much of that behavior must hold over a decision window. Alerts, dashboards, runbooks, and reviews connect the measurement to action. Error budgets decide whether the team can keep changing the system at the same pace or must repair the reliability foundation first.

The test is whether the same evidence would convince a user, an operator, and a business owner that the data system can be depended on.

Measure the Promise Boundary

The first mistake is measuring the component instead of the promise. Database process uptime, cluster health, CPU, disk, and connection count are useful diagnostic signals, but they are not the same as user-visible reliability. A component can be green while the workflow is red.

Start at the promise boundary. For a checkout flow, the promise may be that valid orders are durably accepted and visible to fulfillment within a short window. For a search index, the promise may be that accepted catalog updates become searchable and deletes disappear within a bounded time. For a billing ledger, the promise may be that every accepted financial fact is stored once, reconciles to control totals, and can be restored.

This boundary matters because data systems often fail between components. A write succeeds in the primary store but does not publish an event. A consumer processes the event but cannot update a derived table. A cache refreshes after a delete and reintroduces stale data. A dashboard query succeeds while the upstream warehouse model is four hours late. A replica route meets latency targets by serving data too old for the decision being made.

Good SLIs therefore describe the behavior a user or operator would recognize, not merely the status a vendor console can observe. Component metrics help explain why an SLI is burning. They should not replace the SLI.

The Six Reliability Signals for Data Systems

Most production data-system promises draw from six signal families: availability, latency, freshness, correctness, durability, and recovery readiness. A critical workflow should be reviewed against all six, although it may need formal objectives for only some of them. Together they ask whether work can complete in time, whether visible facts are current and trustworthy, whether accepted facts survive, and whether the team can recover them within the promised window.

Availability should be counted at the workflow boundary. If an API returns success before the write is durable, the SLI should not count that as success unless the business promise explicitly allows asynchronous acceptance. If a timeout leaves the client unsure whether a write happened, count and review unknown outcomes separately; they often hide duplicate side effects and support pain.

Latency should include the path the caller waits on. Query execution time is only one part of the experience. Pool wait, lock wait, transaction retry, cache miss, network hop, search refresh, and fan-out to derived stores may all dominate. Percentiles matter more than averages because data systems often fail in the tail: one tenant, route, partition, query shape, or hot key can be much worse than the median.

Freshness should be measured as age of visible facts, not as a vague “pipeline healthy” status. Useful freshness SLIs include source-to-index lag, warehouse model watermark age, materialized-view staleness, cache invalidation lag, replica replay delay for routed reads, age of oldest unprocessed event, and age of feature values used for decisions.

Correctness SLIs need checks the team is prepared to act on. Control totals, invariant monitors, duplicate-detection rates, missing-delete counts, idempotency violations, schema compatibility failures, and reconciliation mismatches can all be reliability signals. A correctness alert without a repair path becomes noise; a correctness check tied to an owner and runbook becomes a trust mechanism.

Durability and recovery readiness keep the SLO discussion honest. A system that accepts important facts but cannot prove they survived, cannot restore them, or cannot rebuild derived copies is not reliable in the sense users need. The recovery plan owns the detailed procedure; the SLO layer records whether that procedure is still operationally true.

From Journey to SLO

An SLO should be precise enough that two reviewers can calculate whether it passed. That requires a named journey, a population, a measurement source, a target, a time window, and narrow exclusions.

A weak objective says, “Search should be fast and fresh.” A usable objective says, “For published catalog updates from active sellers, 99 percent become visible in customer search within five minutes over a rolling 28-day window, excluding updates rejected by validation before commit.” That statement names the user-visible behavior, the target, the window, the population, and the exclusion.

Another weak objective says, “The database should have 99.99 percent uptime.” A stronger checkout objective says, “99.95 percent of valid checkout order-write attempts either commit durably and return a success response or return a clear non-commit failure within two seconds over a rolling 28-day window.” This separates known failure from unknown state, which matters for retries, duplicate orders, and support handling.

SLO windows should match decisions. A short burn-rate alert detects acute incidents. A longer window supports product and reliability trade-offs. Many important systems need both: a fast-burn page when a critical SLI is degrading quickly, and a monthly or 28-day budget review that decides whether to continue migrations, increase rollout pace, or reserve time for reliability work.

Exclusions should be rare and written before they are needed. Invalid user requests, rejected validation errors, test traffic, or explicitly degraded maintenance windows may be excluded when they are outside the promise. Dependency failures, tenant spikes, data migrations, and operational mistakes should not be excluded by habit; often they are exactly the conditions the reliability promise must cover.

Tier SLOs when the promise truly differs. A premium tenant, regulated ledger, user-facing write path, internal dashboard, and exploratory analytics model may justify different targets. Tiering should be documented as a product and operational choice, not discovered later through escalation pressure.

Error Budgets Turn Measurements into Decisions

An error budget is the amount of unreliability the SLO permits during its window. A 99.9 percent availability SLO allows about 0.1 percent of the measured population to fail in that window. The math is less important than the behavior it creates: when the budget burns too quickly, the team changes what it is doing.

For data systems, error budget burn can come from more than failed requests. A team can spend freshness budget while a pipeline is stuck, correctness budget while reconciliation drift grows, latency budget during hot-key contention, recovery-readiness budget when restore drills age out, and durability budget when backup verification repeatedly fails. Treating only uptime as budget burn encourages the team to ignore the failures that most damage trust.

Budget policy should be written in advance. While the budget is healthy, planned feature work, migrations, and capacity changes continue under normal monitoring. Early burn triggers a review of recent deploys, offending tenants or routes, and slow-burn reliability work. High burn freezes risky migrations and large rollouts unless an accountable owner accepts the risk. Once the budget is exhausted, reliability repair, incident follow-up, or an honest renegotiation of the promise comes before adding new risk.

The point is not to punish a team for incidents. The point is to regulate change rate with evidence. If every incident is treated as exceptional, the system never learns. If every budget burn becomes a blanket freeze, the policy will be bypassed. A useful error-budget policy names who decides, what work stops, what work continues, which risks can be accepted, and how the decision is revisited.

Data migrations deserve special attention. A migration that increases correctness drift, p99 latency, consumer lag, support tickets, or restore uncertainty should have stop conditions tied to SLIs. “Proceed unless someone complains” is not an operating policy. “Pause rollout when reconciliation mismatches exceed the SLO threshold or when freshness burn exceeds the agreed limit” is.

Alert on Symptoms, Diagnose with Causes

Alerting should follow the SLO stack. Page on fast-burning, user-impacting symptoms or imminent loss of a critical promise. Failed writes, unknown write outcomes, unacceptable p99 latency, stale user-visible data, correctness drift, missing deletes, restore failure, or lag that will soon violate a freshness promise can justify waking someone when the action is urgent.

Open tickets for slow-burn risks. A restore drill that is approaching its maximum age, a growing reconciliation backlog, repeated backup warnings, increasing cache invalidation lag, or a capacity trend may need ownership and planning rather than an immediate page.

Use dashboards for diagnosis and review. A useful dashboard answers: which promise is burning, which population is affected, when it started, what changed recently, which dependencies are unhealthy, how much budget remains, who owns the runbook, and whether users are still exposed. CPU, disk, locks, queue depth, replica lag, cache hit rate, and deploy markers belong here because they explain the symptom.

Runbooks should be short enough to use under stress and specific enough to prevent improvisation. They should say how to confirm impact, identify scope, stop the bleeding, preserve evidence, mitigate safely, verify recovery, communicate, and escalate. If a runbook has not been used in a drill or incident, treat it as a hypothesis.

Every alert also needs a retirement path. If an alert never changes action, tune it, demote it to a ticket or dashboard, or remove it. Alert fatigue is not an on-call personality problem; it is usually a measurement design problem.

Reliability Reviews

SLOs become useful when they are reviewed by people who can make trade-offs. A reliability review should not be a tour of dashboards. It should produce decisions about engineering work, rollout pace, ownership, targets, and accepted risk.

Review burn by workflow, tenant, route, region, query fingerprint, data class, job, dependency, and deploy type. The distribution often matters more than the total. A global availability SLO can look healthy while one high-value tenant or one checkout path is failing. A freshness SLO can look acceptable in aggregate while deletes lag longer than creates, which may be unacceptable for privacy or trust.

Compare the burn to the planned investment. If most damage comes from correctness drift, adding read replicas may not help. If most damage comes from lock contention on a hot entity, freshness work may be secondary. If restore-readiness budget keeps burning because drills fail access or dependency checks, the recovery program needs attention before the team can claim durable reliability.

Review the promise itself. Sometimes the system is worse than the promise and needs repair. Sometimes the promise is stricter than the business needs and causes waste. Sometimes the workload has changed enough that an old target is no longer defensible. A mature SLO practice can tighten, loosen, split, or retire objectives when evidence supports the change.

Data-System SLO Spec

Use this spec for each important data-system promise. It should be short enough to review and exact enough to operate:

• Name the user journey, decision, tenant class, or operator action that depends on the data.
• State what must be available, fast, fresh, correct, durable, or recoverable.
• Define the SLI: its numerator and denominator, percentile, freshness age, reconciliation result, or drill evidence.
• Point to the measurement source: the metric query, log event, trace, reconciliation job, backup record, or restore evidence.
• Set the target, rolling or calendar window, measured population, and narrow exclusions.
• Separate page, ticket, and dashboard behavior by urgency and actionability.
• Link a runbook that covers confirmation, scope, mitigation, verification, communication, and escalation.
• Name the team accountable for measurement quality, review cadence, and budget decisions.
• Write down what happens during early burn, high burn, exhausted budget, and an accepted-risk override.

Specificity changes the review:

• For customer search, measure source-to-index freshness for creates, updates, and deletes. Require 99 percent of committed catalog changes to become visible within five minutes over 28 days, and track deletes separately.
• For a billing ledger, measure reconciliation mismatches across ledger entries, statements, and settlement records. Permit no unresolved high-severity mismatch older than one business day.
• For checkout writes, require 99.95 percent of valid attempts to finish with either durable success or a clear non-commit failure within two seconds over 28 days.
• For a critical dataset, require a passing restore drill within its cadence and a measured restore duration inside the RTO.

Exercises

Define SLOs for a search index. Include query availability, p99 query latency, source-to-index freshness, delete propagation, rebuild readiness, and the alerting split between page, ticket, and dashboard. Treat creates, updates, and deletes separately; a system that freshens creates quickly but leaves deleted records visible has not kept the same promise.

Define SLOs for a billing ledger. Include durable write acceptance, unknown write outcomes, reconciliation mismatch age, statement correctness, backup or restore evidence, and the runbook for repair. Decide which signal wakes someone, which blocks a migration, and which becomes a reliability review item.

Audit one existing dashboard. For each chart, label it as SLI, diagnostic cause, capacity signal, or unused. If a chart is not tied to an SLO, runbook, review decision, or capacity plan, either give it a job or remove it from the primary operating view.

Production Review Questions

A data-system SLO is ready for production review when it can answer five questions without interpretation:

1. Which user or operator promise is being measured?
2. What exact evidence proves success, failure, freshness, correctness, durability, or recovery readiness?
3. What target and window decide whether the promise was kept?
4. What action happens when the budget burns quickly or repeatedly?
5. Who owns the measurement, runbook, review, and risk decision?

If those answers are missing, the team may still have useful metrics. It does not yet have an operating promise.