The Decision Room: A Repeatable Database Selection Process

A Product Name Is Not a Decision

Consider this architecture decision record:

Title: Move notification data to a managed key-value store

Context:
The notifications service is growing and the current relational database
will not scale. Other teams use a managed key-value store successfully.

Decision:
Move notification preferences and delivery history to the new store.

Consequences:
The service will scale better. We need to migrate the data.

The proposal may be right. The record gives nobody a way to tell. “Notification data” hides two different workloads. “Will not scale” hides the failing resource. Other teams’ success says little about this team’s access paths, invariants, recovery needs, or operating capacity. Even the consequence is only the decision repeated in future tense.

A durable data choice needs a room. It may be an architecture meeting or an asynchronous review, but it must make the same evidence visible to everyone and leave behind reasoning that an operator can inspect later. The workload owner explains what users are trying to do. The data owner identifies truth, access paths, and migration boundaries. Reliability brings restore and on-call consequences into the design. Security, privacy, and governance bring access, deletion, retention, residency, and audit obligations. In a small team, one person may carry several of these responsibilities; none of the responsibilities may disappear.

The room exists to answer a harder question than “Which database do we like?” It asks what changed, what the system must preserve, which uncertainty can overturn the choice, and who can operate the result.

First Decide Whether a Database Decision Exists

The notification team has a real production pain. Campaigns insert delivery history fast enough to create replica lag, and support queries over that history make the lag worse. Preference reads share the same primary. During a campaign, the database work that records what was sent can delay the path that decides whether a message may be sent.

That is a diagnosis worth bringing to the room. It still does not prove that the team needs a new system class.

Many apparent database decisions are narrower repairs. A missing index, an unbounded retention policy, a reporting query on the primary, an oversized connection pool, a stale contract, or an untested restore procedure can make the platform feel wrong while leaving its basic fit intact. Opening a selection process creates research, migration, operational, and decommissioning work. Spend that effort when a new capability, a measured saturation point, a trust requirement, or a change in ownership creates long-lived operational gravity.

The opening question is therefore precise: what changed, and why is tuning or repair no longer an adequate response?

For notifications, the answer separates the data before comparing stores. Preferences are small mutable records read on every send decision. Delivery history is append-heavy and queried by user, campaign, and time. Treating both as one object called “notification data” caused the original proposal to skip the most important design choice: where the boundary belongs.

Gate 1: Make the Workload Testable

The room begins with facts about production behavior. Data shape and access paths matter, but so do average and peak load, tenant skew, payload size, latency and freshness targets, retention, deletion, replay, export, recovery, privacy, and cost.

Suppose the evidence packet says this:

• Preferences are read by user and category inside the send path. They change infrequently, must return within 30 milliseconds at p99, and remain until account deletion.
• Delivery history peaks at 12,000 append operations per second for twenty minutes during large campaigns. Support reads it by user and time with a 500-millisecond target. It is retained for ninety days and included in privacy exports.
• Campaign traffic is sharply skewed: a few tenants create most of the burst. Average throughput is therefore a misleading test load.
• The present failure is replica lag during peak history ingestion, not slow preference lookup or exhausted primary storage.

These numbers may begin as estimates. They still create a claim that can be measured and revised. “Low latency” and “high scale” do not.

The gate passes when a candidate can be tested against the workload without relying on adjectives.

Gate 2: Put the Invariants Ahead of the Access Path

The strongest notification invariant is simple to say and expensive to violate: a user’s opt-out must be honored before any non-mandatory message is sent. Preference state therefore needs an enforcement location, a failure policy, a signal that exposes violations, and a repair path.

Delivery history carries different promises. A retry may produce the same event twice, but an idempotency key must make the duplicate detectable. A support view may be several minutes stale if its visible freshness agrees with that promise. Deletion must reach the authoritative record, the support projection, exports, and every other retained copy according to policy.

This gate catches a common sleight of hand. A fast key lookup or append path can look attractive because correctness work has quietly moved into application code, caches, reconciliation jobs, and human response. That may be a sound design, but the moved obligation belongs in the decision.

The gate passes when each critical fact is classified: enforced synchronously, allowed to be stale within a bound, repaired after detection, or merely advisory. The design must say where enforcement, detection, and repair happen.

Gate 3: Give the Current Platform a Fair Trial

The existing system is neither entitled to remain nor guilty by default. Compare a measured and reasonably maintained current platform with credible alternatives.

The notification team first partitions delivery history by month, removes unnecessary secondary indexes, batches inserts, and moves long-range support analysis away from the primary. Partitioning makes retention cheaper, and batching reduces write overhead, but campaign ingest still produces enough replica lag to endanger the send path. Preference storage, by contrast, continues to benefit from relational constraints, auditable changes, and flexible administrative queries.

This result narrows the decision. The team has evidence for moving one write-heavy history path; it has no evidence for moving preference truth. Had partitioning and query isolation met the targets, “repair the current system” would have been a successful decision-room outcome.

The gate passes when the record names what was measured, tried, rejected, and left available. A new store should not inherit an undiagnosed retention or ownership problem under a different product name.

Gate 4: Compare Candidate Classes Through the Same Workload

Generate candidates by system class before choosing products. For this workload, credible choices include keeping partitioned history in the relational system; adding an append-oriented event store; using a durable log with a derived support view; or archiving retained events to object storage behind a serving projection. Moving preferences to a key-value or document store is also testable, but it must justify recreating the current correctness and audit properties.

Every candidate receives the same questions. Can it absorb tenant-skewed bursts with the required indexes? How does it enforce or preserve idempotency? What happens to a support query during backpressure? Can retained source data rebuild the serving view inside the recovery window? How does deletion propagate? What does a backfill cost? Which part becomes authoritative, and which parts are derived?

Managed and self-operated variants belong in this comparison. A managed service may reduce patching and routine failover work, but the team still owns modeling, quotas, access control, restore tests, migrations, incident response, and cost. Self-operation buys useful control only when the team can exercise it during an incident.

Hybrid designs are often the honest result. They are not free compromises. Each added store creates a movement path, a freshness promise, a drift detector, a rebuild procedure, a deletion path, an owner, and eventually a decommissioning problem.

The gate passes when the familiar option, the favored option, and the fashionable option face the same production workload and operating criteria.

Prototype the Uncertainty, Not the Happy Path

A prototype is an experiment against the fact most likely to reverse the decision. Building a miniature version of every component usually avoids that fact.

The notification team already knows that an append-oriented store can accept an event. Its uncertainty is whether the whole path survives a campaign-shaped burst without losing idempotency, hiding backpressure, or making the support projection unrecoverable. The prototype therefore uses production-like payloads and tenant skew, exercises retries and throttling, and measures p99 write latency, error rate, lag, and cost. It also rebuilds the support view from retained events inside the recovery window.

Correctness uncertainty calls for races, duplicate messages, stale reads, partial writes, out-of-order delivery, and disagreement during cutover. Migration uncertainty calls for a real backfill, reconciliation queries, stop conditions, and rollback while the two systems disagree. Operational uncertainty calls for the unpleasant action: restore a backup, replay a partition, rotate credentials, delete one user’s data, or hand the runbook to an engineer who did not build the prototype.

Write the success and stop criteria first. A result without a prior threshold is easy to explain away after a team has fallen in love with the candidate.

Gate 5: Make Operations Part of the Choice

A benchmark can admit a candidate to production review. It cannot make that candidate operable.

For the notification split, the room needs named ownership for the preference tables, event producer, history store, and support view. It needs backup and restore procedures for preference truth, replay evidence for derived history, and an agreed recovery point and recovery time. Dashboards must expose preference-read latency, rejected or missing history writes, consumer lag, support-view freshness, throttling, retention failures, and the cost drivers created by campaigns.

Security and privacy are paths through the whole design. The review follows access grants, audit records, encryption boundaries, deletion propagation, export behavior, retention, and legal holds through source and derived stores. Migration planning covers backfill, dual operation, verification, cutover, reconciliation, rollback, and retirement of the old write path.

The useful test is operationally plain: can an on-call engineer diagnose the top failure and perform the first recovery action from the runbook without the original designer present? If restore, deletion, freshness, saturation, or cost remains somebody else’s future work, the candidate has not passed.

The Decision Is a Boundary

The evidence now supports a narrower ADR than the original proposal:

• Keep notification preferences in the relational database as the source of truth. The send path reads that authority for opt-out decisions.
• Move delivery history to an append-oriented event store with user-time and campaign-time access paths. Treat duplicate delivery events as detectable through an idempotency key.
• Build the support view as a derived projection with a five-minute freshness target and a visible lag signal. Rebuild it from retained events.
• Keep the old history path available for low-volume rollback during a bounded migration window. Verify both paths before cutover, then decommission the old one deliberately.
• Reopen the decision if freshness misses its target for two consecutive weeks, projected cost exceeds budget by 30 percent, deletion cannot meet policy, or new query shapes defeat the chosen access paths.

The product names can now be evaluated without carrying the whole argument. The design is about authority, derived state, and movement between them. That boundary explains why preferences remain relational, why delivery history may move, which failure is tolerable, and what evidence would make the split wrong.

Try to Break the Decision Before Approval

Assume that six months after launch, customers receive notifications after opting out. Perhaps a cache became authoritative during an outage, or the send path failed open when preference reads timed out. That story should produce a control, a signal, and a recovery action: an explicit failure policy, monitoring for stale or bypassed preference reads, and a way to pause non-mandatory sends.

Assume support cannot see recent deliveries during an incident. The review should add a freshness alert, a fallback query or status message, and a rehearsed replay. Assume a campaign doubles the bill, a deletion misses an export bucket, rollback cannot translate new writes, or the on-call dashboard shows green infrastructure while user-visible lag grows. Each credible failure should alter the ADR through a test, guardrail, alert, runbook, cost limit, or reversal trigger. Otherwise the pre-mortem is theatre.

A red-team review challenges the strongest assumption rather than inventing many shallow objections. A runbook review makes another operator perform the recovery path. A cost review models steady state, bursts, backfills, retention growth, read and write amplification, egress, and operator time. These rituals earn their place only when they can change the decision.

The room may approve the proposal, narrow it to a specific prototype, repair the current system, or stop because the workload, invariant, owner, or operating plan remains unclear. Approval is not the privileged outcome. Stopping a vague migration is useful engineering work.

Write the Data System ADR While the Choice Is Alive

The record should preserve reasoning that a future engineer cannot recover from a topology diagram. A practical Data System ADR covers six movements:

1. Trigger and workload. Name what changed, then record data shape, reads, writes, latency, growth, skew, retention, deletion, freshness, consistency, privacy, recovery, and cost constraints.
2. Invariants and current diagnosis. State which facts need synchronous enforcement, bounded staleness, detection and repair, or advisory treatment. Record what is actually failing in the current system and which repairs were tried.
3. Candidates and decision. Compare system classes before products. Identify the source of truth, derived stores, access paths, owners, rejected alternatives, and the consequence each boundary creates.
4. Evidence and failure analysis. Record prototype methods, results, limits, pre-mortem failures, controls, tests, alerts, and unresolved risk. A benchmark number without its data, load shape, and success threshold is not evidence another team can reuse.
5. Operation and migration. Cover backup, restore, monitoring, incident response, access, deletion, retention, cost, backfill, dual operation, verification, cutover, rollback, reconciliation, and decommissioning.
6. Assumptions and reversal. Name the workload, growth, team, vendor, compliance, and cost assumptions the choice depends on. Give each important reversal trigger an owner and a practical exit or scope-reduction path.

Write the ADR before approval, while evidence can still defeat preference. A record written after implementation is usually a memorial.

Put the Room to Work

Rewrite an ADR that says, “We chose this database because it scales.” Begin with the workload fingerprint and invariant map. Diagnose the current platform, compare at least three candidate classes against the same evidence, and name the prototype whose result could reject your favored choice. End with owners and reversal triggers.

Then run a pre-mortem for a proposed migration. Choose five failures with production consequences. For each, identify one prevention or containment measure, one detection signal, and the first recovery action. Revise the ADR with the useful findings; do not leave them in a separate workshop document.

Finally, bring a painful workload from a system you know to the five gates. Decide whether it warrants selection work at all. “Repair the current system” and “stop until the requirement is clear” are valid conclusions.

The chapters that follow compare relational, key-value, document, wide-column, search, streaming, analytical, graph, time-series, and vector systems. The decision room supplies their common discipline. Begin with the workload and the facts it must preserve; let a product earn its name late.