The Real Shape of a Production Data System

The Product Name Is Not the Architecture

The weak version of a data-system design review begins with a product name and treats the rest as implementation detail. “Orders will be in Postgres.” “Search will use Elasticsearch.” “Events will go through Kafka.” “Analytics will land in the warehouse.” Each statement may be true. None describes what happens to a fact after the system accepts it.

A production data system is the arrangement that preserves useful facts after users, software, machines, operators, and time have acted on them. It includes the API that accepts commands, the code that enforces or forgets business rules, the primary store, the queue that delays work, the cache that serves old answers, the export that finance trusts, the backup that may or may not restore, the dashboard opened at 03:00, and the people allowed to change or repair any of it.

The named database is one component inside that boundary. Until the facts, flows, possible disagreements, recovery paths, and owners are visible, a technology comparison is mostly theater.

Follow One Order Past the Database

Consider a checkout service. The quick design says that orders live in a relational database. A more useful design begins with the claim the system makes to a customer: we accepted your order.

The checkout API receives cart contents, a payment intent, a shipping address, promotions, and an idempotency key. Application code validates the cart. A transaction records the order, a pending payment attempt, and the idempotency result. Already, the database product is not the whole design: code decides what may be written, and the transaction boundary decides which claims can become true together.

Then the order leaves the immediate write path. A payment provider sends webhooks, sometimes twice and sometimes after the browser has gone away. A queue carries fulfillment work. A search index lets support find the order. A cache supplies the customer’s account summary. A warehouse table feeds revenue reporting. An email system sends a confirmation. Backups preserve some earlier state. Dashboards and reconciliation jobs tell operators where the parts disagree.

One accepted order has become several representations and several promises. The orders database may own order state, while the payment provider remains authoritative for capture and the inventory service owns stock commitments. The support index is derived, but a support agent may rely on it during a customer call. The warehouse is derived, but finance may use it to recognize revenue. “Derived” describes lineage; it does not make the consequence disposable.

Now the boundary has a practical definition. It contains every component and responsibility that can change, preserve, copy, expose, lose, corrupt, leak, or repair a fact the business depends on. It need not contain every library or machine that touches the service. It must contain the surfaces whose behavior can change what a user, operator, or business process is entitled to believe.

Draw the Edges Where Consequences Change

A database box hides several different boundaries. Drawing them separately reveals the contracts the system has to keep.

The transactional edge shows what commits together. In the order service, the order row and idempotency record may commit in one transaction while fulfillment is published later. If the transaction commits and publication fails, a durable outbox or reconciliation path must close the gap. A queue cannot process work it was never told about.

The consistency edge shows where observers may disagree. The account cache might lag for 60 seconds, the support index for five minutes, and the revenue table until the next daily load. Those windows are design choices only if they are bounded, visible, and repairable. Otherwise, “eventual consistency” is merely a label for unexplained behavior.

The ownership edge shows who may change or repair each part. Someone must own schema changes, restore drills, queue incidents, search rebuilds, access policy, and data quality. A component without an owner is still in the architecture; it is simply waiting to become an argument during an incident.

The recovery edge shows what can be restored and what must be replayed, rebuilt, or reconciled. Restoring the orders database does not restore the support index, drain an abandoned queue, or prove that captured payments match order state. A backup is useful evidence, but recovery exists only when the team can restore service around it and validate the resulting facts.

The security edge shows where a fact can be read, written, exported, deleted, or inferred. Sensitive order data may escape the primary store through logs, traces, support tools, analytics notebooks, search indexes, object-storage exports, backups, or local copies. Access control drawn only around the database leaves the rest of the system undesigned.

These edges overlap, but they answer different questions. A search index can sit outside the transaction while remaining inside the security and ownership boundaries. A payment provider can be outside the team’s infrastructure while remaining inside the recovery and reconciliation story. One rectangle cannot express those obligations honestly.

Production Makes the Map Move

In development, the arrangement can look static. Production gives it a history.

Traffic changes. A cache introduced as a harmless optimization becomes the only protection for a hot query. Indexes grow. A plan that worked with one month of data becomes painful with eighteen. Jobs fall behind. A queue that normally drains in seconds accumulates hours of work after a downstream outage.

Product rules change too. A nullable field added for a launch becomes mandatory for billing, support, and analytics. A spreadsheet export created for one audit becomes a weekly operating report. A support script begins as a rescue tool and becomes a parallel write path around application validation. Retention and deletion matter only after enough records have aged or the first deletion request arrives, but the earlier design determines whether the team can satisfy them.

People change. The engineer who understood an odd reconciliation rule leaves. Team boundaries move while credentials, dashboards, and runbooks remain where they were. Production architecture therefore includes memory: why a path exists, who watches it, how it is repaired, and how a new operator learns that responsibility before the pager forces the lesson.

This does not imply maximal architecture. The boundary sketch may show that one database is enough, that a cache adds more risk than it removes, or that a queue creates a recovery problem the workload does not justify. The discipline is not to add components. It is to make the consequences of each component visible.

Make a Boundary Sketch That Can Survive an Incident

Before comparing technologies, make a one-page boundary sketch. Begin with a few critical facts in business language, such as order accepted, payment captured, inventory reserved, and shipment requested. For each fact, record five things:

• where it is authoritative;
• every API, job, webhook, migration, admin tool, or repair path that can change it;
• every cache, index, queue, report, export, or downstream system that copies or acts on it;
• the allowed disagreement, including staleness, duplication, reordering, or missing updates;
• the signals, recovery steps, validation checks, and people responsible when the contract breaks.

For the order service, a compact sketch could say:

• Authority: the orders database owns order state; the payment provider owns external capture events until reconciliation; the inventory service owns stock commitments.
• Accepted change: checkout creates the order and idempotency record together; webhooks update payment state; the refund tool and reconciliation job are audited repair paths.
• Asynchronous work and copies: an outbox feeds fulfillment; the account cache may lag 60 seconds; support search may lag five minutes; revenue reporting is daily.
• Recovery proof: restore the database, replay the durable outbox, rebuild search, then compare paid orders with the provider’s settlement data.
• Operation: the orders on-call owns customer-impacting disagreement; the data team owns warehouse freshness; security owns export access; queue age, webhook retries, and paid-but-unfulfilled orders are visible signals.

The sketch is useful because it can be challenged. If checkout times out after commit, can the retry create a second order? If the outbox publisher stops, which signal ages first and who responds? If the payment webhook arrives before the handler can find the order, is the event durably retried or silently discarded? If search is rebuilt from a restored database, how does support know when it is complete? If an account is deleted, which exports and backups still contain its data?

A sketch that cannot answer every question is not a failure. The unanswered question is precisely what the review needed to expose. Add the missing path, assign the missing owner, or write down the risk before a product feature hides it again.

Read Failure as a Boundary Problem

Invisible boundaries produce recognizable failures. The language around them often gives the omission away.

A cache is “just an optimization” until stale entitlement data grants access. A queue is “just plumbing” until delayed work ships a canceled order. An export is “just reporting” until the business uses it to approve refunds. A log is “just observability” until it becomes the only audit trail. The word just often marks a surface whose consequences have not been designed.

Happy-path diagrams create the same blindness. Retries, duplicate events, dead letters, restore paths, rebuild steps, manual repairs, and operator signals are not incident details to add later. They determine whether the system preserves its claims when normal execution stops.

Derived stores deserve particular suspicion. They are often rebuildable in theory but essential in practice. A search index that takes twelve hours to rebuild is not the source of truth, yet support may be unable to operate without it. The contract must include rebuild time and degraded operation, not merely the word derived.

Primary-store recovery can also conceal failure. A restored database surrounded by abandoned queues, missing indexes, unreplayed events, stale exports, and unverified permissions may be technically available while the business remains unable to operate. Recovery ends when the system’s important claims have been reconciled and validated, not when one process starts accepting connections.

Practice: Draw the Boundary Before the Box

Choose one service and one fact it claims to preserve. Follow that fact from a user action to recovery. Mark every path that can create, copy, delay, expose, corrupt, delete, or restore it. Include the manual tools and reports that people actually use, even if they are missing from the official architecture diagram.

Circle three surfaces that are easy to dismiss: perhaps a cache, queue, spreadsheet export, log stream, search index, materialized table, report, feature store, or object-storage bucket. For each one, complete this sentence:

If this surface is stale, missing, duplicated, leaked, or unrecoverable, then ______ will see ______ and the owner is ______.

Then ask what happens when a write commits but its next step does not, when a restore is complete but a derived surface is absent, and when the named owner is unavailable. The answers turn a database box into a production design.

Once the boundary is visible, the next task is to describe the pressure on it: which reads, writes, invariants, time windows, failures, and operating constraints it must withstand. Only then does a technology comparison have something real to answer.