State Placement, Ownership, and Derived Data

Which copy is allowed to say that a customer exists?

Ledgerline’s account service says the customer was deleted at version 884. Its order database retains invoices under a separate business policy. The catalog search index still contains the customer’s saved-list text. A regional cache has an unexpired profile response. Fraud features preserve a hashed device relationship. An analytics export is already in object storage. A browser holds ephemeral session state. The change-data-capture stream contains both the original record and a deletion tombstone.

“The account database is the source of truth” does not decide what any of those copies should do. It does not identify which facts the order domain may retain, which projection is stale, whether a hash remains linkable, whether a log is rebuildable, how the cache learns about deletion, or which team owns reconciliation.

The useful question is not “where is the data?” It is:

For this fact and operation, which boundary has write authority, and what authority, freshness, rebuild, retention, reconciliation, and transfer contract governs every other copy?

The governing rule is: duplicate state only with explicit authority, freshness, rebuild, retention, and reconciliation semantics.

Inventory facts before stores

Begin with facts and operations, not database products. “Customer,” “order,” and “inventory” are not single records. Ledgerline separates:

• account identity and contact preferences;
• legally relevant order and invoice facts;
• stock units available for reservation;
• reservation ownership and expiry;
• catalog descriptions and merchandising labels;
• search tokens and ranking features;
• fraud signals and model features;
• session and routing hints; and
• deletion, restriction, and retention decisions.

For each fact, identify the operations that create, mutate, decide with, display, export, correct, retain, and delete it. An authority is scoped to a fact and mutation, not awarded to an entire service forever.

Ledgerline uses order authority for order lifecycle and stock authority for reservation availability. Search is authoritative for neither. It may decide ranking from derived features, but it cannot decide that an order exists or stock is available. Fraud may return a risk decision that is authoritative for an admission policy while its input features remain derived from account, order, and device events. Authority is always “authority for what?”

A state inventory should classify each copy:

“No direct writes” matters. A search administrator correcting an order field in the index creates a second authority. Operational tools should repair the source fact or replay/reconcile the projection through a controlled path.

Authority is a protocol boundary

Authoritative state is the boundary at which accepted mutations become the durable basis for later decisions. It needs:

• stable identity for the fact and logical operation;
• validation of the invariant and caller’s authority;
• an ordering rule such as version, epoch, or transactional serialization;
• acknowledgment semantics naming what is durable;
• conflict and ambiguous-outcome handling;
• a recovery point and audit evidence; and
• a schema/contract owner.

Authority does not imply one physical machine. A replicated database can expose one logical write authority. A partitioned system can have one authority per key range. A commutative data type can admit several writers only when its merge law and application invariant make that safe. “Multi-writer” is not an exemption from specifying authority; it is a more complex authority protocol.

Single-writer strategies reduce conflict scope. One writer may mean one leader, one partition owner, one home region, one serialized command stream, or one workflow that owns a transition. Availability then depends on lease/failover and fencing. When ownership moves from epoch e to e+1, storage must reject stale writes from e; routing alone cannot prevent the old owner from acting.

Separate data ownership from storage administration. The order team owns order semantics and schema compatibility even if a platform team operates the database. The discovery team owns the search projection schema and rebuild process, but it cannot redefine order truth. Every contract change needs a producer owner, consumer owners, compatibility interval, and migration evidence.

Derived state is executable policy

A derived copy transforms authoritative facts for a read path, decision, or workload. Common forms differ in purpose:

• a replica preserves roughly the same logical model for locality, scale, or recovery;
• an index reorganizes fields for a query access path;
• a materialized projection selects, joins, aggregates, or transforms facts;
• a cache retains previously computed or fetched results under freshness policy;
• a feature set encodes facts for model training or serving;
• an export captures an as-of view for another boundary; and
• ephemeral state supports an in-flight session, process, lease, or calculation and may not need rebuild.

Derived does not mean unimportant. A stale search index can hide a paid order. A stale authorization projection can expose access. A wrong fraud feature can reject a customer. The classification tells us where correction starts; it does not excuse weak objectives.

Record the derivation function as a versioned contract:

projection generation = g(source schema versions,
                          source positions,
                          transformation version,
                          configuration/model version)

The generation must be distinguishable during rebuild and rollback. If old and new transformations write the same namespace without a generation tag, mixed results become difficult to detect or reverse.

Choose synchronous derivation only for the invariant

A synchronous derivation completes before the originating operation acknowledges. It can make a view current at the commit boundary, but it adds latency, availability coupling, resource demand, and rollback complexity. If an order transaction synchronously updates search, search impairment becomes checkout impairment.

Use synchronous derivation when the derived representation participates in the same correctness decision and cannot safely be recomputed or observed later. Often the better design is to keep the invariant in the authority and update read-optimized copies asynchronously.

An asynchronous derivation acknowledges authority first, then propagates change. It decouples latency and failure but exposes staleness, replay, duplication, reordering, and backlog. It requires a user-visible contract:

• maximum or target commit-to-visible age by operation and mode;
• behavior when the age is unknown or beyond bound;
• version/as-of evidence returned to readers;
• monotonic or read-your-write mechanisms where required;
• reconciliation for missing, duplicate, reordered, or poison changes; and
• recovery time after a declared outage.

Do not call an asynchronous view “eventually consistent” and stop. Eventual convergence without a time, liveness, retention, and repair assumption cannot guide a product or incident decision.

Change-data capture is transport, not completion

Change-data capture (CDC) extracts committed mutations from an authoritative boundary and makes them consumable. Capture can arise from a database log, outbox, event log, or application-owned change table. The record needs stable identity, source position/version, schema identity, operation type, event/commit time, and enough before/after information for consumers’ contracts.

Distinguish milestones:

source transaction committed
change retained for capture
capture position acknowledged
record durably transported
consumer checkpoint advanced
derived mutation committed
index/query path exposes generation
cache invalidation converged
reader observed required version

Only the last relevant milestone satisfies user-visible freshness. A zero transport lag dashboard can coexist with an hour-old search index if apply or indexing is stalled.

CDC retention creates a recovery obligation. A slow consumer can hold a database replication slot or exhaust a log retention budget. Define maximum consumer lag, storage alarm, backpressure, quarantine, snapshot/reseed path, and the authority allowed to abandon a consumer position. Never let a low-value projection threaten the authoritative store by retaining unbounded change history.

Schema changes need source and consumer coordination. Additive fields can still change service demand; default values can alter meaning; deletes may lack previous keys if capture identity is insufficient. Test old producer/new consumer and new producer/old consumer behavior across the retained replay window.

Spend a freshness budget end to end

Ledgerline promises that eligible order changes become visible in search within 5 seconds for 99% of successful changes during normal mode. The population, outcome, percentile, mode, and observation points are part of the statement.

Its modeled budget is:

Budgets are not independently composable percentiles. The table is a modeled allocation, not a claim that summing each stage’s p99 yields the end-to-end p99. Preserve per-change correlation IDs and timestamps so the end-to-end distribution is observed directly. Use stage distributions to localize consumed time.

Track age in both clock and source-position terms. Clock skew can distort timestamp differences; a source position can look close while records are expensive. Useful signals include:

• commit-to-captured, captured-to-durable, durable-to-applied, and applied-to-visible age;
• oldest unapplied source position by partition;
• arrival and useful apply rate in records, bytes, and service-demand units;
• poison/quarantine count and age;
• projection generation and source high-water mark;
• reader version/as-of evidence; and
• reconciliation mismatch rate and correction age.

Freshness policy varies by fact. Search ranking may tolerate five seconds. A just-placed order view may carry the committed order version and fall back to authority if the projection is behind. Stock availability should not use an unconstrained projection to admit a reservation. Account deletion may require the cache to fail closed after a revocation version even if ordinary profiles serve stale during an outage.

Place reads near demand and writes near invariants

State placement balances latency, service demand, data gravity, coordination, residency, failure, recovery, and cost.

Co-location can remove network rounds, reduce serialization, preserve cache locality, and make a transaction possible. It can also join scaling and failure domains. Remote access preserves ownership and avoids copies but pays network latency and dependency availability on every call. Derived local state spends storage and propagation work to remove remote work from the read path.

Use an operation-level placement test:

1. What fact and invariant does the operation depend on?
2. Does it mutate authority or only read a declared version/freshness?
3. What is the call rate, payload, fan-out, and service demand?
4. What latency and availability objective applies by geography and mode?
5. Can a bounded derived view answer correctly?
6. What propagation, rebuild, retention, security, and deletion cost does the copy create?
7. What happens under partition, lag, authority failure, and recovery?

For Ledgerline catalog search, local regional projections are justified by high read rate and bounded freshness. For stock reservation, remote authority or region-local stock ownership is justified by the no-oversell invariant; copying availability everywhere does not create safe write authority. For order display immediately after commit, a session version lets the read path use the projection when it has caught up and fall back to order authority within a remaining deadline.

Avoid distributed joins by accident. A read endpoint that synchronously calls order, account, catalog, shipment, and fraud owners for every row has chosen remote state composition on its critical path. A materialized view may bound latency, but it must preserve versions, partial-data semantics, privacy, and replay.

Data gravity includes more than bytes at rest. It includes update rate, indexes, compute, keys, network, cache warmth, operator tools, backups, and the time to move or rebuild. Place computation near a large authoritative state when moving the query is cheaper and policy permits; create derived state near repeated reads when its lifecycle cost is justified.

Rebuildability is an operational claim

“We have the log” is not a rebuild plan. A projection is rebuildable only if the required source history, schemas, transformations, keys, reference data, ordering, and compute remain available for the whole rebuild. The log may begin after the earliest retained fact, omit deletes, reference an expired schema, or contain encrypted fields whose key is gone.

A safe rebuild uses generations:

1. declare source snapshot boundary and change-stream position;
2. create a new projection generation without mutating the serving generation;
3. scan the snapshot under source CPU, I/O, network, and concurrency limits;
4. apply live changes from the recorded position with idempotent identities;
5. validate counts, checksums, invariants, sample queries, and freshness;
6. shadow representative reads and compare semantic results;
7. pause or bound cutover delta where required;
8. atomically switch a versioned routing pointer;
9. retain the prior generation for bounded rollback; and
10. delete it only after retention and incident windows pass.

The rebuild itself is production workload. Ledgerline’s search projection is 2.4 TiB. A builder can read at 160 MiB/s, but the authoritative source exposes 400 MiB/s and grants rebuild only 25% to protect foreground service:

source rebuild cap = 400 MiB/s × 0.25 = 100 MiB/s
2.4 TiB = 2.4 × 1,048,576 MiB = 2,516,582.4 MiB
snapshot lower bound = 2,516,582.4 MiB / 100 MiB/s
                     = 25,165.824 s
                     ≈ 6.99 h

That is not catch-up time. If live changes arrive at 32 MiB/s and apply cost is approximated by the same bandwidth unit, only 68 MiB/s remains for historical progress:

net historical progress = 100 - 32 = 68 MiB/s
snapshot + concurrent catch-up lower bound
  = 2,516,582.4 MiB / 68 MiB/s
  = 37,008.565 s
  ≈ 10.28 h

The executable fixture at examples/performance-engineering-system-design-handbook/part-04/state-lineage/ reproduces these conversions. Real rebuild time is longer because record cost, decoding, indexing, compaction, pauses, retries, skew, validation, and final cutover are omitted. If live arrival equals or exceeds useful apply capacity, the rebuild never catches up.

Admission must respond to foreground harm. Define source latency/queue guards, maximum rebuild share, pause/resume hysteresis, consumer lag ceiling, storage budget, and an abort that retains the checkpoint. Schedule rebuilds by tenant/cell and failure domain so one software defect cannot rebuild every projection simultaneously.

Reconciliation needs an owner and a terminal state

Asynchronous systems miss, duplicate, reorder, and poison changes. Reconciliation compares derived state with authority or a trusted lineage checkpoint and repairs divergence.

Use layers:

• continuous sequence-gap and lag detection;
• invariant and count checks per partition/generation;
• sampled semantic comparison of authority and read result;
• targeted replay by stable source interval;
• partition or tenant rebuild; and
• full generation rebuild when lineage cannot be trusted.

Repair must be idempotent and version-aware. A late replay of version 882 must not resurrect a value deleted at 884. Tombstones need identity, authority version, effective policy, and sufficient retention to reach every obligated consumer or force its rebuild from a post-deletion source.

Name the owner of unknowns. A poison change cannot remain forever in a dead-letter queue with freshness dashboards green. Define retry budget, quarantine visibility, product behavior for affected facts, escalation, manual repair authority, and terminal reconciliation evidence.

Duplication spends governance and security budget

Copies buy read latency, throughput, independence, query shape, and recovery options. They also multiply:

• storage, replication, index, backup, and egress cost;
• schemas, access policies, encryption keys, and credentials;
• deletion, retention, legal-hold, export, and audit paths;
• attack surface and operator access;
• change/rebuild demand and incident modes;
• data catalog and lineage obligations; and
• the probability that one stale copy influences a decision.

Minimize fields before copying. A search index rarely needs payment details. A cache key must include every dimension that changes authorization or representation; otherwise it can cross tenant or policy boundaries. Encrypting a copy does not remove retention or deletion duties, and hashing may not make data anonymous when linkage remains possible.

Retention is fact- and purpose-specific. The authority’s retention does not automatically apply to invoices, fraud evidence, backups, analytics, or logs; each needs approved policy and ownership. This is architecture guidance, not jurisdiction-specific legal advice. Translate applicable policy into explicit system contracts with qualified privacy and legal owners.

Deletion is a state transition across lineage, not a single row removal:

deletion decision committed at authority version 884
-> tombstone retained and propagated
-> online projections remove or restrict fact
-> caches invalidate or fail validation
-> exports/recipients receive their contractual action
-> backups age out or follow approved restoration procedure
-> rebuild sources cannot recreate prohibited state
-> evidence records completion without retaining the deleted payload

Backups deserve precise wording. Immediate selective removal may not be feasible in an immutable backup format; the system still needs access controls, retention expiry, restore-time deletion replay, and evidence that restored data cannot silently re-enter serving copies. The policy owner decides the required behavior.

Transfer state without creating two authorities

Scaling, cell movement, tenant tier migration, region evacuation, and storage replacement all move state. Separate copy from authority:

1. allocate destination and declare generation/epoch;
2. take a consistent source boundary and record its position;
3. copy under foreground resource budgets;
4. stream changes while source remains authoritative;
5. validate content, schema, keys, and destination performance;
6. serve shadow or bounded reads with version comparison;
7. stop new authority changes briefly or use a proven handoff protocol;
8. advance authority to epoch e+1 and make storage reject e;
9. route a canary and observe invariants, lag, tails, and errors;
10. drain stale routes, sessions, retries, and writers; and
11. preserve rollback to a coherent authority, not two writable copies.

If rollback occurs after the new authority accepted writes, “point routing back” is not rollback. The old location must catch up and receive a newer fenced epoch, or the new location must remain authority while the defect is repaired. Record ambiguous operations by stable identity.

Schema and contract ownership travel with authority. A destination that cannot understand every retained change cannot safely become authority. Test downgrade/rollback compatibility and the oldest replayable schema before cutover.

State-placement canvas

Fact and operations:
  identity, mutations, decisions, readers, geography, data/security class

Authority:
  owner, invariant, write path, version/epoch, acknowledgment, ambiguity

Copies and lineage:
  replica/index/projection/cache/feature/export/ephemeral;
  source positions, transformation and generation versions

Read paths:
  operation, locality, query shape, version/as-of evidence, fallback

Derivation:
  sync/async reason, CDC identity/order/schema, failure and poison behavior

Freshness:
  population, objective, stage budget, unknown/stale response, telemetry

Rebuild:
  snapshot/log sufficiency, bandwidth/service-demand cap, catch-up,
  validation, shadow, cutover, rollback, recovery objective

Reconciliation:
  gap/invariant/sample checks, replay identity, terminal owner

Governance:
  field minimization, access, encryption, residency, export, audit

Retention and deletion:
  policy owner, tombstone horizon, backups/restore, completion evidence

Transfer:
  destination capacity, copy/catch-up, epoch fence, canary, drain, rollback

Economics:
  storage, propagation, rebuild, egress, operator and governance cost

Reject a copy with no named authority, an asynchronous path with no freshness and reconciliation behavior, a projection with no viable rebuild source, or a deletion policy that ends at the first database.

State-lineage drills

Classify the copies. For an order record, regional replica, search index, profile cache, fraud feature, analytics export, CDC stream, and session hint, state authority, permitted writer, freshness, rebuild source, and retention owner. Identify any accidental second writer.

Spend five seconds. Reproduce the 0.35 + 0.45 + 1.20 + 1.40 + 0.60 = 4.00 s modeled path and 1.00 s margin. Add a 1.1-second p99 transport regression. Decide which gate or product behavior changes; do not add stage percentiles as though they were end-to-end evidence.

Choose sync or async. Place stock reservation, search indexing, order confirmation email, risk decision, and invoice export. State the invariant, acknowledgment boundary, stale behavior, and recovery consequence for each.

Reject the remote join. An order-list endpoint calls five authorities per row. Compare critical-path remote composition, an asynchronous materialized view, and a bounded hybrid with version fallback. Include deletion and schema evolution.

Plan the rebuild. Reproduce 2.4 TiB at a 100 MiB/s cap as about 6.99 hours and at 68 MiB/s net historical progress as about 10.28 hours. Add index amplification, pause windows, and a 20% demand spike; define an honest recovery objective.

Cut over a generation. Build search-v43 from source position p, catch up, compare reads, switch the routing pointer, detect a semantic mismatch, and roll back without dropping changes accepted after cutover.

Trace deletion 884. Follow authority, log, search, cache, feature store, export, backup, and ephemeral session. For each, specify action, deadline, evidence, restoration behavior, and owner without inventing a universal legal rule.

Move authority. Transfer one stock partition from epoch 18 to 19 while clients retry and an old worker is paused. Specify fencing, duplicate identity, read versions, canary, drain, and the only safe rollback paths.

Challenge the log. Remove one historical schema, one encryption key, delete before-images, and shorten retention below rebuild time. Determine which projections remain rebuildable and which need a new snapshot contract.

Durable rules for state

1. Define authority per fact and mutation; “source of truth” without scope is not a design.
2. Give derived systems only the write capabilities needed to project, fill, invalidate, or reconcile.
3. Choose synchronous derivation only where the invariant justifies latency and availability coupling.
4. Treat CDC acknowledgment, consumer apply, index visibility, cache convergence, and reader observation as different milestones.
5. State freshness by population, operation, mode, percentile/window, and stale/unknown behavior.
6. Place authority near required coordination and derived reads near repeated demand when lifecycle cost is justified.
7. Prove rebuild inputs, versions, keys, capacity, catch-up, validation, cutover, and rollback.
8. Reconcile with stable identity and versions so late replay cannot resurrect older truth.
9. Price every copy in storage, propagation, rebuild, governance, security, and deletion work.
10. Propagate retention and deletion across lineage, including restore behavior and recipients.
11. Move state by copying and validating first, then transferring authority through a fenced epoch.
12. Keep data-domain ownership explicit even when infrastructure teams operate the stores.

Distributed architecture now has a complete set of boundaries: topology, delivery, effects, time, failure, geography, elasticity, tenancy, and state authority. The next question is compositional. A low-latency request/response service must choose a path through these mechanisms without turning every protection into another synchronous dependency or every fast read into ungoverned derived state.

Evidence and transfer limits

• RFC 9111 defines HTTP cache freshness, validation, and invalidation semantics and explicitly notes that an unsafe request does not guarantee global invalidation of every appropriate response. Ledgerline’s internal caches need their own version, authorization, and lineage contracts in addition to HTTP behavior.
• Current PostgreSQL logical decoding documentation describes extracting committed database modifications into streams and notes that old row availability for updates/deletes depends on replica identity. It is a current implementation example, not a universal CDC completion or retention guarantee.
• Gupta and Mumick’s primary materialized-view maintenance paper develops the view-maintenance problem across available information, mutation type, view language, and data instance. Its techniques do not supply Ledgerline’s ownership, freshness, privacy, rebuild-capacity, or operational cutover contracts.
• LinkedIn Engineering’s account of the log as a data-system abstraction explains logs as ordered integration and replay infrastructure while also identifying schemas and compatibility as separate concerns. A retained log is valuable evidence, not proof that every required rebuild input survives.
• The deterministic fixture in examples/performance-engineering-system-design-handbook/part-04/state-lineage/ reproduces the five-second budget and rebuild lower bounds. It assumes constant rates and omits distributions, decode/apply CPU, compaction, index amplification, skew, retries, failure, validation, and cutover; replace its inputs with traced end-to-end freshness and staged rebuild evidence for a real system.