Production Data Systems Handbook / Chapter 57
Appendix C: Invariant Register Template
A reusable register for naming, enforcing, monitoring, and repairing the facts a production data system must preserve.
Purpose
Ask a team what its data system must never get wrong and the first answers are often too broad to design against: payments must be correct, tenant data must be secure, dashboards must be accurate. An invariant register turns each promise into a fact that can be true or false, then follows that fact through prevention, proof, production detection, and repair.
Use the register during architecture review, migration planning, operational readiness, and incident review. Begin with business facts such as “a payment attempt produces at most one capture.” Add technical facts when their violation has a business consequence, such as “a deleted private document disappears from every search index within the deletion SLO.”
The register should expose weak designs. If a critical fact has no enforcement mechanism, no production signal, or no safe repair path, an empty cell is useful evidence. Do not fill it with an aspiration.
Write the Fact Before the Mechanism
State an invariant without naming the implementation that currently protects it. “Payment IDs have a unique index” describes a mechanism. “A payment attempt produces at most one capture” describes the fact the mechanism must preserve. The second statement remains useful if the database, processor, or retry path changes.
Choose the scope at which the fact could be broken: a row, account, tenant, partition, stream, service, region, or the whole system. Enforcement must meet that boundary. A database constraint may protect a fact inside one store; it cannot by itself protect a payment capture that crosses a local transaction and an external processor.
Then record the rest of the operating chain:
- Severity follows the harm of a violation, not the difficulty of implementation.
- Enforced by names the mechanism that prevents the fact from becoming false.
- Test proves that mechanism under the conditions that threaten it: concurrency, retry, replay, migration, or mixed application versions.
- Monitor detects real violations or dangerous drift in production. A dashboard without an accountable response is not a monitor.
- Failure impact names the user, financial, legal, or operational harm.
- Repair says how to stop further damage, preserve evidence, and restore truth. Prefer compensating entries to rewritten history where auditability matters.
- Owner is the accountable team, even when several systems participate.
- Review trigger names the change that can invalidate the entry, such as a new processor, consumer, region, schema, or incident.
Severity and Response
Use critical when a violation can cause financial or privacy harm, legal or safety exposure, irreversible data loss, or customer-visible corruption. Prevent it in the write path when possible; alert immediately, preserve evidence, and repair under incident control.
Use high when the likely result is a wrong user decision, billing error, persistent downstream drift, degraded trust, or support escalation. Detect it quickly, stop its spread, repair with owner review, and add regression coverage.
Use medium for recoverable inconsistency, incorrect derived views, delayed operations, or noisy analytics. Monitor the trend and repair through an owned job or backlog. Use low only for bounded, temporary, cosmetic, or non-critical analytical errors, and still give cleanup a threshold and an owner.
Register Template
The table earns its width because a review must follow each invariant across the same operating attributes. In a living register, use one row per invariant and link long repair procedures or test plans rather than hiding them in a cell.
| Invariant | Scope | Severity | Enforced by | Test | Monitor | Failure impact | Repair | Owner | Review trigger |
|---|---|---|---|---|---|---|---|---|---|
| [Fact that can be true or false] | [Boundary] | [Critical / High / Medium / Low] | [Preventive mechanism] | [Proof under relevant stress] | [Production detection] | [Concrete harm] | [Safe recovery path] | [Accountable team] | [Change or event] |
Worked Entry: Duplicate Payment Capture
Suppose a client times out after requesting a capture. It retries while the processor also redelivers a callback. “The endpoint is idempotent” is not yet a useful invariant: it neither identifies the business action nor explains what happens across the external boundary.
The fact is: a payment attempt produces at most one capture. Its scope is the stable payment-attempt ID shared by the command, ledger entry, and processor request. The invariant is critical because a violation overcharges a customer and creates financial and audit work.
Prevention has several parts. A unique idempotency key represents the payment attempt rather than an individual HTTP request. A transaction couples the accepted command to its local ledger state. Every processor retry reuses that business identity, and responses and callbacks are deduplicated against it. None of these controls alone covers the whole path.
The test races concurrent requests, injects a timeout after processor acceptance, and replays duplicate callbacks. It must prove one capture and one authoritative ledger effect, not merely identical HTTP responses. In production, a reconciliation query compares capture records with processor outcomes and raises a duplicate-capture or ledger-balance exception.
Repair begins by pausing the affected capture worker and preserving requests, callbacks, and ledger evidence. The payments team then uses the finance-approved refund or compensating-ledger path; it does not delete history until the numbers look right. A new processor, ledger migration, or retry-policy change forces review of the entry.
This is how the completed row reads when the details must be scanned together:
| Invariant | Scope | Severity | Enforced by | Test | Monitor | Failure impact | Repair | Owner | Review trigger |
|---|---|---|---|---|---|---|---|---|---|
| A payment attempt produces at most one capture. | Payment-attempt ID across command, ledger, and processor. | Critical | Business idempotency key reused at the processor; ledger transaction; response and callback deduplication. | Concurrent retry, ambiguous timeout, and duplicate-callback replay. | Processor-to-ledger reconciliation; duplicate-capture and balance exceptions. | Customer overcharge, financial correction, support escalation, audit exposure. | Pause worker, preserve evidence, refund or post a compensating entry through the approved path. | Payments platform. | New processor, ledger migration, or retry-policy change. |
Match Enforcement to the Boundary
A database constraint is strong for local uniqueness, references, non-null fields, and state transitions inside one store. It does not protect a fact spanning another store or region.
A transaction can preserve a multi-row or multi-table fact inside one transactional boundary, provided its isolation level prevents the anomaly in question. A conditional write or compare-and-swap protects a key or aggregate version from lost updates, but callers must treat conflicts and retries as part of the contract.
An idempotency key protects a retried business command, duplicate message, webhook callback, or external side effect only when its identity represents the business action. A transport-generated message ID is often too narrow.
Some facts cannot be prevented atomically. A search index may retain a deleted private document after the source transaction commits. In that case, a durable tombstone and idempotent delete consumer reduce the window, while reconciliation detects missed propagation and replay or rebuild repairs it. Reconciliation is a necessary control for a cross-system fact; it is not an excuse to omit prevention where immediate harm can be stopped.
A workflow gate fits high-risk corrections, exports, or destructive migrations. It needs audit evidence, separation of roles, and an emergency path or it merely relocates the risk to an operator.
Review the Register Under Change
For each critical or high invariant, ask:
- Can the statement be proved false with a concrete counterexample?
- Does enforcement cover the boundary where the fact can be broken?
- Do tests exercise concurrency, retries, replay, migrations, and old application versions where relevant?
- Will the production signal arrive before the harm becomes difficult to contain?
- Does repair preserve evidence and restore truth without concealing history?
- Is one team accountable when several systems participate?
Every critical invariant needs prevention, detection, and repair. When one is missing, the design is not production-ready. The blank cell is the decision the team still owes.
Continue reading
Full table of contents