Testing Data Systems: Properties, Faults, and Production Confidence

The Last Item Can Be Sold Twice

A team is ready to release inventory reservations. Its tests create a reservation, read it back, reject invalid input, and cancel it. Then two checkout workers reach the last item at the same time. Both read one unit available; both reserve it. Every request behaved as its example test predicted, and the business fact is wrong.

Concurrency is only the first difficulty. A retry after an unknown timeout can create a second payment authorization. A stale cache can resurrect stock already consumed in the source database. A replay can repair one derived view and duplicate another. A schema rollout can leave old and new consumers disagreeing about what a reservation means.

The release is claiming more than successful CRUD. It is claiming that confirmed reservations never exceed sellable stock, repeated attempts have one logical effect, derived availability converges on the ledger, and the system can recover those facts after failure. Those claims, not the endpoint list, should determine the tests.

Testing cannot supply an invariant the design does not enforce. It can reveal where the design depends on lucky ordering, quiet traffic, one code version, clean input, or an operator remembering an unwritten step. The useful question is therefore exact: under which conditions could this guarantee become false, and what evidence would expose the failure?

Turn the Claim into an Executable Property

A property test begins with a small model of stock. For each product, the model tracks received, reserved, sold, released, and adjusted units. The generated operations include reservations, cancellations, stock corrections, duplicate commands, failed payments, and retries after unknown outcomes. After each sequence, the ledger must reconcile and confirmed reservations must not exceed sellable stock.

The model does not need to reproduce the database or its locking scheme. It needs to express the business truth simply enough to act as an independent oracle. The production implementation processes the same operation sequence; the test then compares its source records, emitted events, derived availability, and visible result with the model.

This is stronger than checking that POST /reservations returns success. It can discover that a cancellation followed by a delayed payment confirmation releases stock twice, or that two individually idempotent consumers create a duplicate external side effect when combined.

Generated work should be hostile in ordinary ways: duplicated commands, reordered events, cancellations racing with confirmation, old schema versions, missing optional fields, invalid transitions, and no-op repairs. Data systems often fail when two legitimate behaviors meet. Exotic corrupt input is not required.

Keep the first failing case. Shrink the random sequence to the smallest reproducer, then retain its seed, operations, relevant records, and state snapshot as a regression fixture. A property test that discovers a production-class defect and forgets the reproducer has discarded its best evidence.

The same invariant should often exist twice: as a property before release and as a reconciliation check in production. The test explores failures the team can imagine; the monitor detects the combinations it did not.

Force the Race to Happen

Random operation sequences can still miss the decisive interleaving. A concurrency test should force two buyers to reach the reservation boundary together. Use a barrier so both attempts read, contend, or commit at the critical point; otherwise a supposed race test may merely be a slow sequential test.

Vary the mechanism that protects the invariant: transaction isolation, optimistic version checks, compare-and-swap, uniqueness constraints, partition ownership, and retry behavior. Vary the read route too. A primary read and a lagging replica may give the same request different evidence about whether stock remains.

Correctness has a cost. Record lock and pool wait, p99 latency, deadlocks, retry count, rejected work, and queue depth while forcing the race. A design that prevents oversell by making every reservation wait without bound has traded a correctness failure for an availability failure.

Shape the load around the risk rather than an impressive aggregate throughput number. The dangerous case may be one popular product, one large tenant, or one partition receiving most writes. Uniform traffic can conceal the hot key that dominates production.

The result should say what remained true and what it cost: “No oversell occurred, but p99 crossed the checkout objective at 38 concurrent reservations and deadlocks required three retries.” “The test passed” cannot support a release decision.

Cross Time and Version Boundaries

The reservation does not stop at its transaction. Events update a cache or read model; schema versions coexist during deployment; replay becomes a repair tool after drift. Tests must follow the fact across those boundaries.

A replay test should begin with damaged or missing derived availability and end with facts reconciled to the ledger. Reprocess from a named checkpoint with duplicates, late cancellations, and an old event version. Prove both that replay repairs the view and that it does not repeat payment, notification, or shipment side effects. Retained input, stable event identity, offset control, merge rules, and verification queries are part of the test, not setup trivia.

For a migration from reserved_quantity to reservation line items, run old producers with new consumers and new producers with old consumers. Exercise unknown fields, missing optional values, old events, rollback, and the backfill. A schema can remain structurally valid while its meaning changes—from missing to deleted, cents to dollars, local time to UTC, or advisory to authoritative—so contract tests must protect semantics as well as shape.

Run the backfill first on representative slices: a large tenant, deleted records, null edge cases, old versions, timezone boundaries, and facts previously corrected by hand. Compare control totals and sampled user-visible records before the new read path becomes authoritative. “We can replay” is optimism. A named checkpoint, deduplication rule, delete behavior, expected duration, and reconciliation query make it reviewable.

Break the Designed Boundary, Then Recover the Facts

Before injecting a fault, write the expected behavior. If the reservation consumer stops, should checkout block, reject risky purchases, or continue while availability becomes stale? At what lag must the system change behavior? Which alert fires, who can stop the drill, and what evidence proves recovery? Killing a node without these answers produces activity, not confidence.

Begin with controlled failures in an isolated or guarded environment: pause the consumer, slow the database, exhaust a connection pool, reject a subset of messages, revoke a credential, or force a leader change. Observe user impact, queue growth, alerts, runbook action, correctness drift, and recovery time. Fault injection belongs in production only when mature controls, stop conditions, rollback authority, and an intentionally limited blast radius make that risk defensible.

Recovery must restore service and trust. Restore the reservation database into an isolated environment, start the application against it, verify permissions and replication, reconcile ledger and stock totals, resume from known checkpoints, and confirm that fresh backups work after the restore. Rebuilding a cache or search index likewise ends with source and output facts agreeing, not merely with a completed job.

Recovery has two clocks. Recovery time measures how long service takes to return to an acceptable state. Recovery point measures how much data may be lost or reconstructed. A drill that times the restore but ignores missing or duplicate writes, access, downstream consumers, and post-restore backup has proved only that bytes can be copied.

Previous incidents supply the best fault catalogue. The timeout, stale replica, retry storm, and stalled consumer from the preceding chapter should survive as repeatable tests. An incident is not fully learned from while its lesson remains only in prose.

Write the Evidence Plan

For each important claim, write a short evidence plan:

1. State the guarantee in user or operator terms and name the invariant, its scope, source of truth, and allowed exceptions.
2. Name the operation sequence, overlap, fault, migration state, replay point, or recovery event that could violate it.
3. Reproduce the risky production shape: volume, skew, partitioning, tenant mix, latency, and version mix. More load is not necessarily more representative load.
4. Choose the cheapest test that can expose the failure. Unit tests protect local rules; integration and contract tests protect boundaries and meaning; property and concurrency tests protect state across sequences and interleavings; migration, replay, fault, and recovery tests protect change and survival.
5. Define acceptance evidence before the run: invariant checks, control totals, p99, lock waits, lag, queue depth, recovery time, recovery point, or a verification query.
6. Retain the seed, operation history, fixture, checkpoint, snapshot, and relevant diagnostics needed to reproduce a failure. Name the owner, cadence, and release or operating decision that depends on the result.

Also record what the plan deliberately does not test. Some risks are low impact, impractical to simulate, or better controlled through monitoring and rollback. An explicit gap can be reviewed. A hidden gap looks exactly like confidence until the system fails there.

Confidence Drill

Use these exercises to turn the chapter into review practice.

1. For an account-transfer system, state the conservation invariant and design generated deposits, withdrawals, transfers, duplicates, failed commands, and timeout retries. Add one forced interleaving where two transfers compete for the same balance. What evidence distinguishes a preserved invariant from a merely successful response?
2. Design a quarterly restore drill for one production database. Name the backup, isolated restore environment, permissions and application checks, reconciliation query, recovery time, recovery point, owner, and retained evidence. What would make you refuse to call the drill successful?
3. Choose a current service and identify one guarantee it does not test under concurrency, replay, migration, or failure. Design the cheapest credible test. Then vary the problem so the original test is insufficient: add a hot tenant, an old consumer, a lost response, or a missing operator.

Finish by writing down what remains trusted without evidence. Perhaps nobody has restored without the primary operator, replay excludes external notifications, or concurrency tests omit the skew of the hottest tenant. The next chapter turns demonstrated behavior into an operating promise. That promise is credible only where this chapter leaves evidence—and honest only where it names the gaps.