Time, Ordering, Identity, and Causality

Which happened first: an order being accepted or its payment being recorded?

Mercury’s audit export offers an apparently obvious answer:

12:00:00.120  payment-service  PAID       order=o-1842
12:00:00.180  order-service    ACCEPTED   order=o-1842

Yet PAID was produced only after the payment service consumed the OrderAccepted fact. In the teaching trace, acceptance occurred at actual model time 1,000 ms on a node whose wall clock was 80 ms ahead. Payment occurred 40 ms later on a node whose wall clock was 70 ms behind:

actual order:     ACCEPTED at 1,000 ms ──message──> PAID at 1,040 ms
displayed order:  PAID at 970 ms          <────── ACCEPTED at 1,080 ms

The displayed timestamps invert the causal order by 110 ms. Neither process is necessarily faulty. Both readings can be plausible within their clock conditions, network delay, and synchronization uncertainty.

The repair is not “synchronize harder” and hope. Mercury must decide what question it is asking:

• use a monotonic clock to measure elapsed time inside one clock domain;
• use UTC wall time, source, and uncertainty for user and civil-time meaning;
• use aggregate versions, epochs, fencing tokens, or transactional order for correctness;
• use event, command, causation, and effect identities to reconstruct a distributed chain; and
• use event time plus an explicit lateness policy to assign facts to domain-time windows.

The operating rule is: use clocks for measurement and user semantics; use explicit versions, epochs, or causal metadata when correctness depends on order. A timestamp can be valuable evidence without being authority.

Four meanings hide inside the word “time”

The same 64-bit field cannot safely perform every temporal job.

“Timestamp” should therefore be qualified. An audit record might carry occurred_at, recorded_at, ingested_at, a monotonic duration, aggregate_version, leader_epoch, and causation_id. That is not redundant decoration. Each field answers a different question.

Physical time itself should often be treated as an interval rather than an exact point. If a source reports time t with bounded uncertainty ε, the defensible claim is that the instant lies in an interval such as [t − ε, t + ε], under the source’s stated model. Two intervals that overlap do not provide a safe order. An implementation may expose a different interval convention; use its contract rather than copying notation.

NTP’s specification distinguishes offset, round-trip delay, dispersion, jitter, and synchronization distance. These are not synonyms for “the clock is accurate.” Offset is an estimate relative to a source; network asymmetry limits what round-trip exchange can infer; dispersion grows with time; and a disciplined clock can step or slew according to implementation policy. Monitor the bound and the clock state, not merely whether a daemon is running.

Duration belongs to a monotonic clock

Suppose Mercury writes wall time at request start, the clock is corrected backward by 240 ms, and it then subtracts wall time at completion. A real 110 ms operation appears to take −130 ms. A forward correction can make it appear much slower. Expiry queues, backoff, rate windows, latency histograms, and deadline accounting inherit the defect.

Use a monotonic source for elapsed duration within one process or supported clock domain:

started = monotonic_now()
deadline = started + 250 ms

before_each_wait:
    remaining = deadline - monotonic_now()
    if remaining <= 0: cancel_or_reject()

The code’s contract still needs detail. Does the monotonic clock continue while the host is suspended? Is it shared consistently across cores and virtual CPUs? What are its resolution and read cost? Can a process migrate between machines while preserving the deadline? POSIX defines CLOCK_MONOTONIC as a clock that cannot be set and represents monotonic time since an unspecified point; platform behavior and suspend semantics still need the platform’s exact documentation.

Do not transmit a raw monotonic timestamp to another host and subtract it there. The origins are unrelated. Transmit an absolute protocol deadline only when all participants share an adequate time contract, or transmit the remaining budget while subtracting local transit and processing time conservatively. Chapter 25’s end-to-end deadline remains the operation’s authority; a clock only implements its accounting.

Correctness order needs an owner and a scope

Mercury originally resolved two inventory updates by selecting the row with the greatest updated_at. That rule silently handed correctness to whichever writer’s wall clock read later. It failed under skew and under a retry that replayed an older command with a newly generated timestamp.

The invariant is narrower and stronger:

For one inventory item, only the authority for the current allocation epoch may advance the aggregate from version v to v + 1; a retry of the same command returns the stored outcome, and an older epoch can never regain write authority.

An explicit write looks like:

apply(command_id, item_id, expected_version=41, authority_epoch=42, delta=-3)

accept only when:
  authority_epoch == current_epoch(item_id)
  expected_version == current_version(item_id)
  command_id has no conflicting prior intent

on commit:
  current_version = 42
  record command_id and outcome atomically

The aggregate version orders mutations for one item. It does not order unrelated items. The command identity suppresses retry of the same intent; it is not the version. The authority epoch distinguishes leadership generations; it is not wall time. These separate fields prevent an attractive but ambiguous “latest timestamp wins” rule.

Epochs must be fenced at the authority

Imagine leader A holds epoch 41 and pauses for 12 seconds. The coordinator elects leader B with epoch 42. B writes successfully. A resumes and still believes its lease is valid. Stopping A at the coordinator is insufficient because the paused process may already hold a connection or queued write.

Every authority-changing operation carries the fencing token to the state owner:

storage.current_epoch = 42

write(epoch=42, version=42) -> accept
late_write(epoch=41, version=43) -> reject stale authority

The storage service, lock authority, device, or other irreversible boundary must reject a token lower than the highest accepted epoch. A client-only check has no force after the client pauses. Fencing also needs atomic persistence with the protected mutation; checking epoch in one store and writing in another recreates a race.

Sequence numbers and versions need declared allocation rules:

• per connection, they can detect loss or reordering only within that connection’s lifecycle;
• per producer, they need a stable producer generation to distinguish restart from replay;
• per aggregate, they serialize one entity but not a cross-aggregate invariant;
• per partition or log, they order accepted records in that partition, not events everywhere; and
• globally allocated sequence numbers create an ordering service whose latency, availability, capacity, and recovery enter the critical path.

Stronger scope is not automatically better. Pay for only the order the invariant needs.

Causality is a partial order

Lamport’s happens-before relation begins with three rules:

1. events within one process follow that process’s order;
2. sending a message happens before receiving that message; and
3. the relation is transitive.

If neither (a \rightarrow b) nor (b \rightarrow a), the events are concurrent in this model. Concurrent does not mean simultaneous in physical time. It means the recorded communication relation does not establish that either could have influenced the other.

A Lamport scalar clock advances on local events and moves beyond a received clock value. It preserves a useful implication:

a happens-before b  =>  L(a) < L(b)

The converse is not guaranteed. L(a) < L(b) alone does not prove causation, and scalar values cannot identify every concurrent pair. A tie-breaker such as process ID can create a deterministic total order for a protocol, but that imposed order is not newly discovered physical or causal truth.

Vector clocks retain one logical component per participant or tracked dimension. In the fixture:

A = [3, 1]
B = [2, 4]
J = [3, 4]

A and B are incomparable: each has one component greater than the other, so the model treats them as concurrent. Both are component-wise less than or equal to J, with at least one strict component, so both happen before J. The additional information costs metadata, comparison work, lifecycle management for participants, and compaction or approximation when membership is dynamic.

Hybrid logical clocks combine a physical-time component with logical correction so timestamps stay close to wall time while preserving a causal-order property under the algorithm’s assumptions. They can improve audit range scans and snapshot selection without requiring a full vector. They do not make the physical clock exact, reveal every concurrent relation, or remove the need for an authority rule. “Hybrid” names a representation and update algorithm, not a universal consistency guarantee.

Choose causal metadata from the decision:

Identity must outlive the retry it is meant to recognize

A globally unique identifier solves a naming collision problem within a declared generation model. It does not automatically establish business identity, causality, or order.

Mercury carries several identities through the order path:

• command_id identifies one acceptance intent and its canonical request hash;
• (order_id, aggregate_version) identifies one ordered aggregate transition;
• (source, event_id) identifies one immutable published fact;
• causation_id names the direct command or event that produced the next record;
• correlation_id groups the broader checkout journey;
• effect_key = (order_id, reservation_version) identifies the inventory outcome; and
• trace_id groups one observation attempt, which may change on replay.

Collapsing them loses information. Reusing trace_id as an idempotency key can combine distinct effects. Regenerating event_id on retry makes one fact look like several. Treating a time-sortable UUID as aggregate order lets independently generated values compete without one aggregate authority.

RFC 9562 defines several UUID layouts. UUIDv7 places a Unix-epoch millisecond timestamp before random bits, improving time-oriented sorting and often index locality. The RFC also discusses counters, clock rollback, monotonic generation, collision resistance, privacy, and distributed-node considerations. Those are identifier-generation properties. A v7 value created after observing another value is not thereby a proof of causality, and two nodes can generate sortable values whose embedded times reflect skew.

Identifier design trades off:

• locality: random keys can scatter B-tree inserts; time-ordered keys improve locality but can create a hot right edge or reveal volume/time;
• coordination: central sequences give dense order but add an authority and bottleneck; decentralized IDs avoid that hop but weaken ordering claims;
• privacy: embedded time, node, tenant, or count information can leak operational facts;
• size: wider keys increase indexes, cache footprint, comparisons, transport, and dedup storage;
• lifetime: a key retained for 24 hours cannot recognize a seven-day replay; and
• scope: uniqueness within one source is different from uniqueness across every environment and tenant.

The effect’s authority should enforce semantic identity even when transport IDs differ, as Chapter 32 established.

Last-write-wins is a conflict policy, not a law of time

“Last” in last-write-wins means “the write selected by a timestamp and tie-break policy,” not necessarily the latest user action, latest causal update, or correct business result.

Consider two disconnected updates to an address:

device A: set city=Nairobi, source clock 10:03:05.900
device B: set postal_code=00100, source clock 10:03:05.400

Replacing the entire object with A because its timestamp is greater can discard B even if the updates touched independent fields. Conversely, merging fields can violate an invariant when the fields must change together. A retry can acquire a fresh server timestamp and defeat a more recent semantic version.

Last-write-wins can be acceptable for a cache hint, presence signal, or explicitly lossy preference where one deterministic value is better than coordination and the loss is bounded. State the policy:

• timestamp source and uncertainty;
• conflict unit: object, field, key, or operation;
• deterministic tie-break;
• maximum offline/replay horizon;
• consequence of discarding a concurrent value; and
• audit or reconciliation path.

For inventory, payment, authority transfer, and schema migration, prefer an explicit version or invariant-preserving operation. The counterexample matters: better clock synchronization makes timestamp selection more plausible, but it does not turn an application policy into the right invariant.

Watermarks are statements about expected progress

Chapter 32 used a watermark to decide when a window was complete enough to emit. The time distinction now becomes sharper. Event time comes from a domain source; ingestion and processing times come from the pipeline; the watermark comes from a progress policy.

If Pulsepipe has seen maximum event time 12:10 and applies a two-minute out-of-orderness allowance, it may advance an operational watermark to 12:08 for the declared source class. An event stamped 12:07 that arrives afterward is late relative to that policy. It is not logically impossible. A disconnected device, restored partition, clock fault, replay, or source bug can still produce it.

Watermark design therefore records:

• event-time source and validation;
• per-source or per-partition progress and idleness rules;
• measured lateness distribution and outage envelope;
• retained state and result-latency cost;
• correction, retraction, adjustment, quarantine, or drop behavior; and
• meaning of provisional and final to the consumer.

Never use a watermark to authorize destructive expiry of the only source evidence unless its completeness claim is strong enough for that separate invariant.

Lease safety needs bounded assumptions and a fence

A lease gives a holder a right for a bounded term. It can reduce coordination during the term and allow an authority to move on after the term, but only under a stated clock and failure model.

Mercury models a 30-second allocation lease. For the exercise, the team assumes:

• clock uncertainty allowance: 75 ms at each side of the authority relationship;
• delivery/action allowance before the holder can stop using the right: 120 ms;
• free-running drift bound during a 1,800-second impaired synchronization interval: 50 ppm; and
• authoritative storage rejects any operation below the current epoch.

The drift allowance is:

50 × 10^-6 × 1,800 s = 0.09 s = 90 ms

The intentionally conservative teaching sum is:

early-stop margin = 2 × 75 ms + 120 ms + 90 ms = 360 ms
holder stop point = 30,000 ms - 360 ms = 29,640 ms

This is modeled arithmetic, not a universal lease proof. Some protocols use one authority’s clock, uncertainty intervals, acknowledged grant times, renewal rounds, bounded delay, or no client clock in different ways. Pauses can exceed the action allowance. Network delay is not generally bounded on an asynchronous network. A partitioned old holder may ignore a local timer. The protocol must define which assumptions are safety requirements and what happens when they are violated.

The fencing token is the final defense at the resource. When epoch 42 is installed, an epoch-41 operation is rejected even if the former holder’s local clock or process state says the lease remains valid. Expiry limits how long authority waits; fencing prevents stale authority from acting after replacement.

Reconstruct causality without forging a global story

An audit should preserve multiple orders instead of sorting everything by display timestamp and calling the result history.

For Mercury, a causal evidence row contains:

record_id: immutable audit record identity
source: emitting authority and software/schema version
occurred_at: domain/wall time, source, precision, uncertainty
recorded_at: authority commit time where available
event_id: immutable fact identity
causation_id: direct predecessor command or event
correlation_id: broader journey
aggregate_id + aggregate_version: entity order
authority_epoch: writer generation
source_position: log/partition position when applicable
effect_key + effect_state: durable business outcome
trace/span links: observation evidence, not authority

To investigate order o-1842, begin at the authoritative effect and follow semantic identity backward: reservation effect → consuming event → outbox record → order mutation → acceptance command. Check aggregate versions and epochs. Then place wall-clock observations with their uncertainty and look for missing edges. A clock-order anomaly becomes a property of evidence, not permission to rewrite the causal chain.

Audit storage itself must handle duplicates, late records, missing spans, schema changes, and retention. Causal links can be absent when instrumentation fails. A cycle in causation_id is a data-quality defect or invalid model. A disconnected component is “unknown relation,” not automatically concurrent. Preserve raw identities so a later repair does not fabricate certainty.

Stronger order spends latency, metadata, and failure budget

Ordering mechanisms move cost among coordination, waiting, metadata, authority concentration, and application conflict handling.

The Spanner paper demonstrates one carefully engineered use of a time API that exposes uncertainty and waits out uncertainty to support external consistency. The lesson is not that ordinary wall clocks can replace coordination. The implementation invests in time references, monitoring, bounded uncertainty, replicated state, and commit protocol; larger uncertainty directly increases waiting.

A design that replaces per-key versions with one global sequence may simplify replay while lowering write availability during sequencer impairment, adding cross-region latency, and concentrating index traffic. A design that removes causal metadata may improve bytes per record while making concurrent conflict detection impossible. State the trade rather than describing stronger order as free correctness.

Operate the time dependency through change

Clock and order mechanisms need rollout and incident controls like any other critical dependency. Monitor wall-clock offset and uncertainty by source and host class, synchronization state, last successful source update, frequency correction, clock-step or rollback events, monotonic read failures, leadership epoch churn, stale-token rejections, version conflicts, and causal-link gaps. Segment these signals by zone, hypervisor, image, and time-source path; a fleet average can hide one partition whose timestamps poison last-write policy or retention.

Mixed versions are especially dangerous. Suppose the new writer adds authority_epoch, but an old storage path ignores it. The field exists in traces while the actual authority remains unfenced. Rollout therefore starts at the rejecting boundary: storage understands and records the token, shadow validation reports stale candidates, all writers carry it, and only then does the system depend on rejection for safety. Rollback must not restore an implementation that accepts lower epochs after a higher epoch has committed.

Changing identifier formats also needs a compatibility period. A move from random UUIDs to time-sortable UUIDs can alter index locality, page-split behavior, partition heat, log compression, privacy exposure, and downstream parsers. Store the identifier as opaque bytes or text according to the interface contract; do not make consumers parse embedded time unless the schema promises that meaning. Measure write distribution and index service demand before and during rollout.

Test temporal failure deliberately:

• step or slew wall time within the platform’s supported test environment while monotonic duration checks continue;
• isolate one time source and let declared uncertainty grow;
• pause an old authority across lease expiry and install a higher fenced epoch;
• replay older events with newer ingestion time and verify aggregate order does not change;
• generate high-rate identifiers inside one clock tick and across clock rollback; and
• remove causal headers at one hop, verifying the audit marks a gap rather than inventing an edge.

The stop rule is correctness-first. If clock uncertainty exceeds the proven bound for a time-dependent ordering protocol, affected ordered work waits, rejects, or uses another authority according to design. It does not silently continue with a best-effort timestamp while dashboards remain green.

Applied time contract for Mercury inventory

Mercury’s corrected record is compact enough to use in a design review:

user/civil time:
  field: occurred_at_utc
  source: order authority after validation
  use: display and retention policy; never mutation precedence

duration/deadline:
  source: local monotonic clock
  use: remaining operation budget within one process generation

aggregate order:
  key: inventory_item_id
  rule: expected_version v commits only as v+1 at inventory authority

authority order:
  token: allocation_epoch
  rule: storage atomically rejects token below highest accepted epoch

retry identity:
  key: command_id + canonical_intent_hash
  rule: same intent returns stored outcome; different intent conflicts

causal evidence:
  fields: event_id, causation_id, correlation_id, aggregate_version,
          authority_epoch, effect_key, source_position

lease:
  term: 30 s modeled
  holder margin: 360 ms under stated teaching bounds
  final safety: storage fencing; clock expiry alone is insufficient

Two changes do the intellectual work. updated_at no longer chooses inventory truth, and lease expiry no longer claims to stop a paused former leader by itself.

Time and causality drills

Classify the clock job. For latency measurement, a user-visible timestamp, a seven-day retention date, an aggregate mutation, a leadership transfer, an event-time window, a trace, and a duplicate effect, choose the time/order/identity source. Reject any answer that uses one timestamp for all eight.

Reproduce the inversion. Calculate the 1,080 ms and 970 ms wall readings, 40 ms actual causal gap, and 110 ms displayed inversion. Then vary offsets until the display order agrees. Explain why agreement still does not prove causality.

Repair last-write-wins. Given two offline updates to one object, state the invariant and conflict unit. Choose among version rejection, field merge, operation merge, explicit conflict, or lossy timestamp selection. Name a concurrent counterexample that breaks your choice.

Compare causal metadata. For vectors [3,1], [2,4], and [3,4], identify concurrency and happens-before. Then add a third participant and price the metadata. Decide whether direct causation links or per-aggregate versions solve the actual audit job more cheaply.

Prove expiry assumptions. Reproduce 90 ms drift and the 360 ms teaching margin. List every assumed bound. Inject a 900 ms process pause, an unbounded partition, clock loss of synchronization, renewal acknowledgment loss, and a late epoch-41 write. Show which defense handles each and where safety becomes unprovable.

Audit without sorting fiction. Starting at one durable effect, reconstruct the command, source commit, outbox, broker record, consumer attempt, and effect. Mark authoritative order, causal edges, wall-clock observations, uncertainty, duplicates, and missing evidence separately.

Durable rules for time, order, and identity

1. Name the temporal question before choosing a clock.
2. Measure duration and local deadlines with a monotonic source whose platform behavior is known.
3. Carry wall time with source, precision, uncertainty, and semantic meaning; do not promote it silently to authority.
4. Scope sequences and versions to the invariant they order.
5. Fence authority generations at the protected state boundary, not only at the coordinator or client.
6. Treat causality as a partial order; a convenient total sort is not discovered truth.
7. Give identifiers a namespace, generation rule, lifetime, canonical intent, privacy review, and collision policy.
8. Describe last-write-wins as an explicit loss/conflict policy.
9. Treat watermarks as progress claims with late-data behavior, not completeness proofs.
10. Base leases on bounded assumptions and pair authority transfer with fencing.
11. Preserve event, causation, version, epoch, position, and effect evidence so audits survive replay and skew.
12. Buy stronger order only when its correctness value exceeds coordination, waiting, metadata, concentration, and recovery cost.

Time uncertainty also changes failure interpretation. A missed heartbeat may indicate a dead process, a slow process, a delayed path, an overloaded observer, or a paused runtime. Chapter 34 turns that ambiguity into failure states, dependency budgets, degraded behavior, and recovery-capacity controls.

Evidence and transfer limits

• Leslie Lamport’s primary paper, “Time, Clocks, and the Ordering of Events in a Distributed System”, establishes happens-before and logical-clock reasoning. It does not make a scalar logical timestamp a concurrency detector or physical clock.
• RFC 5905 specifies NTPv4 and defines offset, delay, dispersion, jitter, and synchronization distance. A production clock’s actual error also depends on sources, topology, asymmetry, implementation, operating state, and monitoring.
• The Open Group’s clock_gettime specification defines CLOCK_MONOTONIC semantics for POSIX systems. Suspend behavior, virtualization, resolution, and cross-host use require platform-specific validation.
• The Spanner OSDI paper documents a system that exposes time uncertainty and integrates it with replicated transaction protocol. It is evidence that bounded-time ordering has infrastructure and waiting cost, not permission to infer external consistency from ordinary timestamps.
• RFC 9562 defines UUID formats and discusses monotonicity, clock rollback, collision resistance, privacy, distributed generation, and sorting. UUID uniqueness and time locality do not define Mercury’s semantic effect identity.
• Gray and Cheriton’s lease paper analyzes leases for fault-tolerant cache consistency under its model. Lease safety must be re-derived for Mercury’s clocks, pauses, delivery, authority, and storage fencing.
• Demirbas and Kulkarni’s hybrid logical-clock work describes combining logical causality with values close to physical time for auditability. It does not remove application invariants or uncertainty.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/time-causality/ reproduces the 110 ms display inversion, 90 ms drift allowance, 360 ms early-stop margin, vector comparisons, and epoch-41 rejection. These are deterministic teaching calculations, not measured clock or protocol performance.