Skip to content

Production Data Systems Handbook / Chapter 23

Modeling Writes: Commands, Transactions, Events, and Side Effects

Design write paths that preserve invariants, establish identity, handle retries, publish facts, coordinate side effects, and expose repairable failure modes.

The Cancellation That Was Only Half True

At 10:02 a support agent accepts a customer’s request to cancel order O-731. The order service changes the status and returns “cancellation accepted.” At 10:03 the warehouse buys a shipping label from a carrier. At 10:04 the payment service asks the provider to void the authorization and receives no response.

Is the order cancelled?

The database can answer only part of that question. It can prove that the order service accepted a command and changed facts under its control. It cannot make the carrier forget a shipment request or make a payment provider participate in the same commit. If the model has only active and cancelled, those external obligations disappear behind a reassuring word.

A write path should separate three things: the command, which expresses an intention; the transaction, which establishes facts inside one authority; and the effects, which carry consequences across boundaries. The design is honest when each can be inspected independently and the product never presents an unfinished effect as a finished fact.

A conceptual write flow shows a command and idempotency key passing validation and authorization, then a local transaction that commits state with an outbox record, followed by event relay, consumer deduplication, and an external side effect.
The local transaction can commit state and an outbox obligation atomically. Everything after that boundary has its own state, retry, and uncertainty; consumer deduplication does not make an external receiver atomic.

Define What the Command Is Allowed to Promise

CancelOrder is more useful than update order. The business verb brings its questions with it. Who may cancel? Which order states still permit it? Does cancellation mean stop new fulfillment work, recover an existing shipment, release inventory, void an authorization, issue a refund, notify the customer, or all of these? Which outcome may the API promise immediately?

The command record carries the actor, target, reason, authority source, request and correlation ids, expected order version, idempotency key, and parameters that affect the decision. A retry with the same key and same parameters should find the same command. Reusing the key for a different order or reason is a mismatch, not a new interpretation. The retention window must cover realistic client retries and repair; an in-progress duplicate needs a stable pending answer.

Identity is only the entrance to the design. The command handler still passes three different gates.

Validation decides whether the request is coherent: the order exists, the reason is supported, and any required customer instruction is present. Authorization decides whether this actor may cancel this order now; an agent may need tenant scope, a delegated role, or a break-glass reason. Invariant enforcement decides which state transitions are legal under concurrency. An order already handed to a carrier may require a return workflow rather than cancellation. A stale support screen must not overwrite a newer shipped version with cancelled.

The last gate needs a mechanism at the authority: a conditional update on the expected version, a lock, a constraint, a serializable transaction, or a single writer. Reading ready_to_pack and later writing cancelled is not safe if another command can ship the order between those operations.

The immediate promise should match the boundary. If external resolution is outstanding, cancellation_pending or “cancellation accepted” is true; “cancelled and refunded” is not. A final cancelled state can require the specific effect outcomes the product considers complete.

Commit the Local Truth and Its Obligations

Suppose the order service owns order lifecycle and the reservation in one transactional store. One short transaction can do the following work:

  • claim the idempotency key for CancelOrder;
  • verify the expected order version and cancellation invariant;
  • move O-731 from ready_to_pack to cancellation_pending;
  • prevent new local fulfillment assignment and release the local reservation when policy permits;
  • append an audit record with actor, reason, previous state, and command id; and
  • write an outbox event recording that the cancellation was accepted at a new source version.

Either those facts and the publication obligation commit together or none of them does. A deadlock or confirmed rollback can be retried through the command identity. If the caller times out around commit, the service looks up the command before repeating work and returns the recorded state.

The transaction should not wait on the carrier, payment provider, email service, webhook target, or search index. Remote latency lengthens lock time, and remote success cannot be rolled back if the local commit later fails. A cross-service invariant needs a workflow with visible intermediate states, not a wider controller method.

This boundary also determines the event name. OrderCancellationAccepted is a fact the source can defend. OrderCancelled would be premature if the book’s business meaning includes shipment recovery or payment reversal. Commands ask for work; events report committed facts. Blurring the two encourages consumers to treat an intention as an outcome.

The Outbox Preserves the Arrow

Writing the order and then publishing an event leaves a fatal gap: the database can commit and the process can die before the publish. An outbox row written in the order transaction makes publication a durable obligation. It names the event id, order id, source version, type, payload, command or trace id, and creation time. A relay reads committed rows and publishes them.

The relay may publish twice when it sends successfully but loses the acknowledgement. The outbox prevents a forgotten event; it does not create exactly-once execution. A consumer records the event id in an inbox or applies only a newer source version. Its inbox record and local state change should commit together so that a crash cannot separate the effect from the proof of deduplication.

For the warehouse, consuming OrderCancellationAccepted may create a durable local command to stop fulfillment. For search, the consumer updates the order projection only if the cancellation version is newer. For analytics, the event may revise a count. These are separate consumers with separate completion rules. Replay and retention must be long enough to rebuild them, and pruning must not erase the evidence needed for the promised recovery window.

An inbox protects the consumer’s local transaction. It does not prove that a carrier API, payment provider, or webhook receiver performed an effect once. That uncertainty begins at the next boundary.

Give Every Side Effect a State of Its Own

The warehouse consumer receives the event after a label purchase has already begun. Its local effect record says which operation is required, the stable receiver key, the last attempt, and one of a small set of states such as pending, attempting, succeeded, declined, uncertain, or manual_review.

If the carrier confirms cancellation, fulfillment can close its obligation. If the carrier reports that the parcel was collected, the order workflow needs a return or interception policy; relabeling that outcome “cancelled” would hide customer and cost consequences. If the call times out, the worker records uncertain and checks carrier state before an unsafe retry when the receiver cannot deduplicate the request.

Payment has a different completion rule. An uncaptured authorization may be voided. A captured payment may require a refund, whose settlement can take its own path. The provider idempotency key, provider reference, requested amount, attempt state, and reconciliation evidence belong in durable data. “No response” is not “failed.” It may mean the provider acted and the local system does not yet know.

Email should normally wait for the state the message claims. A cancellation-confirmed email sent while the carrier effect is uncertain can make the product lie in prose even if every database field is internally consistent. Give the delivery a deterministic identity, suppress it if it becomes stale, and retain enough attempt state for support. Webhooks likewise need delivery ids, authentication appropriate to their contract, retry schedules, and dead-letter repair.

Search indexing is derived work: retry it, measure lag, reject older source versions, and rebuild it from authoritative facts. Search failure must not roll the order back. It should, however, affect whether a support agent may act from the stale projection.

Side-effect policy follows harm. A projection can often be replayed automatically. A duplicate email is embarrassing. A duplicate refund, shipment purchase, access grant, or deletion request can be costly or irreversible. The latter effects deserve stronger preconditions, queryable uncertainty, reconciliation, and an operator path.

Compensation is another command, not a rollback across time. Voiding payment, releasing inventory, requesting a return, or restoring access can fail independently and leaves its own audit history.

Finish the Workflow Deliberately

The order service needs a rule for leaving cancellation_pending. It might require fulfillment to confirm stopped or return-required, payment to confirm voided or refunded, and every security-critical effect to complete. Search and analytics may remain derived obligations outside the customer-facing completion rule. Product policy decides the set; the data model should make it explicit.

That usually requires a coordinator or state transition driven by effect results. Each result is itself a committed fact with a source identity. The coordinator advances the order only when the current version and required outcomes permit it. A late ShipmentPurchased result may move the workflow to return_required instead of back to active. An operator override records who acted, why, and which unresolved obligations remain.

The distinction prevents one success from erasing another failure. CancelOrder can be accepted, carrier cancellation can fail, payment reversal can succeed, email can remain suppressed, and search can lag—all at the same time. Those are not contradictory statuses. They describe different authorities.

Operational signals should follow those states: age of the oldest unpublished outbox row, consumer lag and dead letters, effects stuck in pending or uncertain, compensation failures, source-version conflicts, and reconciliation drift. A runbook begins from an observable state such as “payment void uncertain for 20 minutes,” then says how to decide, repair, and prove completion. “Restart the worker” is not enough.

Preserve the Explanation of What Happened

At 10:20 support should be able to reconstruct the path: agent A-44 submitted command C-92 under a named authority and reason; order version 418 entered cancellation_pending; outbox event E-551 was relayed; fulfillment reported shipment_already_collected; payment void P-731-V remains uncertain; the confirmation email is suppressed; and the workflow awaits a return decision.

An audit trail answers who changed what, when, why, from which previous value, through which command, and with which downstream consequences. It may need actor type, authority source, command and request ids, safe before-and-after values, reason or ticket, source version, and correlations to events and effects. It should not copy secrets, credentials, tokens, or unnecessary personal data simply because they passed through the handler. Audit data has its own access, retention, redaction, and export rules.

Domain events are not automatically sufficient audit records. A projection may need only OrderCancellationAccepted and a source version. An investigator may need the actor, prior state, policy decision, override reason, and effect references. One record can serve both jobs only when its contract genuinely meets both needs.

Keep a Failure Ledger, Not a Happy-Path Diagram

The flow figure shows the main boundaries. A design review also needs a record of what happens when the system stops at each one. Use a form with room for causality instead of squeezing paragraphs into cells:

Command and completion promise:
Actor, target, authority, reason, command id, idempotency rule:
Facts and invariants owned by the local transaction:

For each step or effect:
  Durable state before the attempt:
  Fact or obligation established on success:
  Failure or ambiguous outcome:
  What the caller or user is told:
  Retry precondition and stable identity:
  Compensation, if a real compensating action exists:
  Monitor and accountable owner:
  Repair or reconciliation action:
  Durable proof that repair is complete:

Audit correlations and sensitive-data limits:
Event meaning, source version, ordering, retention, and replay:
States exposed by the product:
Evidence that would reverse this design:

Fill the ledger for command receipt, authorization, invariant enforcement, local commit, outbox relay, each consumer transaction, each external call, workflow completion, and reconciliation. The answers should differ. A constraint conflict can return a current-state error. An unknown commit result requires command lookup. A relay failure leaves an outbox obligation. An old search event is rejected by source version. An uncertain carrier or payment result requires receiver evidence before another risky attempt.

Then transfer the design to checkout. Define exactly where PlaceOrder establishes identity, reserves inventory, records the accepted price, and writes its outbox event. Decide whether payment is authorized before or after that transaction and what the user sees while the outcome is pending. Add idempotency to the create-order endpoint: specify the key scope, request fingerprint, uniqueness rule, in-progress response, parameter-mismatch behavior, result replay, retention window, and the durable record that resolves a timeout after commit.

Break both commands on paper. Stop before and after the local commit, after publish but before acknowledgement, after the consumer’s local commit, and while an external outcome is unknown. For every stop, name the fact that is true, the obligation still open, and the record an operator will use next.

A production write path does not make every consequence synchronous. It makes partial truth impossible to mistake for completion. That is what gives the read side an honest state to serve.