State Machines and Workflow Modeling

The cancellation that did not cancel

An order is waiting for a payment provider. The buyer cancels it. The application changes status from payment_pending to cancelled and asks the provider to void the payment.

Seconds later, a payment_captured webhook arrives.

The event is not necessarily late. The provider may have captured the payment before it received the void request; the two messages merely reached this application in the opposite order. The cancelled label now claims more than the system knows. Should the webhook be ignored because the order is terminal? Should it revive the order as paid? Has inventory already been released? Who owes the buyer a refund?

A state machine is useful here because the problem is movement, not labels. It makes the system say which event may move an order from one condition to another, which facts make the move legal, and what evidence the move must leave behind. A workflow model carries that reasoning across time: remote calls, retries, timers, crashes, and repair.

The first repair is conceptual. After a cancellation request during payment, the order is not yet cancelled. It is cancellation_pending until the payment outcome is known. That intermediate state is not implementation decoration; it changes which events are legal, what the buyer should see, and which work operators must watch.

Give each fact one owner

The order owns the buyer-facing lifecycle. It does not need to absorb every fact in checkout:

• an inventory reservation owns the stock hold and its expiry;
• a payment attempt owns provider request and outcome identities;
• the order owns whether checkout may advance, must unwind, or is finished;
• an outbox record owns the promise to deliver work after a local commit;
• transition history records how the order reached its current state.

This separation prevents a familiar explosion of states such as payment_request_sent, payment_webhook_received, and release_inventory_message_retried. Those are useful facts, but they do not all change what may happen to the order.

A state earns its place when it changes at least one consequential answer: which commands are legal, what the user should see, which timeout applies, what recovery is possible, or what operations must inspect. By that test, cancellation_pending belongs in the order lifecycle. An outbox attempt count does not.

The vocabulary is small:

• a state is the order’s current lifecycle condition;
• an event asks the lifecycle to move;
• a guard decides whether that move is legal using current durable facts;
• a transition is the accepted movement and the local facts it records;
• a side effect is work outside that local transaction.

An invariant is broader than a guard. “A delivered order never returns to an active checkout state” must survive every ordinary event. “This provider capture belongs to the current payment attempt” is a guard on one transition.

Write the disputed paths first

The happy path is easy to remember. Begin the transition table with the paths on which people or systems could disagree. This table earns its rectangular form because every row shares the same five attributes and will become a test case.

The table also exposes a policy choice. This business automatically refunds a capture discovered during cancellation. A regulated product, a high-value order, or an inconsistent amount may instead move to manual_review. The mechanism does not choose that policy; it makes the choice visible.

Every missing cell has meaning. delivery_confirmed from inventory_reserved is illegal. A second copy of an event already applied is a stable replay. An event for an older payment attempt is stale. An outcome whose amount conflicts with local expectation needs investigation. Returning one generic error would erase distinctions the caller and operator need.

Trace one order through time

Follow order 1842. Inventory is reserved, payment attempt pay-7 starts, and the order reaches payment_pending at version 2.

At 10:00:04 the buyer requests cancellation. The command sees no recorded capture, writes version 3 as cancellation_pending, appends a transition event, and creates an outbox item requesting a void. Those writes commit together.

At 10:00:05 the provider’s capture webhook arrives with event ID evt-91. The provider says it captured pay-7 at 10:00:03. The handler does not compare wall-clock arrival order and guess which intent wins. It applies the protocol:

1. evt-91 has not been processed before.
2. pay-7 is the current attempt and its amount matches the order.
3. payment_captured is legal from cancellation_pending.
4. The transition records the capture, moves the order to refund_pending, and promises a refund in one local transaction.

The void outbox item may already be in flight. Its worker must therefore read the current attempt and tolerate “already captured” from the provider. The refund worker uses a stable key derived from the capture, so a retry cannot create two refunds. When the provider confirms that refund, the order moves to cancelled and inventory release becomes durable promised work.

Now send evt-91 again. It is not an illegal transition to panic over and it is not a second capture to apply. It is the same provider fact repeated. The handler acknowledges it and returns the result already associated with that event ID.

Finally, send a capture for pay-6, an abandoned attempt. That is not a replay of the current event and must not move the current order. Store enough detail to reconcile the provider-side fact, then route it to the chosen recovery policy. “Duplicate,” “stale,” and “conflicting” describe different evidence.

Make the protocol executable

Do not let controllers, workers, webhooks, and admin tools assign order state directly. Give them one command boundary:

apply(order_id, event, event_identity, payload) -> result

result is one of:
  applied
  already_applied
  stale
  illegal
  conflict
  needs_review

Inside apply, the sequence is deliberate:

begin transaction
  order = load current state and version

  if event_identity already exists:
      return its recorded result

  rule = transitions[(order.state, event.type)]
  if no rule:
      record rejected event
      return illegal

  evaluate rule.guard against current durable facts
  derive next state and local records
  append transition history
  insert any outbox items with stable uniqueness keys
  update order where version = observed_version
commit

If the versioned update affects no row, another command moved the order after it was read. Reload before deciding whether this event is now already satisfied, stale, illegal, or safe to attempt again. The next chapter examines the concurrency boundary in detail; the workflow’s responsibility is to name the outcomes rather than hide the race.

The storage might begin this simply:

CREATE TABLE orders (
  id BIGINT PRIMARY KEY,
  state TEXT NOT NULL,
  state_version INTEGER NOT NULL,
  current_payment_attempt_id TEXT,
  entered_state_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL
);

CREATE TABLE order_transitions (
  id BIGINT PRIMARY KEY,
  order_id BIGINT NOT NULL,
  event_type TEXT NOT NULL,
  event_identity TEXT,
  from_state TEXT NOT NULL,
  to_state TEXT NOT NULL,
  actor_id TEXT,
  reason TEXT,
  result TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  UNIQUE (order_id, event_identity)
);

CREATE TABLE workflow_outbox (
  id BIGINT PRIMARY KEY,
  order_id BIGINT NOT NULL,
  effect_type TEXT NOT NULL,
  effect_key TEXT NOT NULL UNIQUE,
  payload TEXT NOT NULL,
  attempts INTEGER NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMP,
  delivered_at TIMESTAMP,
  last_error TEXT
);

The current row answers what may happen now. Transition history answers how the order got there and whether an input was already classified. The outbox answers which external work remains owed. These are related facts, not three copies of the same state.

The outbox does not make a database, card network, warehouse, and email service one atomic system. It gives the local system a durable promise it can retry, inspect, and repair. External operations still need their own stable identities: provider event IDs for incoming facts, idempotency keys for outgoing requests, and request IDs that distinguish the current attempt from an abandoned one.

Time is an event source

Waiting states need a clock policy. A reservation may expire, a payment query may be retried, a refund may exceed its service target, and a warehouse lease may lapse.

Model a timer as another event with identity, for example (order_id, cancellation-timeout, scheduled_at). When it fires, check the current state and attempt before acting. A delayed timer for an order that has already been refunded is stale, not permission to reopen it.

Retries belong to delivery records or attempt records, not to vague hope. A worker should know its attempt count, next eligible time, last error, and whether the failure is retryable. Backoff reduces repeated pressure; it does not decide when to stop. The workflow needs a terminal or human-owned outcome for work that cannot succeed automatically.

Operate the states that are allowed to wait

Once waiting and recovery states are explicit, operations can ask useful questions:

SELECT id, state, entered_state_at
FROM orders
WHERE state IN ('cancellation_pending', 'refund_pending',
                'fulfillment_pending')
  AND entered_state_at < ?
ORDER BY entered_state_at;

Age alone is not a diagnosis. The useful view joins the current attempt, undelivered outbox work, attempt count, next retry, last error, and responsible owner. An operator should be able to distinguish “waiting normally,” “retry scheduled,” “automatic retries exhausted,” and “evidence conflicts.” Repair commands pass through the same transition boundary and record actor and reason; an admin console is not an exemption from the protocol.

Reconciliation supplies a second line of defense. Compare local payment facts with provider settlements, shipped orders with warehouse records, and old waiting states with their outbox promises. History is valuable only if the system can turn disagreement into a bounded repair action.

Keep the machinery proportional

An enum plus centralized transition code is enough for a short lifecycle with few timers. Persisted history earns its cost when audit, support, or recovery matters. A long-running workflow engine becomes attractive when durable timers, many remote steps, human pauses, retries, and visibility dominate the problem. A saga may be necessary when several services own independent facts and compensating actions are the only honest response to partial success.

Begin with the lifecycle, not the product category. An engine cannot rescue unnamed states, ambiguous ownership, or missing policies. It can durably execute a protocol only after the protocol exists.

The same reasoning transfers to an approval flow. Votes should remain facts owned by a review round; the document need not acquire states called one_approval_received and two_approvals_received. A resubmission starts a new round so an old vote cannot approve changed content. In a job runner, retry_scheduled differs from dead because one is waiting for automatic work and the other has exhausted that policy. The domains change, but the test for a meaningful state does not.

Put the model under pressure

For the order above, write tests that demonstrate the protocol rather than merely visiting each happy-path state:

• cancellation and capture arrive in both orders;
• the same capture event arrives twice;
• an old attempt reports success after a new attempt begins;
• a refund request succeeds remotely and the worker crashes before recording delivery;
• a stale cancellation timer fires after the order is refunded;
• an operator retries an exhausted refund with an actor and reason;
• two commands observe the same version and only one transition commits.

Then vary one policy. Suppose inventory must remain reserved during refund_pending for at most fifteen minutes, after which the business prefers to release it even if the refund is unresolved. Identify the new timer, the invariant it may relax, the event identity, the transition it permits, and the evidence an operator would need afterward. If the answer is only another status label, the workflow is still implicit.

A trustworthy state field is the visible edge of a protocol. Behind it are the events that may move it, the facts that guard each move, the local record of what happened, the work still owed to other systems, and a recovery path when those systems disagree. Once those are explicit, the next question is whether two legal commands can both act on the same stale state. That is where concurrency-safe design begins.