Case Study: Event-Driven Notifications and Workflow

Six Months of Reminders Leave at Once

A subscription company replays six months of PaymentFailed events to rebuild a billing projection. The replay succeeds. So does a notification consumer that nobody disabled.

Old failures become fresh queue work. One customer paid five months ago; another has since opted out of non-critical email; a third closed the account. All three receive reminders. Provider retries deliver several messages twice. Support can see the complaints but cannot explain which event, workflow, preference decision, or attempt produced each send.

Every infrastructure component behaved plausibly. The log returned retained facts. The consumer created work. The queue redelivered after timeouts. The provider accepted requests. The failure lies in the meaning between them: a historical fact was treated as current permission to contact a person.

That mistake gives the chapter its governing rule. A user-visible notification is a durable promise, not a callback attached to an event. The system must be able to show why it intended to send, whether it was allowed to send at that moment, whether the visible effect had already happened, and what remains to be done.

Follow One Reminder Through the Failure

Invoice I-204 failed on 3 February. The billing service committed the new invoice state and a PaymentFailed event. A consumer opened reminder workflow W-771, scheduled to evaluate the invoice three days later. Before the timer fired, the customer paid. An InvoicePaid event moved W-771 to canceled_paid. No notification intent was created.

In July, the backfill reads the February failure again. The old consumer does not ask whether it is rebuilding a projection or handling live traffic. It creates a new intent and places a send job on the queue:

February: PaymentFailed(I-204)
              |
              +--> W-771 waiting --> InvoicePaid --> canceled_paid

July replay: PaymentFailed(I-204, mode=rebuild)
              |
              +--> bad consumer creates N-992
                         |
                         +--> queue delivery D-1
                                  |
                                  +--> provider accepts reminder

The trace separates records that the original design had blurred together. The event is evidence that a payment failed in February. The workflow owns the time-dependent promise to remind unless payment occurs. The notification intent names one proposed user-visible effect. The queue delivery only coordinates an attempt. The provider call crosses the irreversible boundary.

Once those meanings are explicit, the primitive choices become ordinary.

A log retains business facts for multiple consumers and permits replay. It needs compatible schemas, retention, partitioning, lag monitoring, and a declared replay policy. It cannot decide that an old fact should have a new external effect.

A workflow owns a decision that survives time and can be canceled. “Wait three days, then remind unless this invoice is paid” has a timer, state transitions, and a terminal condition. A delayed queue message alone cannot explain what should happen after payment, a policy change, an operator pause, or a changed reminder schedule.

A queue coordinates execution: evaluate this workflow, render this intent, call this provider, or process this callback. It needs leases or visibility timeouts, concurrency controls, bounded retry, backoff, and dead-letter handling. A queue message is not the notification record. Losing it should be detectable from the still-open intent; redelivering it should not create a second visible effect.

The useful design-review question is therefore concrete: which durable record tells a responder what should happen next? “The message is in the queue” is an execution fact, not an answer.

Give the Promise Its Own Identity

The system supports security alerts, payment reminders, weekly digests, and account-restriction notices. These families do not share one correctness policy. A security alert loses value quickly but may justify an emergency channel. A digest can wait and should fail closed when preference truth is unavailable. A payment reminder must stop when the invoice is paid. A restriction notice may value durable production and attempt evidence even when delivery cannot be proved.

Each workflow can create a notification intent containing the recipient, channel, template family, business object, reason, reminder stage, semantic send window, priority, and policy class. Delivery attempts hang from that intent and record the render version, provider request key, provider response, error class, retry number, and terminal outcome.

The intent also owns a durable idempotency key. For the first email reminder on I-204, a suitable scope might be:

recipient U-18
invoice I-204
reason payment_failed
stage reminder_1
channel email
policy window 2026-02-06

An event ID or queue message ID is too weak. Replay can produce new delivery identifiers for the same effect, while a later reminder stage is a legitimate new effect. The key must match what the recipient would perceive as a duplicate.

The sender reserves that key durably before calling the provider. If two workers race, only one may open the attempt. If the provider accepts the request and the worker crashes before recording success, the result is unknown, not “failed.” The repair path uses the stable provider request key, provider lookup, callback evidence, or reconciliation before resending.

Ordering is similarly narrow. The company does not need a global order across every notification. It needs transitions for I-204 to respect the invoice’s state: failure, scheduled reminder, payment, cancellation. A current preference decision must dominate an older send proposal. Analytics can tolerate other orderings and repair them with event time and watermarks. Naming the ordering domain avoids buying global coordination for a local promise.

Permission Is Read at the Last Responsible Moment

Even a valid intent is not permanent authorization. Immediately before the provider call, the sender reads the current invoice state, account state, channel eligibility, consent and preference record, suppression list, regional policy, template status, and route health. It records the versions or decision evidence used.

For N-992, the paid invoice is enough to suppress the reminder. For a digest, a later opt-out is enough. For a closed account, account policy may suppress every non-mandatory message. The February event remains true, but truth about the past is not permission in July.

Unavailable authority must have a policy too. Non-critical mail can wait or suppress when the preference service cannot answer. A security or account-access message may use a narrowly defined emergency path, with the policy class and evidence recorded. Silently sending because a queue item exists lets infrastructure choose product and legal policy.

This check belongs close to the side effect because delay is normal. Workers crash, queues redeliver, workflows wait, and consumers catch up. Authorization copied into an old event grows stale while the work is in motion.

The Workflow Makes Time Inspectable

The payment reminder advances through a small state machine. waiting_for_due_time can move to evaluation, cancellation after payment, expiry, or manual pause. Evaluation reads current invoice and policy truth. It then creates an intent, suppresses with a reason, or cancels. An allowed intent moves through ready, attempting, sent, unknown, retrying, failed, or paused states.

The state names are less important than their evidence and guards. Every transition records its cause. Timers are queryable. A paid invoice cannot leave evaluation with a ready-to-send intent. A paused route cannot begin a provider attempt. An unknown provider outcome cannot become a fresh attempt until lookup or policy permits it. Terminal suppression remains visible to support rather than masquerading as missing work.

This is where a durable workflow earns its cost. Support can answer why a reminder did not arrive. On-call can count timers that have not fired, promises older than their SLO, unknown provider outcomes, and workflows paused by route. Product can change the reminder interval without distributing timing rules through several consumers and cron jobs.

Retries Have a Product Deadline

A network failure may be worth retrying. An invalid address is not. A preference suppression waits for policy change, not exponential backoff. An unknown provider outcome requires investigation before another external call. The retry policy classifies these outcomes and gives each notification family a maximum useful age.

Once that age expires, the system needs a terminal disposition or a human owner. A dead-letter queue is only a transport holding area. Its entries must still join to intents whose class, age, business impact, retry safety, and owner are known. A DLQ nobody reads has converted an open product promise into quiet failure.

Operators also need controls narrower than “stop notifications”: pause one template, channel, provider route, campaign, workflow type, or replay job without deleting state. A pause limits damage; it does not resolve the affected intents. Their eventual suppression, retry, customer communication, or manual closure still needs an explicit decision.

Repair the Replay Without Replaying the Harm

The incident response begins by pausing the side-effecting consumer and the replay job, while leaving the log, workflows, intents, attempts, and provider references intact. Responders record the replay source range, purpose, mode, owner, allowed consumers, side-effect policy, and verification query.

They then join every intent created by the replay to current business and permission state. The cases separate naturally:

• intents not yet attempted are suppressed when the invoice is paid, the account is closed, permission is absent, or the message has expired;
• duplicate attempts are matched by semantic key and provider reference, with no further send;
• unknown outcomes are resolved through provider evidence before retry;
• users already contacted incorrectly are grouped by message class and impact for support, policy, and customer-communication decisions;
• genuinely missed, still-valid reminders require an explicitly authorized repair mode rather than another general replay.

After containment, replay defaults to rebuilding state and evidence. A consumer that may create intents must recognize live, projection rebuild, workflow restoration, and side-effect repair as different modes. The repair mode names its approved scope and expiry. “This consumer should be disabled during backfills” is a memory, not a control.

Operate the Promise, Not the Worker

CPU, queue depth, and provider error rate remain useful, but none reveals whether user-visible promises are aging silently. The primary operational view starts from intents: creation rate, eligibility results, suppression reasons, dedupe decisions, terminal outcome, and age against the SLO for each notification class.

Workflow timer lag shows whether decisions are waking up. Attempt age and callback lag expose provider uncertainty. DLQ age reveals abandoned execution. Preference lookup failures and emergency-path use show policy risk. Replay mode and active side-effecting consumers must be visible together.

Correlation follows the same chain that support needs: event fact, workflow, intent, queue delivery, attempt, provider request, and callback. For one user, a responder should be able to read why the system proposed contact, which current authority allowed or denied it, what the provider accepted, and what remains unresolved.

That evidence also prepares the next case study. Notification promises are operable only if the metrics, logs, traces, and retained records can be queried under incident load without their own cardinality and retention design collapsing.

The Notification Contract

Before adding a message family, write one page that answers these questions in ordinary language:

1. What fact, command, timer, or operator action can begin the promise?
2. What visible outcome is promised, and when does it lose value?
3. Which log, workflow, intent, and queue records own facts, decisions, effects, and execution?
4. Which current authorities—business state, consent, preference, suppression, template, region—must agree at send time?
5. What semantic key identifies a duplicate, and what ordering domain must be preserved?
6. Which errors retry, for how long, and who owns unknown or dead-lettered work?
7. Which replay modes are side-effect free, and who can authorize a repair send?
8. Which pause control, query, alert, support view, and escalation closes the operating loop?

For an incident drill, replay the six-month PaymentFailed range in rebuild mode. Prove that no provider attempt can open. Then inject three variations: I-204 was paid before the reminder, the preference service is unavailable, and the provider accepted a request whose callback never arrived. For each variation, identify the workflow transition, intent disposition, evidence, and next safe action.

The architecture is ready when those answers live in durable state. A log may remember that a payment failed, a workflow may remember that a reminder was canceled, and a queue may try work more than once. None of them alone owns the promise. The promise becomes defensible only when past fact, current permission, visible effect, and unfinished work can be told apart.