Threat Modeling for Engineers

The refund console from the previous chapter accepts an operator’s decision, publishes work to a queue, and eventually changes money and customer state. Its data is classified and its access rules look sound. Then someone asks a small question: what happens if a legitimate request arrives twice?

“Replay attack” is not yet useful. It does not say which request can be replayed, who can obtain it, what state permits it, or whether the system would notice. Threat modeling turns that loose concern into a path the team can interrupt and an engineering decision it can verify.

A high-priority threat is not finished until it changes the design, creates implementation work, gains a test, and reaches detection or response. Anything left over is residual risk, owned explicitly rather than hidden in the document.

Model when an assumption can change

Threat modeling belongs at a change boundary. Require it for a new sensitive or critical system, public API, webhook, privileged workflow, payment path, authentication or authorization change, data or tenant boundary, AI capability, infrastructure or delivery path, key, dependency, or vendor. Reopen the model after an incident or near miss: reality has invalidated at least one recorded assumption.

The depth should follow the consequence. A small internal change may need one engineer and a reviewer to update two flows. A high-impact service may need prework, a facilitated workshop, independent challenge, and a formal release record. “Small diff” is not a reason to skip the work if that diff changes an enforcement point.

Begin with artifacts the team can inspect: a context diagram, data-flow diagram, asset and data inventory, identity and authorization model, trust boundaries, external dependencies, and the controls already in place. Bring incident history, vulnerabilities, exceptions, telemetry, and response procedures too. These keep the conversation attached to the system that exists.

For the refund console, the first useful sketch is deliberately plain:

refund operator
  -> browser session
  -> console API
  -> refund command queue
  -> payment worker
  -> payment provider

commerce database -> console API
payment worker -> audit event stream -> security monitoring

Now label identities, protocols, stores, and trust boundaries. Which component authenticates the operator? Which service authorizes the refund amount? Who can publish to the queue? Is the worker trusting the queue’s existence, the message’s producer identity, or the decision encoded in the message? If the team cannot name the identity crossing a boundary, it cannot choose the enforcement point.

Walk the flow before naming categories

Have the implementer narrate one real operation from entry to durable effect. At each boundary, ask what the receiver trusts and how that trust could be misplaced. The refund request exposes several different questions:

• Could an attacker spoof the operator or could a support role be confused with a refund role?
• Could the amount, customer, destination, or approval state be changed in transit or in the queue?
• Could a valid command be replayed, or could an operator deny making it because the audit trail is incomplete?
• Could the reason text or payment identifiers escape through an error, trace, dead-letter payload, export, or support view?
• Could one tenant’s identifier select another tenant’s refund, or could an expensive path exhaust the worker or provider quota?
• Could a compromised runner, package, image, vendor, or privileged insider alter the path while using legitimate authority?

These questions cover the familiar STRIDE concerns—spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege—without mistaking their names for findings. Add prompts that fit the system: supply-chain compromise, insider misuse, tenant isolation failure, and abuse of AI prompts, retrieval, models, data, or tools. A category is a search aid. “Tampering” written in a worksheet is not a threat scenario.

Use other methods when the investigation needs them. An attack tree is useful when a consequential outcome has several routes. Write the attacker’s goal at the root, branch into distinct ways to achieve it, and add the prerequisites beneath each branch. For “issue an unauthorized refund,” the branches may include stealing an operator session, changing the authorization decision, publishing a forged queue message, or compromising the payment credential. Breaking one branch does not prove the root goal impossible.

A PASTA-style risk-driven analysis helps the team spend time on paths with meaningful business consequence instead of generating an exhaustive list of equally weighted possibilities. Adversary-behavior mapping, including ATT&CK where it fits, helps when the design question reaches operations: it connects a plausible path to the telemetry, detection test, and response action needed to observe and interrupt it. The method should sharpen the next decision, not enlarge the meeting.

Keep a small abuse-case library for recurring local paths—cross-tenant object access, replayed state changes, unsafe administrative recovery, sensitive data in diagnostics, forged build output, and untrusted AI input reaching a privileged tool. Reuse the question and the relevant boundary, never a predetermined answer. Each service still has to establish whether the path exists and which control can break it.

Turn “replay” into an engineering scenario

A scenario needs an actor, action, target, enabling condition, and consequence. For example:

An attacker who obtains a previously valid refund command republishes it before the original command’s identity expires. The worker treats the duplicate as new and asks the payment provider to issue a second refund, changing money and customer state while leaving two apparently valid audit events.

That sentence gives the team something to challenge. Can an attacker read the queue? Does redelivery happen during ordinary failure? Does the provider accept the same request twice? Where is the authoritative refund state? Likelihood depends on those facts; it should not be smuggled into a severity label.

Record each path in a form that can survive the workshop:

Scenario: actor + action + target + enabling condition + consequence
Impact and likelihood assumptions:
Required design control:
Implementation tasks and owner:
Verification and evidence:
Detection and response:
Residual risk and acceptor:

The control must survive the ordinary behavior of the system. Queues redeliver, workers restart, and a provider may time out after accepting a request. “Consume each message once” is therefore a brittle promise. Give every refund decision a stable operation identifier, enforce the allowed state transition in the authoritative store, and use the same identifier as the provider’s idempotency key. Authenticate producers and limit their publish scope, but do not ask queue access control alone to provide business idempotency.

Implementation work follows naturally: persist the operation identifier with the decision; make duplicate processing return the recorded outcome; bind amount, destination, tenant, and approval state to that decision; and prevent a closed or reversed case from re-entering the issue path.

Verification should include a duplicate delivered before completion, a duplicate after completion, concurrent delivery to two workers, a worker crash after the provider accepts but before local success is stored, a stale command after account closure, and reuse of an identifier with different refund details. The expected result is one business effect and a stable, explainable outcome—not merely one successful HTTP response.

Detection and response complete the path. Record the operation identifier, actor, case, state transition, provider reference, outcome, and trace identifier without copying the sensitive reason text. Monitor repeated identifiers, conflicting payloads, invalid state transitions, and unusual refund rates. A response procedure needs authority to pause the worker or provider credential, reconcile the trusted ledger against provider results, correct customer state, preserve evidence, and decide whether credentials or sessions must be revoked.

Residual risk remains: a compromised operator with current authority may create distinct, valid refund decisions until behavior or review catches the abuse. That is a different path, requiring limits, approval rules, behavioral detection, and reconciliation rather than replay defense. Naming it prevents one good control from being credited with more protection than it provides.

Run the workshop as a decision session

Send the diagrams, inventory, known incidents, and release question as prework. Invite the system owner who can state purpose and risk authority, the implementer who knows the actual flows, the security reviewer who can challenge assumptions, and the operator who will own telemetry and recovery. Assign someone to record decisions and evidence; facilitation and recording are both real work.

Time-box the session, but do not time-box it into false completion. Confirm scope and the release decision, walk boundaries and sensitive operations, write concrete scenarios, then select the paths whose impact and plausibility demand treatment. Chapter 8 supplies the consistent rating method; here the immediate distinction is between a release blocker, owned follow-up work, and a recorded assumption that requires no action now.

For each high-priority path, settle these questions before moving on:

1. Where can the path be prevented or its impact contained?
2. What code, configuration, architecture, or operating task must change?
3. What test or review will prove the intended behavior?
4. What evidence would reveal attempted or successful abuse?
5. Who can contain and recover from it?
6. What remains possible after the controls, and who accepts that risk?

The workshop may uncover a question that cannot be answered in the room. Record an owner and due date for the investigation, plus the release consequence if the answer is unfavorable. An unresolved assumption should never masquerade as a control.

Traceability matrix

Keep one compact ledger. It earns a tabular form because a reviewer must compare each priority threat across the same release obligations.

Link every row to its implementation tasks, test results, alerts, runbook, owner, and exception record. The ledger is an index into engineering work, not a substitute for it.

Maintenance

Threat models decay as architecture and evidence change. Reopen the relevant paths when a boundary, store, queue, vendor, model, tool, privileged workflow, public endpoint, or data class changes; when a dependency receives broader privilege; when a vulnerability or incident breaks an assumption; or when an exception reaches its expiry. A pull request or architecture decision that changes a named flow should link back to its scenario. Incidents should update the model with observed paths, missing signals, and response friction.

Watch for documents that remain polished while the system moves underneath them. A diagram without changed controls is theater. A security monologue wastes the implementer’s knowledge. A long unranked threat list hides the release decision. A control without a test is an aspiration, and a design without detection or response assumes prevention will be perfect.

Before release, a reviewer should be able to identify the scope and boundaries, follow each high-priority scenario from actor to consequence, see which design and implementation decisions interrupt it, inspect proof of those decisions, and find the signal and response owner for failure. The residual-risk record should say what remains possible and who has authority to accept it.

The replay question is finished only when duplicate delivery has become safe, conflicting behavior becomes visible, recovery has an owner, and the remaining insider path is carried into risk rating. That is the useful boundary of a threat model: not the last imaginable attack, but the point where credible paths have become reviewable engineering and operating decisions.