Appendix I: AI Threat Model Worksheet

Follow the Confidential Text

A team calls its retrieval assistant “internal” because employees are its only users. Yet the assistant indexes supplier manuals, can read confidential operating notes, and can draft tickets in a project visible to an external maintenance contractor. One supplier manual contains hidden instructions that make the assistant copy an operating note into a ticket. Authentication succeeds. Retrieval returns an allowed document. The ticketing call uses an approved tool. Every component behaves as configured, and the system still discloses the note.

The word internal described the login page, not the action path. An AI threat model follows assets and authority across the complete AI-enabled system: who can influence an input, what the model can observe, which identity acts, where output travels, and who can see the consequence. Its useful result is not a long threat list. It is a small set of credible abuse paths tied to controls, tests, owners, and explicit residual-risk decisions.

Work From a Concrete Diagram

Freeze a named architecture version. Draw data flows, identities, trust boundaries, external dependencies, retrieval sources, model calls, tools, human approvals, logs, and downstream consumers. Include people who know the product promise, the workflow, the platform, the data, privacy obligations, security controls, and incident route; no single owner is likely to see the whole path.

Now choose one consequential verb—read, rank, send, pay, publish, delete—and trace it from influence to effect. Mark each place where content becomes instruction, one identity becomes another, or a suggestion becomes an action. A model may generate a tool request, but an independently enforced policy must decide whether that identity may perform that action on that resource under those conditions. Retrieved content, uploaded files, model output, and tool responses remain untrusted unless a stronger boundary is demonstrated.

AI Threat Model Worksheet

REVIEW IDENTITY
System and version:
Architecture / data-flow diagram version:
Review date and facilitator:
Product, security, engineering, privacy, and domain participants:
Deployment context, users, regions, and exposure:
Decision requested: design / pilot / launch / change / incident review
Consequential verbs and prohibited actions:

SYSTEM BOUNDARY AND TRUST BOUNDARIES
Components inside the review boundary:
Models and model providers:
Data stores, retrieval indexes, and pipelines:
Prompts, policies, orchestration, memory, and caches:
Tools, APIs, code execution, and physical actions:
User, admin, service, model, and agent identities:
External systems and downstream consumers:
Trust boundaries and privilege transitions:
Last reversible step before each consequential action:

ASSETS AND SECURITY OBJECTIVES
Sensitive or regulated data:
Credentials, tokens, keys, and permissions:
Model, prompt, policy, and evaluation intellectual property:
Embeddings, indexes, logs, feedback, and incident evidence:
Integrity of decisions, rankings, content, and actions:
Availability, budget, and capacity:
User safety, rights, and organizational trust:
For each asset: confidentiality / integrity / availability / safety objective:

ACTORS AND ASSUMPTIONS
Legitimate users and administrators:
External attackers and abusive users:
Insiders and compromised accounts:
Suppliers, providers, and subprocessors:
Downstream systems that may over-trust output:
Assumptions that must be tested:

ENTRY POINTS AND CHANGE CHANNELS
[ ] User prompt or conversation     [ ] Uploaded file or image
[ ] Retrieved document              [ ] API or webhook
[ ] Tool result or external website [ ] Feedback or annotation
[ ] Training or tuning data         [ ] Model or dependency update
[ ] Prompt or policy update         [ ] Admin interface
[ ] Logs, exports, or analytics     [ ] Supply-chain artifact
Other:

ABUSE PATH RECORD (repeat per path)
Path ID and title:
Actor and objective:
Preconditions and required access:
Entry point:
Step-by-step path across trust boundaries:
Identity and authority used at each step:
Asset or person affected:
Maximum credible consequence and blast radius:
Existing preventive controls:
Existing detective controls:
Containment and recovery controls:
Assumption most likely to fail:
Control gaps and owner:
Test case and expected safe behavior:
Evidence location and date:
Likelihood / impact / detectability:
Residual risk and decision:

AI-SPECIFIC THREAT SWEEP
[ ] Direct or indirect prompt injection
[ ] Sensitive-information disclosure or cross-tenant retrieval
[ ] Data, feedback, retrieval, or model poisoning
[ ] Excessive agency or confused-deputy tool use
[ ] Insecure output passed to code, query, browser, or workflow
[ ] Model extraction, inversion, or membership inference
[ ] Cost exhaustion, denial of service, or runaway loop
[ ] Evasion, jailbreak, policy bypass, or adversarial input
[ ] Unsafe memory or context persistence
[ ] Hallucinated authority, identity, citation, or tool result

CONVENTIONAL AND SUPPLY-CHAIN SWEEP
[ ] Authentication and session failure [ ] Broken authorization
[ ] Injection and unsafe parsing        [ ] Secrets exposure
[ ] Dependency or artifact compromise  [ ] Misconfiguration
[ ] Network or storage exposure         [ ] Logging leakage
[ ] Vendor change without notice        [ ] Weak backup or recovery

CONTROL DESIGN
Identity and least-privilege enforcement:
Instruction/data separation and input handling:
Retrieval authorization and content provenance:
Tool allowlist, parameter constraints, and approval gates:
Output validation before downstream use:
Rate, time, cost, and action limits:
Security telemetry and privacy-safe logging:
Kill switch, credential revocation, isolation, and rollback:
Incident owner and tested response route:

VERIFICATION AND DECISION
Security test suite and evidence:
Adversarial / red-team cases:
Control-owner sign-off:
Unresolved high-severity paths:
Decision: block / redesign / limited pilot / approve with conditions
Residual-risk owner and expiry:
Retest triggers and next review:

Choose Paths by Consequence, Not by Vocabulary

The threat sweeps are prompts for discovery, not findings. Turn a checked label into a path an attacker or abusive insider could actually complete. Rank those paths by consequence, exploitability, exposure, and detectability, but do not let a numeric product average away a catastrophic outcome. Unknown reach, unauditable supplier behavior, and untested isolation are uncertainties to resolve or bound, not low scores.

For a consequential path, ask an independent reviewer to challenge the claimed attacker capability, blast radius, and time to containment. Convert each control on which the decision depends into evidence: an authorization test, adversarial case, configuration assertion, recovery exercise, or monitored canary. A control that exists only in the diagram is a design intention.

Work the Path: The Supplier Manual

Return to the assistant. The protected asset is the confidential operating note; the security objective is that only the internal operations group can read it. The attacker controls a supplier manual accepted into the retrieval index. The path crosses supplier content, ingestion, retrieval, prompt assembly, generation, the assistant’s service identity, the ticket API, and the contractor-visible project. The maximum credible consequence is not “a bad answer.” It is disclosure of every note the service identity can retrieve, repeated across as many tickets as the tool permits.

The first proposed control is a system instruction telling the model never to reveal confidential material. A seeded supplier document defeats it. This failure changes the design rather than merely adding another prompt. Retrieval authorization is evaluated for the requesting employee; the assistant runtime receives no ambient access to unrelated notes; a policy service limits ticket destinations and fields; and the review screen shows the source text, destination, and exact diff before submission. Rate limits and an alert on blocked cross-boundary attempts reduce and reveal repeated attacks.

The team reruns the seeded document against the assembled system, not only the model. The expected result is a blocked draft, a security event tied to the source and service identity, and no ticket side effect. It then revokes the ticket credential and rehearses isolation of the affected index. The evidence supports a limited pilot in a sandbox project. Production ticket creation remains disabled until destination authorization and the recovery exercise pass.

Make the Decision Survive a Broken Control

Before approval, disable or defeat one control on a material path. Let a malicious document enter the index, remove the human approver, use an over-privileged service identity, silence an alert, or make credential revocation fail. Observe how far the action travels and whether the system stops in a bounded state. This is where a neat diagram meets containment time, backup ownership, and recovery reality.

A completed worksheet should allow a reviewer to answer three questions without relying on the meeting’s memory: Which paths can produce the maximum credible harm? Which tested controls interrupt each path before the last reversible step? Who owns the remaining risk, until when, and what change forces a new review? If any answer is missing, the honest decision is to block, redesign, or narrow the deployment—not to fill the blank with confidence.

Pair this worksheet with Security Architecture for AI Applications, Red Teaming, Adversarial Testing, and Abuse Cases, and the Launch Readiness Checklist.