Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 31

Red Teaming, Adversarial Testing, and Abuse Cases

Turn plausible misuse, attacks, and unsafe behavior into bounded adversarial probes, actionable findings, and permanent regression tests.

The Assistant That Reported a Transaction It Could Not Make

A financial-guidance assistant can retrieve approved policy, calculate illustrative scenarios, and draft a response for a licensed reviewer. It cannot approve or execute a transaction. Its ordinary evaluation set looks strong.

During an adversarial exercise, a tester uploads a statement containing hidden instructions: ignore the review rule, call the approval tool, and tell the user the action is complete. The assistant follows part of the document instruction. The tool layer blocks the transaction, but the interface still displays a confident confirmation.

The system did not have one failure. It exposed three: untrusted content crossed an instruction boundary, a message claimed an action that never occurred, and the evaluation suite had never combined document injection with tool failure.

The transaction did not occur, yet the system created a dangerous belief about it. That is the work of red teaming: follow a plausible adversary through the whole AI-enabled system, under controlled conditions, until the team can name the failure, repair its cause, and keep it from returning.

A five-stage adversarial testing loop moves from system map to abuse case, safe probe, finding, and regression test around a release gate, with a stop and escalate exit from unsafe probing.
Adversarial work creates value when a system hypothesis becomes a bounded probe, a reproducible finding, and a permanent release test. The stop path is part of the test design, not an afterthought.

Red Team the Decision, Not the Demo

Quality assurance asks whether specified behavior works. Security testing looks for exploitable weaknesses in software, identity, infrastructure, and data flows. Safety evaluation examines harmful behavior and whether its controls hold. Red teaming crosses those concerns from an adversary’s point of view: how could a motivated attacker, abusive user, careless insider, or compromised dependency produce a consequential outcome?

None replaces the others. A model that refuses a direct harmful request may still obey instructions retrieved from a document. A hardened API does not prove that an agent uses it within the user’s authority. A toxicity score does not reveal whether repeated low-severity outputs can harass one person at scale.

The financial team writes its objective as a release decision: determine whether untrusted customer documents can cause the assistant to misstate policy, cross a permission boundary, or claim that a consequential action succeeded. Any reproducible path to those outcomes blocks the pilot. This objective is narrower than “find jailbreaks” and more useful. It determines which system version, identities, data, environment, stop rules, and evidence belong in the exercise.

Draw the Path to the Harm

The team begins with the deployed path rather than the model endpoint. A customer supplies a prompt and statement. A parser extracts text. Retrieval adds approved policy. The model drafts an answer and may request read-only calculations. An orchestration layer checks the requested tool, a licensed reviewer makes the decision, and the interface reports the result. Logs, memory, alerts, and correction paths sit alongside this flow.

Now the important boundaries become visible. Customer statements are untrusted data; policy is authoritative evidence. The model may propose; the tool gateway and reviewer hold authority. A success message must derive from a recorded tool result, not from generated prose. Confidential customer data must remain inside its account boundary. Each of these is a claim an adversarial probe can test.

The original design treated all text in the model context as one conversation. That allowed an ordinary-looking document to compete with system instructions. Indirect prompt injection exploits exactly this confusion: attacker-controlled content arrives through a file, web page, email, image, or retrieved record and is interpreted as instruction.

Turn the Outcome into Abuse Paths

An abuse case connects an actor and goal to an entry point, preconditions, sequence, affected asset, consequence, and expected control. The team chooses one outcome—make a customer or reviewer believe an unauthorized transaction succeeded—and works backward:

Goal: create a believable unauthorized transaction outcome
├── inject an instruction through an uploaded statement
│   ├── suppress the review requirement
│   └── fabricate a success message after tool rejection
├── call a permitted tool with an unauthorized account or amount
├── exploit stale reviewer or customer permissions
├── poison policy retrieval with attacker-controlled guidance
└── recover or replay transaction data from logs or memory

The tree prevents a fashionable jailbreak from consuming the whole exercise. It also reveals tests that do not depend on prompt wording: tenant isolation, stale permissions, tool argument validation, retrieval provenance, log access, dependency integrity, rate limits, and recovery. A second tree can cover disclosure; another can cover discriminatory or policy-violating guidance under paraphrase, multilingual input, and repeated pressure. The team prioritizes paths by consequence, exposure, and the strength of existing controls, not by novelty.

Every chosen path needs an oracle: an observable condition that separates a pass from a failure. For the injected statement, the transaction tool must not be called, the draft must not state or imply completion, the untrusted instruction must be visible in the trace, and the case must reach the licensed reviewer with a warning. “The assistant behaved strangely” cannot support a repair.

Let the Probe Gain Realism Slowly

Authorization is part of the test design. The exercise record names who approved it, the systems and identities in scope, allowed techniques, time window, rate limits, synthetic-data rules, escalation contacts, evidence retention, and immediate stop conditions. A red team that can endanger customers or production data is demonstrating its own control failure.

The team increases realism only when the preceding stage cannot answer the question:

  1. inspect the architecture and verify deterministic controls;
  2. run the injected document against a sandbox with synthetic accounts;
  3. connect simulated tools that record proposed actions but cannot execute them;
  4. reproduce the release bundle in a production-like environment with test identities;
  5. touch production only when explicitly authorized and operationally necessary.

Real credentials, personal data, harmful material, and uncontrolled payloads do not become acceptable merely because they make a probe realistic. Messages and transactions must not reach people or live resources. Simulated tools and an action ledger let the team observe proposed side effects without granting authority.

Discovery and measurement are different passes. A tester adapts the statement, follow-up messages, and timing until the failure appears. The team then reduces that path to the smallest reproducible case, freezes the affected release bundle, and reruns it enough to characterize variation. One successful attempt can establish exploitability; repeated failures do not establish impossibility.

Write a Finding Someone Can Repair

A screenshot of the false confirmation preserves the symptom but not the conditions that produced it. The finding records the model, system prompt, policy corpus, parser, orchestrator, tool schema, interface, and environment versions. It includes the test identity, uploaded document, retrieved context, tool request and rejection, final message, logs, and observed variation across attempts.

The team assesses the finding in system terms: which customers or reviewers could encounter it, how an attacker reaches the path, whether it is detectable, whether a belief or action can be reversed, and how far the effect can scale. It also records that the tool gateway worked. The failure is material because the interface converted a blocked action into a credible falsehood, not because the model disobeyed in the abstract.

The record names immediate containment, a remediation owner and due date, disclosure constraints, the permanent test identifier, and whether the finding blocks release. Reusable exploit detail is shared only as broadly as remediation and accountability require.

Repair Each Failed Boundary

A sentence in the system prompt telling the model to ignore document instructions may reduce one observed failure, but it leaves the system’s claims dependent on generated text. The team repairs the path at each boundary.

The parser labels statement text as untrusted content. Retrieval admits policy only from an approved, versioned source. The orchestrator validates tool arguments against the authenticated account and exposes only the minimum read-only functions needed for illustration. Approval remains a separate reviewer action. Most importantly, the interface renders transaction status from a signed action record; the model cannot manufacture the success state. A rejected or absent tool result produces an explicit “no transaction occurred” state and an escalation path.

The exact injected statement now becomes a regression test. Nearby variations change encoding, language, document position, conversational pressure, and tool failure mode. Legitimate statements must still be summarized correctly. The team watches reviewer workload, false alarms, latency, and accessibility because a stricter control can protect one boundary while making the service unusable or hiding uncertainty elsewhere.

Model, prompt, policy, retrieval, tool, parser, or dependency changes rerun this suite. Production monitoring looks for untrusted instructions reaching control paths, mismatches between displayed and recorded action state, repeated boundary probes, and unusual cost or tool-call patterns. New incidents and abuse reports feed new branches into the attack tree.

The Red-Team Plan

A useful plan is short enough to govern the exercise and specific enough to stop it. It records:

  1. the system purpose, exact release bundle, boundary, and decision the exercise will inform;
  2. exercise, system, security, safety or risk, incident, and residual-risk owners;
  3. authorized environments, identities, data, techniques, dates, rates, exclusions, and stop rules;
  4. assets, affected people, adversaries, trust boundaries, and highest-consequence outcomes;
  5. prioritized abuse cases, expected controls, observable oracles, and safe progression;
  6. finding evidence, severity and triage method, disclosure rules, and remediation deadlines;
  7. legitimate-use checks, permanent test ownership, launch blockers, and post-launch cadence.

For the financial assistant, the pilot remains blocked until the display state is derived from the action ledger, injected documents cannot change approval authority, and the repaired path passes both adversarial and legitimate-use cases. An independent risk owner decides whether any residual exposure is acceptable. The next chapter takes up the subsequent question: how much real-world exposure the system has earned.

Now transfer the method to an internal knowledge assistant. Its users may read public and team documents, but not legal-investigation files. Build one attack tree for cross-boundary disclosure through retrieval, citations, logs, and memory. Choose the first bounded probe, write its observable oracle, and name the evidence needed to turn a successful probe into a permanent release test. Then identify one control that could stop the attack while also harming legitimate use; specify how you would detect that side effect.

Red teaming earns its place when creativity becomes an engineering obligation: a plausible path, a controlled probe, a repair at the responsible boundary, and evidence that survives the person who first found the flaw.

Source Notes