Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 35

Incident Response for AI Systems

Contain AI failures, preserve behavior evidence, coordinate accountable decisions, recover safely, and convert incidents into regression protection.

The Leak That Disappeared During Debugging

An employee reports that a knowledge assistant quoted part of a confidential acquisition memo. The on-call engineer retries the prompt and receives a safe answer. Assuming a transient model anomaly, the team updates the system prompt and clears the retrieval cache. By the time security joins, the relevant index snapshot is gone, the provider model alias has changed, and no one knows which permissions or retrieved chunks were used.

The quick patch may have reduced exposure, but it also destroyed the path needed to determine scope, affected users, root cause, notification duties, and whether the incident is truly contained.

Respond to AI incidents with the familiar priorities of protecting people and operations, while preserving the AI-specific evidence that explains probabilistic behavior. Contain first, preserve proportionately, diagnose across the system, and recover through an evaluated release.

A six-stage AI incident flow moves from detect to contain, preserve, diagnose, recover, and learn; the preserve stage captures input, output, context, tool call, version, and policy evidence, while a parallel lane communicates with users, owners, and reviewers and containment can return to a known-good state.
Containment precedes explanation, but evidence preservation must travel with it. Recovery is a controlled release, and communication runs throughout the incident rather than waiting for the postmortem.

Decide What Counts Before the Page Fires

An AI incident is an event in which an AI-enabled system causes, contributes to, or creates a credible risk of unacceptable harm, unauthorized action, control failure, material service failure, or breach of policy, security, privacy, contract, or another applicable obligation.

That definition reaches beyond a broken model. Confidential text can escape through retrieval, output, memory, logs, or a tool. A prediction can concentrate harm in one population while aggregate quality remains steady. An agent can attempt an action that its approval control should have blocked. Poisoned data, runaway cost, a failed appeal path, or missing telemetry can each put the system outside its accepted boundary.

Not every wrong output requires incident command. Declare and escalate according to consequence, exposure, affected population, data sensitivity, autonomy, duration, reversibility, failed controls, and time-sensitive notification triggers. Repeated low-severity reports may together reveal a systemic incident. These thresholds, and the authority to act on them, belong in the playbook before an alert arrives.

Use the organization’s existing incident-command system. Add the AI, data, domain, human-factors, privacy, safety, vendor, or legal expertise the event needs. A separate “AI process” that bypasses experienced security and reliability responders creates another seam during an emergency.

The First Fifteen Minutes: Contain and Preserve

At 09:12, the knowledge-assistant report reaches on-call. The team knows only that one answer contained a confidential sentence. A retry is safe. The most dangerous interpretation of that result is that the incident has disappeared.

The incident commander separates two urgent questions: What must stop now? What evidence will disappear when it stops? Operations disables the affected retrieval index and routes users to search and human support. The investigation lead records the active release bundle and takes a protected reference to the index snapshot before the cache is cleared. The system owner identifies the tenants and workflows that share the retrieval path. A scribe records facts, hypotheses, actions, owners, and timestamps separately.

Contain the outcome at the narrowest boundary that reliably limits harm. Revoke an unsafe tool or credential; pause an affected route, source, tenant, or cohort; force abstention or human review; isolate suspect data or memory; roll back to a known-good release bundle; or disable the system when narrower controls cannot hold. Then verify with scoped tests and telemetry that new exposure has stopped.

A prompt patch is not containment when permissions still admit restricted documents. A model rollback is not containment when the suspect index remains live. Nor does disabling generation repair recommendations already copied into downstream work.

Preservation travels with containment, but it is not a license to duplicate sensitive data. Capture the behavior path proportionately: the input and displayed output; model, prompt, policy, code, and interface versions; retrieved document identifiers, ranks, permissions, corpus, and index; tool calls and approvals; relevant data and feature references; human edits and decisions; and the logs, releases, evaluations, and risk records needed to reconstruct what happened. Record integrity, access, retention, and deletion controls.

Where exact content cannot be retained, use secure references, structured metadata, hashes, redacted extracts, or reproducible seeded cases. State the resulting evidence gap. An investigation whose material is copied into an open incident channel may compound the original breach.

Preauthorize emergency disablement. On-call should not have to find an executive before stopping an actively harmful action. Restoration is different: only named authority may accept material residual risk and reopen a high-impact capability.

The Next Hour: Find the Reach of the Failure

At 09:31, the preserved trace shows a public document and a restricted chunk in the same retrieval context. The index was rebuilt six hours earlier. That evidence makes a model-only explanation less likely, but it does not yet prove how the chunk crossed the permission boundary.

Build a timeline and an affected-version map before chasing a favorite root cause. Start at intended use and policy: was this request in scope, and what should have happened? Follow the data and retrieval path through source permissions, parsing, chunking, indexing, ranking, and memory. Check the model and prompt for version changes or injectable instructions, then the tool, application, cache, queue, retry, and interface paths. Finally inspect the human workflow: what reviewers saw, what they could override, which alerts fired, and which ownership or change controls failed.

Reproduce in an isolated environment with the incident bundle, not today’s mutable system. Test rival explanations. In this case, the parser copied a parent document’s public classification onto child chunks during the rebuild; the index accepted that metadata; retrieval returned the restricted text; and the answer interface displayed it without a sensitivity check. No single component explains the incident. The aligned failures do.

Scope follows the shared path. Which indexes used that parser release? Which tenants shared its pipeline? When did the faulty classification begin? Which retrieved chunks reached outputs, caches, logs, human decisions, or other systems? Report counts and affected groups with confidence and known blind spots. Do not turn “we found no successful tool call” into “no data was exposed”; retrieval context or output may already have disclosed it.

Communication Is Part of Containment

At 09:45, responders can confirm a retrieval-control failure and that the affected index is offline. They cannot yet claim a complete user count. That is enough for a useful update:

We confirmed that the assistant returned text from a restricted document. The affected retrieval index has been removed from service, and users have been routed to a non-generative fallback. We are determining the time window and affected accounts. Preserve related reports and do not copy suspect output into this channel. The next update is at 10:15.

Responders need assignments, safety rules, and current containment state. Leaders need impact, uncertainty, continuity choices, and resource decisions. Affected people need plain consequences, protective action, correction or remedy, and a contact route. Vendors need precise evidence and a requested action without unnecessary disclosure. Qualified privacy, legal, contractual, insurance, or regulatory reviewers need jurisdiction- and role-specific facts to assess preservation and notification duties.

Say what the system did and what the organization knows. “The AI went rogue” hides the failed controls; “no exposure” claims more than the evidence. Notification duties vary by jurisdiction, sector, contract, data, role, and harm, so route facts through the preplanned reviewers rather than improvising legal conclusions.

Recovery Is a New Release

The prior release bundle is available, but rolling it back would remove an accessibility fix. The choice is not between speed and bureaucracy. It is between two known harms under uncertainty. The team keeps generative retrieval disabled, restores the accessible interface over a clean index, and tests the composed candidate rather than reviving either old bundle wholesale.

The original leak must no longer reproduce, but one passing prompt proves little. Test the failure family: source and child-chunk authorization, index construction, cache handling, indirect injection, output sensitivity, and disclosure monitoring. Then test legitimate use, accessibility, affected segments, safety, privacy, latency, cost, fallback capacity, and reviewer workload. Correct or quarantine affected messages, memory, indexes, decisions, and downstream records where possible.

Restore exposure progressively from a known-good state. The approving owner records residual risk; heightened monitoring has an owner and an exit condition. If the team cannot show that the restored scope is acceptable, the capability stays reduced or retired.

That gate hands the work to change management. The incident supplies regression cases, affected components, stop conditions, and a reason for the change; it does not grant an exception from evaluation.

Close the Learning, Not Only the Page

A review reconstructs intended behavior, the event and response timeline, affected people, evidence gaps, failed controls, and the technical, workflow, organizational, or vendor conditions that aligned. It tests whether containment and communication worked, then assigns corrective actions with owners, deadlines, verification, and risk priority. Similar systems using the parser and index pipeline enter the review scope.

Blameless learning protects good-faith reporting and honest inquiry. It does not erase accountability for ignored evidence, unauthorized risk acceptance, or remediation left undone. Track actions to verified closure and update the evaluation set, monitoring, runbook, threat model, launch gate, training, and vendor controls that should prevent or reveal recurrence.

Rehearse this chapter before a real leak. In a 60- to 90-minute tabletop, reveal the report, restricted chunk, recent index rebuild, sensitive cache logs, shared tenant pipeline, and rollback-accessibility conflict one at a time. Require participants to declare severity, assign command, contain, preserve, scope, communicate, choose a safe fallback, and define recovery evidence. The useful output is not a score. It is the set of authorities, access paths, tools, and policies that failed under rehearsal and now have owners.

AI Incident Response Playbook

Prepare

  • Define incident categories, severity, declaration authority, command roles, and contacts.
  • Inventory systems, owners, release bundles, vendors, data classes, affected groups, and notification reviewers.
  • Test feature disablement, tool revocation, safe fallback, known-good rollback, evidence capture, and communication routes.

Detect and contain

  • Confirm the report without delaying protective action.
  • Limit unsafe actions, exposure, data flow, and downstream effects.
  • Protect reporters and affected people from retaliation or repeated harm.
  • Verify containment with scoped tests and telemetry.

Preserve and diagnose

  • Freeze or reference model, prompt, policy, corpus, index, data, code, tool, and interface versions.
  • Preserve inputs, outputs, context, calls, approvals, logs, decisions, and deployment events proportionately.
  • Maintain integrity, access, privacy, retention, and a time-stamped decision log.
  • Test hypotheses across data, model, retrieval, tools, software, workflow, and governance.

Communicate and recover

  • State confirmed facts, uncertainty, impact, protective action, and next update.
  • Route notification questions to qualified jurisdictional, contractual, privacy, and legal reviewers.
  • Correct affected decisions, data, messages, memory, or permissions where possible.
  • Pass regression, segment, safety, operational, and staged-release gates before restoration.

Learn

  • Complete the review, assign actions, verify closure, and inspect similar systems.
  • Update evaluations, monitoring, runbooks, threat models, documentation, training, and governance.

Closure Failure: False High-Risk Predictions

Write a post-incident review for this contrasting failure. Several false high-risk predictions have produced complaints. The weak team raises the threshold and closes the incident when the page becomes quiet. It never identifies which release, segments, decisions, or reviewers were affected, and the new threshold increases missed cases.

The stronger team pauses automated routing, restores human review, preserves scores, features, labels, thresholds, explanations, and reviewer actions, and maps affected decisions. It finds that an upstream category changed meaning without a data contract update. Recovery repairs the schema, reprocesses affected cases, evaluates both error directions by segment, notifies the appropriate owners and affected-party support route, and adds schema and outcome regressions.

An incident ends when risk is controlled and learning is verified—not when the page becomes quiet.

Source Notes