Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 6

Why AI Fails

Diagnose AI failures across framing, data, model, system, workflow, and operations, then connect each failure to evidence, controls, and ownership.

The Failure That Looked Like a Model Bug

A regional retailer deploys an assistant that drafts answers to delivery questions. The pilot team checks one hundred conversations and approves the release. Six weeks later, complaints rise. The assistant promises refunds outside policy, cites expired delivery windows, and occasionally exposes details from another order.

“The model got worse” is the first diagnosis. It is also too vague to guide a repair.

The team pauses policy citations and cross-order retrieval, then reconstructs what changed. A seasonal promotion altered the mix of requests. The retrieval index kept superseded policy pages. Account permissions were applied after retrieval rather than before it. Support agents accepted drafts quickly during peak volume. The dashboard measured latency and thumbs-up clicks but not policy violations, cross-account retrieval, corrected promises, or complaints.

Replacing the model might alter the symptoms without repairing any of those causes. The useful question is: At which system boundary did an assumption first stop being true, how did the failure travel, and why did no control catch it?

Individual outputs may surprise you. The classes of failure usually should not.

Trace the Harm Backward

Begin with the outcome, not the component suspected by the loudest person in the room. Record who or what was affected, the severity and duration, whether the effect can be reversed, and what must be contained now. A false refund promise, disclosure of another customer’s order, and a slower response are not interchangeable errors; they require different containment and authority.

Next, reconstruct the exact system that produced the outcome: model, prompt, application code, corpus, data snapshot, policy, tool, permission, dependency, interface, and operating conditions. Preserve inputs and intermediate states where doing so is lawful and safe. Without version identity, a team cannot distinguish a model change from a stale index, a changed population, or a broken authorization path.

Then follow the path in both directions. Forward, trace input to context, output, human decision, system action, and real-world effect. Backward, ask which assumption each transition required. Stop at the earliest assumption that became false. That is often where recurrence can be prevented, although later safeguards may still need repair.

The retailer’s chain is concrete:

  1. A customer asks about a delayed order during a promotion.
  2. Retrieval searches documents before applying account scope and returns another order plus an expired policy page.
  3. The model drafts a fluent answer from that context.
  4. The interface shows the draft without policy age or source ownership.
  5. An overloaded agent accepts it.
  6. The customer acts on the promise; the affected account holder is exposed.
  7. The dashboard records a fast response and no immediate negative click.

The visible text is one link. Repairing only that link leaves the unsafe retrieval, review, and monitoring path intact.

Six Places an Assumption Can Break

Use six layers to orient the investigation. They are locations in a causal system, not departments to blame.

  1. Framing: the team automates the wrong task, chooses a proxy that diverges from the real outcome, or leaves the action after the output ambiguous.
  2. Data: evidence is missing, stale, biased, mislabeled, impermissible, unrepresentative, or leaked across an evaluation boundary.
  3. Model: learned behavior does not generalize, uncertainty is poorly expressed, rare but severe errors disappear in an average, or the model cannot meet a deterministic requirement.
  4. System: retrieval, permissions, tools, validation, integration, or dependencies convert uncertain output into an unsafe result.
  5. Workflow: people lack the evidence, time, authority, training, or incentive to review and correct the system.
  6. Operations: drift, harm, security events, cost, complaints, or changes are not detected and assigned to someone who can act.
A six-stage diagnostic map traces AI failure entry points through framing, data, model, system, workflow, and operations, with examples of the broken assumption at each stage.
Trace harm backward instead of stopping at the visible output. The first broken assumption determines which evidence, control, and owner can prevent recurrence.

One incident usually crosses layers. Product incentives can create a bad target. Procurement can prevent access to useful logs. Interface design can make review impractical. Management can reward speed while declaring accuracy essential. “Bad data,” “hallucination,” and “human error” name regions of the map; none is a root cause until the mechanism and missed safeguard are known.

What the Pilot Failed to Prove

The retailer’s hundred-conversation review offered evidence only for the sampled conversations, the versions tested, and the conditions under which reviewers worked. Production changed all three.

The seasonal promotion created data drift: the distribution of incoming requests changed. If the relationship between request features and the desired answer had changed—for example, because a new delivery policy altered which promise was correct—that would be concept drift. Neither label proves harm. A harmless formatting change can trigger a drift alert, while a consequential policy change can leave aggregate statistics nearly stable. Compare input change with outcome quality, important segments, interventions, and system versions.

The pilot also needed protection against leakage. Evidence about generalization is invalid when evaluation contains information unavailable at decision time, when the same customer or document appears on both sides of a split, or when evaluation examples influenced development. Repair the split and rerun the release decision; do not merely adjust the score.

Average quality would still be insufficient. Predictive systems can overfit, miscalibrate confidence, and concentrate error in rare groups. Generated answers can be fluent yet unsupported, incomplete, contradictory, unsafe, or inconsistent with policy. NIST calls confidently presented false or erroneous generative content confabulation. The practical response is to constrain the task and its evidence path: governed sources, claim verification, structural validation, abstention when evidence is missing, and qualified judgment for consequential decisions. Prompting cannot supply absent ground truth.

Tests must match the job. A summarizer needs omission and attribution tests. A classifier needs threshold and segment analysis. A ranker needs position-sensitive measures and downstream outcome checks. An agent needs permission, action, termination, verification, and recovery tests. A single benchmark cannot represent all of these requirements.

The Application Determines the Blast Radius

The cross-account detail did not escape because a model possessed a customer record. The application retrieved a record before enforcing the account boundary and then exposed the result to generation. That is an authorization defect with an AI component in the path.

Prompt injection illustrates the same principle. Untrusted instructions can arrive through user text, retrieved documents, web pages, email, images, or tool output. Retrieval-augmented generation and fine-tuning do not eliminate this risk. The consequence depends on what the surrounding application permits the model to read, influence, or do.

Keep trusted policy separate from untrusted content. Authorize retrieval before content enters context. Expose narrow tools rather than broad shells or generic APIs. Validate arguments and outputs with deterministic rules. Use least-privilege identities, rate and spend limits, idempotency, and auditable actions. Require informed approval for consequential changes, then verify effects through an independent read. These controls must remain effective when the model is confused or manipulated. A refusal is useful defense in depth, not an access-control boundary.

Human Review Is an Operating Condition

The retailer had a human in the loop. It did not have meaningful review.

During the pilot, reviewers had time to inspect examples. During the promotion, the queue grew and acceptance was faster than correction. The interface omitted source age and account scope. Throughput targets rewarded clearing the queue. The review control disappeared even though the box remained on the architecture diagram.

Automation bias is over-reliance on an automated suggestion. Its opposite—rejecting useful assistance indiscriminately—can also worsen outcomes. Calibrated reliance requires visible evidence and uncertainty, easy correction, training on known failure patterns, sampling of accepted and rejected outputs, and measures of correction quality rather than celebration of a low override rate.

Specify review as work: who decides, when, with what evidence, under what queue load, using which authority, and where a disagreement or appeal goes. Set a workload threshold beyond which the system reduces scope, falls back, or pauses. Otherwise the system borrows a person’s name without borrowing their judgment.

Deployment Changes the Evidence

Once a model influences decisions, its outputs can change the data used to judge or retrain it. A fraud model blocks transactions, so investigators see fewer resolved outcomes among blocked cases. A recommender creates exposure for selected items and later reads the resulting engagement as preference. A risk score causes additional scrutiny and therefore more recorded violations in the scored group. A support draft shapes the agent’s final language, which may return as future training material.

A four-step feedback loop connects model output, human or system action, changes in the world, and new training data, with random audits, outcome data, and segment checks shown as ways to interrupt biased observation.
Deployment changes what becomes observable. Preserve independent outcome measurement, audit samples, and segment checks so the system does not mistake its own effects for neutral evidence.

Feedback is not inherently harmful. The danger is treating post-deployment observations as independent truth. Preserve appropriate exploration or random-audit samples, measure outcomes beyond cases selected by the system, distinguish exposure from preference, and record policy changes. Convenient feedback can otherwise intensify the original bias while making the next model look better on its own evidence.

Repair the Chain, Then Narrow the Claim

The retailer does not restart with a new model and the old release claim. It assigns a repair to every failed transition.

The corpus gains owners, policy versions, expiry metadata, and freshness tests. Retrieval enforces account scope before search results enter context, with cross-account isolation tests that block release. The answer must attach support for policy claims or abstain. The interface shows the governing source and its age. Review quality is sampled, reviewer load has a stop threshold, and policy violations, corrections, complaints, and permission denials join latency in monitoring.

The team first replays the incident against the complete repaired path. It then uses peak-season cases, rare severe cases, stale sources, missing evidence, hostile retrieved content, and dependency failures. A monitored canary begins only after the regression evidence passes. One owner has authority to pause the system; security, support, data, and product owners know which decisions belong to them.

The release claim becomes smaller and more defensible: the assistant may draft answers for the authenticated customer’s order from current approved policy, while unsupported refund promises and any cross-account retrieval block delivery. It may not turn a passing average into permission to ignore a severe failure.

For future incidents, keep a short causal record: the outcome and affected parties; exact system versions; propagation path; first broken assumption; missed detection; containment; corrective owner; regression test; and release rule. This is enough to connect diagnosis to action without turning the review into a taxonomy exercise.

Practice: Find the First Broken Assumption

For each case, trace the path from input to outcome. Name the first broken assumption, its layer, one containment action, and one signal that could detect recurrence.

  1. A demand forecast performs well overall but repeatedly under-orders a newly introduced product family. The training set predates the launch, and the dashboard reports only aggregate error.
  2. A résumé assistant follows an instruction hidden in an uploaded document and sends candidate details to an unapproved endpoint through a general-purpose tool.
  3. A clinical note summarizer has stable offline scores, but clinicians begin copying summaries without opening the source record as appointment slots shorten.

Do not stop at “drift,” “prompt injection,” or “automation bias.” For the forecast, decide whether product-family coverage and segment error would reveal the failure. For the résumé assistant, identify the authorization and egress control that should hold even if hostile text changes model behavior. For the summarizer, specify an omission or attribution sample and a workload signal tied to an explicit fallback.

The habit to carry forward is simple: start with consequence, reconstruct the path, find the earliest false assumption, and bind the repair to evidence and an owner. Skepticism becomes useful when it can tell you what to observe and when to stop.

Continue with The AI Appropriateness Question to put these failure costs into the decision before a build begins.

Source Notes