Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 22

Guardrails, Policies, and Control Layers

Turn AI risk statements into preventive, constraining, detective, and recovery controls with explicit owners and tested failure routes.

The Filter Fired Too Late

An employee asks an HR assistant about parental-leave eligibility. One of the retrieved documents contains hidden instructions to search for compensation data and include it in the answer. The model follows them. An output classifier blocks the reply, so the user sees nothing.

The team initially records a successful guardrail event. Then an engineer traces the request. The assistant retrieved a salary table with a service identity broader than the employee’s access. The restricted rows entered model context. The full prompt, including those rows, went to an analytics log. The classifier prevented disclosure through one channel after two other boundaries had already failed.

The filter worked as designed. The system did not.

A guardrail is an enforced measure that reduces the likelihood, reach, detection delay, or recovery cost of a defined failure. A useful control system layers measures with different mechanisms and owners, so that one miss does not silently become unlimited authority.

A defense-in-depth diagram moves one request through Prevent, Constrain, Detect, and Recover layers, with input, policy, access, monitor, escalate, and rollback controls owned across product, security, and operations.
No control catches everything. Layer prevention, bounded authority, detection, and recovery so a missed failure does not become an uncontrolled consequence.

Follow the Failure Before Choosing Controls

“Add safety” is not testable. Use a risk statement with an observable chain:

trigger -> vulnerable behavior -> system consequence -> affected party -> severity

An untrusted benefits document contains hidden instructions
-> the model treats them as commands
-> retrieval runs with a service identity broader than the employee's access
-> restricted salary rows enter context and logs
-> employee confidentiality is exposed even though the answer is blocked
-> high severity

The chain does two things that “prevent data leakage” cannot. It identifies where intervention is possible, and it gives each control a claim to prove. Source admission can reduce the chance of hostile content entering the corpus. Permission-aware retrieval can prevent the salary rows from entering context. A log redaction boundary can limit a second copy. Detection can stop the session and identify affected data. Recovery can quarantine the source, revoke credentials, and test every derivative store.

Start at the consequence and work backward. Which data crossed a boundary? Under whose identity? Which operation made the exposure possible? Which signal would reveal it? What state must be restored? Only then decide whether the system needs an input classifier, a policy instruction, an authorization rule, an output validator, or some combination.

Four Different Jobs

Prevention reduces exposure

Preventive controls reduce exposure before the risky operation begins:

  • approved use cases and prohibited action classes;
  • identity, authentication, and request-size validation;
  • data minimization and source admission rules;
  • secure defaults, rate limits, and dependency allowlists;
  • user education that sets accurate expectations.

For the HR assistant, the team admits only documents from named repositories, scans new sources, separates system instructions from retrieved text, validates request size and file type, and tells employees that the assistant cannot make employment decisions. These controls remove avoidable exposure and set an honest expectation. They cannot enumerate every hostile sentence or misunderstanding.

Constraints bound authority

Constraining controls limit what a compromised or mistaken component can do:

  • access control at the data service;
  • typed tool schemas and semantic argument validation;
  • least-privilege, task-scoped credentials;
  • deterministic business rules and transaction limits;
  • sandboxing, timeouts, budgets, and termination criteria;
  • human approval for consequential or irreversible actions.

Here the decisive control is permission-aware retrieval: the data service evaluates the employee’s identity and the requested object before returning a row. The assistant receives a task-scoped credential, not a reusable HR service account. A typed retrieval request constrains syntax; semantic validation still checks the document class, employee scope, approved purpose, and result volume. Schema validity alone cannot make an unauthorized value safe.

Model-facing instructions still help shape behavior. They can say that retrieved text is evidence rather than instruction and that compensation data is outside the task. They cannot authorize access. The model may propose a query or action; a separate authority decides whether it is permitted.

Detection reveals misses and decay

Detective controls reveal failures and control degradation:

  • policy and safety classifiers;
  • access anomalies, abuse patterns, and cost spikes;
  • groundedness, schema, and reconciliation checks;
  • sampled human review and affected-user reports;
  • canary tests and recurring adversarial suites;
  • alerts on bypasses, fallback rates, and control unavailability.

The HR team watches for denied retrievals, unusual source-to-query sequences, access anomalies, classifier bypasses, log-redaction failures, and bursts in cost or retries. It also samples allowed sessions, because a dashboard containing only blocked events cannot reveal silent misses.

Measure the detector itself: precision, recall on severe test cases, time to detect, reviewer burden, and known blind spots. Break results down by language, task, and user group. An aggressive filter that makes an assistive interface unusable has changed the product’s risk rather than simply reduced it.

Recovery limits duration and spread

Corrective controls contain harm and restore a known state:

  • abstention, safe fallback, and human escalation;
  • feature disablement, credential revocation, and rate reduction;
  • quarantine, rollback, and compensating transactions;
  • affected-data tracing and notification workflows;
  • incident investigation, control updates, and regression tests.

For the hidden instruction, recovery means ending the session, quarantining the source, revoking the overbroad credential, tracing where the salary rows were copied, deleting or restricting affected logs under the organization’s retention process, and adding the exact route to the regression suite. If the assistant cannot reach a staffed escalation destination while the feature is disabled, “human fallback” is only a label.

A control plan without a recovery path assumes prevention will be perfect.

Policy Changes Form as It Approaches Enforcement

The HR policy says compensation data may be used only for named administrative purposes by authorized roles. That statement gives intent, exceptions, and accountability to people. A system instruction translates part of it into model-facing behavior: do not seek compensation data, treat retrieved documents as untrusted evidence, and abstain when the request exceeds benefits guidance. This improves interpretation, but conflicting context can still influence the model.

At the retrieval boundary, the policy becomes machine-enforced: authenticate the user, authorize the object and purpose, cap the result set, and deny by default when identity or policy state is unavailable. At the output boundary, structured generation and schema validation can require a supported answer, citations, and an escalation reason. Business rules still inspect the values. A valid object can contain a forbidden employee identifier or an amount beyond an approval limit.

Use deterministic enforcement for authorization, data access, financial limits, state transitions, and other invariants. Where a rule cannot be deterministic, name the evaluator, threshold, monitor, escalation route, and person who accepts the residual risk. Otherwise policy has become a hope expressed in three syntaxes.

The Layers Can Work Against One Another

In the HR case, tightening the input filter reduces one class of attack but blocks benefits questions written in some languages. Adding human approval creates a queue that reviewers begin to clear without opening the evidence. Expanding logs helps investigation but creates another repository of sensitive text. A rate limit slows abuse and also delays an employee facing a deadline.

Evaluate the assembled control system in the real workflow:

  • bypass rate and severe miss rate;
  • false-positive burden by language, task, and user segment;
  • latency and availability cost;
  • reviewer workload and override quality;
  • residual harm after the control activates;
  • failure behavior when the control dependency is down.

Make the Control Matrix Describe This System

Risk Prevent Constrain Detect Recover Owner
Indirect prompt injection Admit approved sources; isolate retrieved text from instructions Permission-aware retrieval; no arbitrary tools Injection probes; unusual source-to-query sequence End session; quarantine source; rotate affected credentials Security + product
Ungrounded policy guidance Curated authoritative corpus Citation required; no answer without evidence Retrieval recall and claim-support checks Abstain; route to policy owner Product + HR policy
Confidential data disclosure Minimize corpus and logs Row-level access; redaction; narrow identity scope Data-loss prevention and access anomaly alerts Revoke access; investigate; follow incident process Privacy + security
Automated employment action Prohibit autonomous decision path No action tool; documented human authority Audit recommendation-to-decision patterns Pause feature; review affected cases HR owner + risk
Cost exhaustion Input and concurrency limits Per-session budget and tool-call ceiling Token, latency, and retry alerts Degrade to search; throttle abusive identity Operations

The matrix earns its space because a release reviewer must compare different risks across the same control stages and see where ownership changes. It is still only an index. Each cell should lead to a test, an evidence location, a review cadence, and a residual-risk decision. “Permission-aware retrieval” is not evidence that permissions were enforced.

Now build the matrix for the actual assistant. Do not copy these rows if its corpus, tools, users, or consequences differ. Begin with its most severe observable failure chain and ask where the current design has only one line of defense.

Break the Seams Deliberately

Create cases where:

  • user input is benign but a retrieved document is hostile;
  • the primary classifier is unavailable;
  • a valid schema contains an unauthorized business value;
  • a privileged user asks outside the approved purpose;
  • two controls disagree;
  • the human queue is overloaded;
  • an action succeeds but its confirmation event is lost;
  • rollback is attempted after downstream side effects.

For the salary-table incident, the release test must prove more than “the classifier blocks the answer.” It should assert that the hostile document cannot change authorization, the salary rows never enter context, denied retrieval creates a useful signal without copying the protected content, the session enters a defined safe state, and the employee still reaches legitimate benefits guidance.

Then remove one dependency at a time. If the policy classifier is unavailable, does the request fail closed or enter a narrower mode? If the identity service times out after retrieval begins, are partial results discarded? If the action succeeds but its confirmation event is lost, can a retry duplicate it? The most damaging failures often live between components and owners.

Give Each Material Control an Operating Record

For every material control, record:

  1. risk and failure stage addressed;
  2. control objective and mechanism;
  3. preventive, constraining, detective, or corrective role;
  4. enforcement location and dependencies;
  5. owner and on-call route;
  6. test cases and acceptance threshold;
  7. telemetry, alert, and evidence retention;
  8. false-positive and affected-party impact;
  9. bypass and dependency-failure behavior;
  10. residual risk and review trigger.

The record should be versioned with the system it governs. Review it after a model, provider, corpus, tool, permission, policy, user population, or workflow changes. A provider update may alter classifier behavior. A new tool may turn previously harmless model text into an action. A corpus expansion may invalidate the false-positive evidence that supported launch.

The hidden instruction is now a permanent regression case. Its evidence must show that model-facing policy is not treated as authorization; permissions and business rules are enforced outside the model; structured output is checked for meaning as well as shape; credentials are narrow and auditable; detection leads to containment; abstention and escalation reach tested destinations; and dependency failure enters an explicit safe mode.

The aim is not to build a system that never encounters a bad input. The HR assistant should encounter the copied instruction, refuse to grant it authority, expose the attempt to the right operator, and remain useful for the employee’s legitimate question. That is what layered control buys: one component may be mistaken without making the whole system helpless.

Source Notes