Appendix J: AI Monitoring Plan Template
Turn production signals for service, data, model, safety, users, and cost into owned thresholds, responses, verification, and re-evaluation.
The Dashboard Is Green
A support assistant’s document migration finishes before dawn. The index job reports success, latency and availability remain green, and model-call errors do not move. By midday, agents are quietly rewriting answers because the assistant cites procedures for the previous product version. Fewer answers reach customers, but no alert fires.
The system is available and the workflow is failing. Monitoring closes that gap only when an observation can change a decision. A useful signal is connected to an expected range, a decision threshold, a named responder, an authorized response, and a verification step. Everything else is telemetry awaiting a purpose.
Work Backward From a Decision
Begin with the decisions the team may need to make: continue, investigate, restrict traffic, fall back, roll back, declare an incident, or reopen evaluation. Then list the launch claims, critical controls, known limitations, expected operating envelope, and credible incident scenarios that could force those decisions. For each one, ask what evidence would show improvement, degradation, abuse, or loss of control.
Do not begin by filling every category with metrics. A small plan that can reveal and contain the most consequential failures is stronger than a comprehensive dashboard nobody can act on. Some outcomes arrive late, so pair them with leading evidence. Task success may take days to establish; retrieval misses, correction spikes, policy flags, or a saturated review queue can open an investigation sooner. The proxy remains a clue, not a substitute for the outcome.
Do not turn drift into an automatic retraining command. A changed distribution may reflect a seasonal event, data pipeline defect, product launch, policy change, attack, or genuine concept change. The first response is diagnosis against the decision context; retraining is one possible controlled change.
AI Monitoring Plan
SYSTEM AND OWNERSHIP
System and production version:
Model, prompt, retrieval, data, policy, tool, and code versions:
Business / product owner:
Technical and on-call owner:
Risk, security, privacy, and domain contacts:
Monitoring plan version and date:
OPERATING ENVELOPE
Approved users, contexts, regions, languages, channels, and volume:
Intended actions and authority limits:
Launch claims and acceptance thresholds:
Known limitations and excluded uses:
Critical controls that must remain effective:
SIGNAL RECORD (repeat for every signal)
Signal name and category:
Question or failure it detects:
Definition, numerator / denominator, and unit:
Source and collection method:
Release, component, workflow, and required segment dimensions:
Baseline, expected range, and uncertainty:
Warning threshold and window:
Critical threshold and window:
Data delay, blind spots, and false-alert risks:
Privacy classification, access, and retention:
Dashboard / query / evidence location:
Severity, notification route, and response target:
Primary owner, backup, and decision authority:
Response playbook:
Recovery verification and close condition:
Last test, last trigger, and retirement condition:
SERVICE AND DEPENDENCY HEALTH
[ ] Availability and error rate [ ] End-to-end latency
[ ] Throughput and queue depth [ ] Provider / tool failure
[ ] Timeout, retry, and fallback [ ] Capacity and saturation
[ ] Version mismatch or stale dependency
DATA AND RETRIEVAL HEALTH
[ ] Schema and validation failures [ ] Missingness and outliers
[ ] Freshness and update lag [ ] Distribution shift
[ ] Coverage by critical segment [ ] Label or feedback delay
[ ] Corpus ingestion failure [ ] Retrieval recall proxy
[ ] Permission-filter violation [ ] Citation/source freshness
MODEL AND GENERATIVE QUALITY
[ ] Task quality and calibration [ ] Prediction distribution
[ ] Abstention and uncertainty [ ] Groundedness / factuality
[ ] Citation correctness [ ] Refusal and policy behavior
[ ] Segment and language quality [ ] Regression-suite result
AGENT, TOOL, SAFETY, AND SECURITY
[ ] Tool-call failure and retry [ ] Approval bypass attempt
[ ] Unauthorized or blocked action [ ] Loop / budget overrun
[ ] Prompt-injection indicators [ ] Sensitive-data flag
[ ] Harm or policy violation [ ] Abuse and anomaly signal
[ ] Kill-switch and fallback health
USER, HUMAN, AND BUSINESS OUTCOMES
[ ] Task success and correction [ ] Adoption and abandonment
[ ] Override and escalation rate [ ] Review queue and time
[ ] Appeal, complaint, and remedy [ ] Accessibility outcome
[ ] Business value and counterfactual
COST AND RESOURCE USE
[ ] Cost per task / successful task[ ] Token or compute use
[ ] Cache and routing efficiency [ ] Budget threshold
[ ] Human review and support load [ ] Environmental proxy if material
ALERT ROUTING AND RESPONSE
Severity levels and paging policy:
Correlation / deduplication rules:
Business-hours versus continuous coverage:
Triage owner and acknowledgement targets:
Pause, rollback, fallback, or traffic-limit authority:
Incident-declaration triggers:
User and stakeholder communication route:
REVIEW AND LEARNING
Daily / weekly / monthly review forums:
Who reviews segment and delayed-outcome evidence:
Threshold-tuning process:
How alerts become defects, evaluations, or regression tests:
Re-evaluation and change-request triggers:
Next plan review and approvers:
DECISION RECORD (repeat for alerts and scheduled reviews)
Time, release, affected scope, and evidence:
Decision and decision owner:
Containment or change performed:
User / stakeholder communication:
Verification result and remaining uncertainty:
Defect, evaluation, regression, or plan update created:
Give Each Signal a Route
Page immediately only when a responder has both urgent work and authority to do it. Route slower quality questions into an owned investigation queue. Bring delayed outcomes and segment evidence to a scheduled decision forum. A notification route is part of the control: a critical cross-tenant event sent to an unstaffed mailbox is not monitored.
Keep distinct signals available for diagnosis, but correlate them when urgency depends on their conjunction. Rising retrieval misses, falling citation correctness, and increased human correction are more useful as one investigation trigger than as three disconnected alerts. Do not compress them into a score that hides which condition changed.
Thresholds need hysteresis or recovery windows so a signal does not flap around one boundary. Record why an alert fired, which version and segments were affected, what action followed, and whether the response improved the intended outcome. Regularly retire alerts that do not change decisions and add signals for failures discovered through incidents, appeals, or manual review.
Work the Migration Failure
In this modeled plan, the assistant’s team keeps service latency and availability, but neither can detect the migration failure. The launch claim is that agents receive an answer supported by a current procedure. The team therefore links three pieces of evidence: the share of current-product queries retrieving archived documents, citation correctness on a blinded sample, and the severity of agent corrections. Each is segmented by repository, product version, language, and release bundle.
After the migration, archived-document retrieval crosses its warning threshold while the sampled quality result has not yet arrived. The alert opens an investigation rather than paging the model team. A trace shows that the documents are fresh but their product-version metadata was lost during ingestion. The content-operations owner pauses the affected connector, routes its queries to the existing search workflow, and rebuilds the index from the last verified manifest. Retraining would not repair any of those failures.
The rebuild makes the freshness gauge green again, but the responder does not close the alert. Recovery requires the archived-document rate to remain within range through the recovery window, the seeded migration case to pass, and the next blinded sample to meet the citation floor. The broken case becomes an ingestion regression test. The decision record preserves why the connector was paused, who restored it, and what evidence justified restoration.
Other signals need different interpretations. If abstention rises after a restricted source is removed, the increase may show that a control is working. The responder compares source availability, task success, and escalation before changing the threshold. A number earns its place in the plan by opening the right question, not by moving in the preferred direction.
Rehearse the Plan While One Signal Lies
Before approval, run a tabletop or controlled exercise in which one reassuring signal is wrong. Let the ingestion job report success while coverage falls, let aggregate quality hide a language regression, delay the outcome labels, or make the primary responder unavailable. Start from the first observable evidence and follow the actual notification route. Require the backup to find the affected release and segments, choose a bounded response, communicate when needed, and prove recovery.
The rehearsal should expose an operational fact, not end with a promise to watch the dashboard more carefully. Repair the query, ownership, authority, playbook, fallback, or evidence path that failed. Record any failure that remains invisible and decide whether complementary evidence, a narrower launch scope, or an explicit residual-risk decision is required.
A reviewer should be able to choose one material launch claim and follow it through the completed plan: what signal challenges the claim, what the signal cannot see, who receives it, what that person may do, how affected people are protected meanwhile, and what evidence closes the loop. If the route breaks, the plan is not yet an operational control.
Use this plan with Observability, Monitoring, and Drift, the AI Evaluation Plan, and the AI Incident Report Template.