What AI Is, and What AI Systems Are

The Demo That Hid the System

A support director watches a convincing demonstration. An assistant summarizes a customer’s history, finds a policy, drafts a reply, and recommends a refund. The model responds in seconds. The room starts discussing launch dates.

Then an operations lead asks ordinary questions. Which customer records enter the prompt? Can the assistant retrieve documents the support agent is not allowed to open? Does it distinguish the current refund policy from an archived one? Who checks a refund recommendation before money moves? What is logged? What happens when the model provider is unavailable? Who owns an incident in which confidential data appears in a reply?

Those questions do not concern the model in isolation. They concern the product and its operating environment. A fluent response can make a model visible while hiding the ingestion pipeline, permissions, retrieval index, business rules, user interface, review workflow, monitoring, fallback, and accountable owners. Yet those hidden components determine whether the assistant is useful, safe enough for its context, and recoverable after failure.

The central model for this handbook is therefore simple: an AI model produces behavior; an AI-enabled system turns that behavior into an outcome. Design, evaluation, and accountability must follow the whole system boundary.

A Working Vocabulary

Artificial intelligence is an umbrella term for computational capabilities associated with perception, language, prediction, generation, recommendation, planning, optimization, or decision support. The boundary of that umbrella is partly technical and partly conventional; it changes as once-novel methods become ordinary software.

Machine learning fits behavior from data, examples, objectives, or interaction instead of specifying it entirely with hand-written rules. Deep learning does that fitting with multi-layer neural networks that learn representations as well as behavior. A foundation model is trained broadly enough to support many downstream tasks through prompting or adaptation.

Other labels describe what a model handles or produces. Generative AI produces text, images, audio, video, code, or other structured content. A large language model processes token sequences and predicts distributions over possible continuations. A multimodal model accepts or produces more than one kind of signal. An agent is a larger system again: it can select actions, use tools, and change an environment across one or more steps.

These terms are not a maturity ladder. A deterministic rule can be safer and better than a learned model. A small classifier can outperform a general model on a narrow task. Retrieval, search, workflow redesign, or a human service can solve a problem without generation. Labels help teams communicate, but they do not substitute for a system description.

Read this nesting as the dominant modern lineage, not as perfect set theory. Generative methods predate foundation models, and a production product can combine learned models, rules, search, and ordinary software.

The AI-Enabled System Stack

A useful system boundary contains at least seven layers.

Data includes training examples, prompts, retrieved documents, features, labels, feedback, logs, and rights to use them. Data has provenance, access rules, retention requirements, missingness, bias, and time dependence.

Model is the learned or configured component that maps input to output. It has a version, capabilities, limitations, cost, latency, and failure distribution. It may be hosted internally or supplied through an external service.

Orchestration assembles context, selects models, retrieves evidence, calls tools, handles state, applies policies, retries failures, and decides when to stop. This layer often creates risks that a model benchmark cannot reveal.

Product presents capabilities through an interface and connects output to a business rule or action. A score shown as advice is different from the same score automatically rejecting a transaction.

Human workflow determines who reviews, corrects, overrides, appeals, or acts. Meaningful oversight requires time, authority, information, training, and incentives; a nominal reviewer is not a control.

Monitoring measures service health, task quality, safety events, drift, cost, complaints, and downstream outcomes. It connects detected problems to a named response playbook.

Governance assigns owners, acceptable-use boundaries, evidence requirements, approval gates, residual-risk decisions, change control, incident accountability, and retirement authority.

Four Different Things Teams Call AI

The same interface can conceal very different operating patterns.

A customer-support assistant may use all four. It predicts intent, retrieves and generates a reply, recommends a resolution, and perhaps calls a refund API. Evaluating only response fluency would ignore routing errors, retrieval access, refund authority, and recovery after a duplicate action.

Draw Boundaries Around Consequences

A system boundary should include anything that can materially change the output, action, exposure, or ability to recover. Start with the user outcome and trace backward and forward.

For a support assistant, trace the request from authentication to conversation history, policy retrieval, context assembly, model call, output filter, interface, agent review, customer delivery, feedback, logs, and incident response. Add external dependencies and manual work. Mark which components can read confidential data, move money, send messages, or alter records. Name the owner at each consequential edge.

Then distinguish four boundaries:

1. Information boundary: what data can enter, be inferred, be retained, or leave.
2. Decision boundary: where a score or generated output becomes a recommendation, approval, rejection, or action.
3. Permission boundary: which identities, models, tools, and people can read or change which resources.
4. Accountability boundary: who accepts risk, monitors performance, handles incidents, and can pause the system.

A diagram is incomplete when it has arrows but no owners, permissions, failure paths, or stop authority.

Use How Machines Learn from Data when the boundary contains a learned component, and Agents, Tools, and Autonomy when generated output can cross into action.

Evaluation Follows the Boundary

Model evaluation asks whether a component performs a defined task on a defined dataset. System evaluation asks whether the complete arrangement produces acceptable outcomes under realistic conditions.

For the support assistant, component tests might measure intent classification, retrieval recall, groundedness, or format compliance. End-to-end tests must also cover authentication, document permissions, stale policies, ambiguous requests, escalation, latency, provider failure, duplicate tool calls, user correction, and audit reconstruction. Online monitoring must connect those technical signals to customer resolution, complaint rate, refund error, reviewer workload, and privacy or security events.

Acceptance criteria need four kinds of evidence:

• task evidence: the output meets the defined use-case requirement;
• risk evidence: known failure modes are tested and bounded;
• operational evidence: monitoring, fallback, rollback, and incident ownership work;
• human evidence: users and affected people can understand, correct, contest, or escalate as the context requires.

A strong model score cannot compensate for a broken action path. A modest model may be entirely adequate inside a carefully constrained workflow.

Governance Is Part of the Architecture

Governance is not a document written after engineering. It is the allocation of decision rights around the system.

The product owner defines the permitted use and success outcome. The technical owner controls architecture, versions, and operations. The data owner governs sources, rights, quality, and retention. Security and privacy owners review exposure and controls. A domain owner defines unacceptable errors. A risk owner accepts or rejects residual risk. An incident owner can contain harm. An executive sponsor decides whether the benefit justifies the operating commitment.

The core artifact is a system boundary record containing purpose, users and affected parties, data flows, model and provider versions, orchestration, action path, permissions, human oversight, evaluation, monitoring, owners, change triggers, and stop conditions. Keep it current as the system changes.

NIST’s AI Risk Management Framework is voluntary guidance for managing risks to individuals, organizations, and society across the design, development, use, and evaluation of AI products, services, and systems. Its system-level scope is a useful reminder that trustworthy behavior cannot be assigned to a model alone. NIST notes that AI RMF 1.0 is under revision as of this review, so teams should version the framework references they adopt rather than citing “NIST compliance” as a timeless state.

The Same Assistant, With a Different Boundary

Return to the support assistant from the opening demonstration. Two service teams can license the same model and build materially different systems around it.

Team A gives the assistant broad access to the document index and lets it place refund recommendations directly into the payment queue. Agents are told to review the draft, but the interface hides the retrieved policy version and makes edits cumbersome. The team logs the final reply, but not the model version, source context, or refund call. It discovers stale-policy errors through customer complaints.

Team B limits retrieval to documents the signed-in agent may open, shows the policy passages behind the draft, and requires confirmation before a refund call. It records model, prompt, source, reviewer, and action versions; samples corrections for quality review; and routes serious disclosure or payment errors into an incident process. The same model behavior has a smaller information and action boundary, better evidence, and a recoverable workflow.

Team B has not made the model infallible. It has made uncertainty visible and responsibility enforceable.

Field Check: Model or System?

Before approving an AI proposal, ask for a one-page account of the system. It should make the input, output, user, affected parties, and action path concrete. It should separate learned behavior from rules, thresholds, and human decisions. It should show where data comes from, who may use it, what leaves the organization, what is retained, and what is logged.

Then follow the consequences. List everything the system can change: messages, records, money, permissions, code, or some other part of the world. Name the component, end-to-end, human, risk, and operational evidence required for approval. Identify who can review, override, hear an appeal, roll back, stop the system, and lead an incident.

Finally, state what invalidates that approval. A changed model, dataset, prompt, policy, vendor, workflow, or risk profile can change the system even when the product name and interface stay the same.

If the proposal cannot answer these questions, it is still a capability demonstration. The engineering work begins when the system and its consequences become visible.

Source Notes

• NIST AI Risk Management Framework, including the current revision notice; verified 2026-07-19.
• NIST AI RMF 1.0, voluntary framework for incorporating trustworthiness considerations into AI design, development, use, and evaluation.