Generative AI, LLMs, and Multimodal Models

Fluent, Useful, and Wrong

A procurement team asks an internal assistant whether a supplier agreement permits customer data to be processed in a new region. The response is polished, cites a clause number, and recommends proceeding. The cited clause belongs to an older agreement. The retrieval system found a superseded document; the model composed a confident answer from the supplied context; the interface displayed the citation without its effective date.

Calling this a “hallucination” is not enough. The failure spans corpus governance, retrieval, context assembly, generation, verification, user interface, and approval workflow. Replacing the model might change the symptom without fixing the system.

Generative models are powerful pattern generators. They can draft, summarize, transform, translate, extract, classify, compare, synthesize, and produce across modalities. Their output is not a database lookup or a proof. The production discipline is to separate generation from evidence, policy, validation, and action.

That separation gives us a better question than “How do we stop hallucinations?” For each claim the assistant makes, where could support come from, what could corrupt it, and what authority—if any—may the answer acquire?

Tokens, Context, and Generated Output

A language model processes sequences of tokens: pieces of text or other encoded symbols. Given a context, it estimates a distribution over possible next tokens and repeatedly samples or selects continuations. Model parameters capture statistical structure learned during training. A context window limits how much encoded material can be considered in one request.

This mechanism explains both capability and risk. Pattern learning supports flexible language use, transformation, and approximate reasoning across many tasks. The same mechanism can produce a plausible continuation when the context lacks evidence, contains conflict, or requests something outside the model’s reliable behavior. Fluency is a property of generation, not a certificate of truth.

A foundation model supports broad downstream use. Instruction tuning and other adaptation can make it follow task descriptions more reliably. Prompting supplies instructions and examples at request time. Fine-tuning changes model behavior using curated examples. Retrieval supplies external evidence without putting every fact into parameters. These mechanisms solve different problems and require separate evaluations.

Multimodal models encode or generate combinations of text, images, audio, video, and other signals. This expands what the system can inspect, but it does not remove uncertainty. In the supplier-agreement example, a scanned appendix may contain the decisive regional restriction. Optical text extraction can drop a negation; a model can associate a footnote with the wrong table row; a signature page can be mistaken for proof that a different version took effect. The team must evaluate extraction within each modality, the links across modalities, and the final task. An apparently correct summary can conceal a broken reading of the source.

The Request Lifecycle Is the Product

A real generative request passes through more than the model.

The system authenticates the user, identifies the permitted task, and applies instruction precedence. It assembles conversational state, retrieved evidence, tool results, user input, and policy constraints. It selects a model and generation configuration. After generation, it may validate a schema, check evidence, screen policy violations, redact data, route to human review, or abstain. Logs and feedback support evaluation and incident response.

This view prevents three design errors. First, model instructions are not the only policy control. Second, content in the context can be stale, hostile, or unauthorized. Third, generated output should not automatically acquire authority merely because it fits a schema.

Follow the procurement question through the lifecycle. Authentication establishes which agreements the requester may inspect. Corpus rules determine which signed versions are in force. Retrieval finds candidate passages. Context assembly preserves the regional clause, its exception, and the effective-date metadata together. Generation drafts an explanation. Verification checks each material claim against the cited text. The interface shows the controlling version and routes any request to proceed to the authorized reviewer. Every stage changes what the answer can responsibly mean.

The wider AI-enabled system boundary supplies the ownership frame. When the request lifecycle can call tools, apply the action controls in Agents, Tools, and Autonomy.

Retrieval and Generation Are Different Components

Retrieval-augmented generation (RAG) selects external material and adds it to the model context. It is useful when answers should depend on current, proprietary, permission-controlled, or citable sources. It does not guarantee correctness.

A RAG system can fail before generation because the corpus lacks the answer, parsing lost a table, chunking separated a condition from its exception, metadata is wrong, access control is applied after retrieval, the embedding search misses a relevant passage, or reranking promotes stale evidence. Generation can then ignore, distort, or overextend good evidence. Citation code can attach a nearby source that does not support the claim.

The original RAG research pattern combined a learned retriever with a generator so outputs could condition on external documents. Production implementations vary, but the separation remains valuable: retrieval quality and answer quality are distinct hypotheses.

In the agreement corpus, retrieval success means more than finding text that resembles the question. The candidate set must contain the governing agreement, preserve the amendment that narrows it, exclude documents the requester cannot access, and expose a conflict if two records both appear current. Only then does it make sense to ask whether the generated answer uses those passages faithfully. A fluent answer cannot repair missing evidence; a perfect retrieval result cannot force the model to respect it.

Groundedness Is Not General Truth

Groundedness asks whether generated claims are supported by specified evidence. A grounded answer can still be wrong if the evidence is wrong, stale, incomplete, or inapplicable. An ungrounded answer may happen to be true but violate a system requirement to answer only from approved material.

Use a claim-evidence discipline:

1. Identify which statements require support.
2. Link each statement to the exact supporting passage or structured fact.
3. Check whether the source is authoritative, current, permitted, and applicable.
4. Detect conflicts and missing coverage.
5. Abstain or escalate when evidence does not justify a conclusion.

For high-impact decisions, generation should summarize evidence for a qualified human rather than declare an outcome. The interface should show sources, dates, document versions, and uncertainty without overwhelming the reviewer.

Choose the Intervention by the Failure

Choose the smallest intervention that addresses the failure.

A vague task or unstable output shape calls first for a clearer specification and schema validation. Facts that are current, private, or must be cited call for retrieval from governed sources. Repeated behavioral failures across a well-defined narrow task may justify evaluating fine-tuning, but only after a baseline exists and with curated examples and regression tests.

Some requirements do not belong inside generation at all. Enforce permissions and business rules with deterministic code or a policy engine. Keep high-stakes judgment with qualified people when reliable evidence or institutional authority is absent. If generated output can cause an external action, place it inside a constrained tool workflow with the approval, permission, and recovery controls developed in the next chapter.

Do not fine-tune to memorize facts that change frequently. Do not use retrieval to compensate for unclear task design. Do not rely on prompting alone to enforce access control or prevent an irreversible action.

Evaluation Must Match the Task

“The responses look good” is not an evaluation plan. Build a representative set of tasks, inputs, contexts, and adversarial cases. Record model, prompt, retrieval corpus, policy, and tool versions.

Evaluate at least these dimensions where applicable:

• task success: does the output complete the user’s job?
• correctness: are task-relevant claims accurate?
• groundedness: are claims supported by allowed evidence?
• citation precision: does each citation support the nearby claim?
• instruction following: are format and constraints respected?
• completeness: are required parts present without invented additions?
• safety and privacy: does output avoid disallowed harm or disclosure?
• calibration: does the system communicate uncertainty and abstain appropriately?
• robustness: do harmless variations preserve acceptable behavior?
• human outcome: can users detect, correct, and recover from errors?
• operations: do latency, cost, logging, fallback, and incident controls work?

Automated checks are useful for schemas, exact fields, citation links, latency, and deterministic policy rules. Human or domain-expert evaluation is needed for nuanced correctness, usefulness, tone, ambiguity, and harm. Model-based judging can help at scale but must be calibrated against human decisions and tested for bias, position effects, and shared blind spots.

Set decision rules before reviewing the final results. A minimum average score is insufficient if one severe data disclosure or unsupported high-impact recommendation should block launch.

Build the Agreement Assistant as an Evidence Service

A company now rebuilds the assistant from the opening failure. Its permitted task is narrow: explain what current supplier agreements say about a proposed processing region and show the evidence. It may not approve a supplier, interpret law, or authorize a data transfer.

The weak version uploads contract files, retrieves similar text, and asks a model to answer. It treats drafts and signed agreements alike, gives every requester the same corpus, and always responds. Its demonstration questions contain no amendments, scanned pages, access restrictions, or missing answers. The system looks capable because the test has removed the work.

The stronger version gives each document an owner, agreement identity, counterparty, region, effective period, execution status, supersession link, permission policy, and extraction provenance. A scanned page and its extracted text remain linked so a reviewer can inspect the original. Retrieval applies user permissions before ranking. Context assembly keeps qualifications and amendments beside the clauses they modify. A conflict detector stops the answer when version records disagree.

The answer links each material claim to an exact passage and displays the agreement version and effective date. When evidence is absent or conflicting, it says what is missing and routes the question to the contract owner. A request to “approve this exception” is not made safer by a refusal phrase alone; the assistant has no approval capability to invoke.

Its evaluation set includes straightforward questions, superseded agreements, amendments that reverse a clause, regional differences, missing documents, scans with damaged text, tables whose footnotes change their meaning, adversarial instructions embedded in files, personal data, and requests for unauthorized approval. The team measures whether retrieval found the controlling evidence before it measures whether generation used that evidence. It then tests claim support, conflict handling, abstention, escalation, latency, and whether reviewers can detect and correct mistakes.

In production, the team samples answers and watches unresolved abstentions, stale-document alerts, extraction failures, complaints, and contract-owner response time. The result is not a smarter chatbot. It is a governed evidence service with generation inside it.

Failure Modes and Layered Controls

Plausible fabrication: the model supplies unsupported details. Require evidence for material claims, validate, and abstain.

Context overflow or dilution: important instructions or evidence are lost in excess material. Curate context, test long requests, and prioritize relevant evidence.

Prompt injection: user or retrieved content attempts to alter system behavior. Treat external content as data, isolate instructions, restrict tools, validate actions, and monitor attacks.

Stale or conflicting sources: retrieval returns obsolete guidance. Version documents, enforce effective dates, detect conflicts, and assign corpus owners.

Sensitive-data exposure: prompts, logs, embeddings, documents, or output reveal restricted information. Minimize data, enforce permissions before retrieval, redact where justified, control retention, and test leakage.

Overreliance: users accept fluent output without checking. Present evidence and limitations, design correction and escalation, train users, and measure reliance behavior.

Silent model or provider change: behavior shifts without system review. Pin versions where possible, run regression evaluation, maintain change records, and keep fallback and rollback plans.

NIST’s Generative AI Profile describes risks and actions for managing generative AI across the AI RMF functions. Treat it as a risk-management reference, not as proof that a particular system is acceptable. The use case, evidence, tests, controls, owners, and residual-risk decision remain local responsibilities.

Write a Release Claim the Evidence Can Carry

Before release, describe the system in one decision record. Name the permitted task, users, affected parties, and prohibited outcomes. Record the model, prompts, corpus, permissions, retention rules, policies, and owners as versioned parts of one system. State which tests cover retrieval, context assembly, generation, claim verification, the interface, and the human workflow; include missing answers, conflicting sources, malicious context, sensitive data, and harmless prompt variation.

Then state the release thresholds and the failures that block launch regardless of an average score. An unsupported recommendation about a high-impact decision, disclosure of a restricted agreement, or silent use of a superseded contract may each be such a blocker. Name the person authorized to accept the remaining risk, the correction and escalation paths available to users, the production signals that trigger review, and the version to which the team can fall back.

Return to the polished answer about processing data in a new region. The corrected system may reach the same conclusion, a different one, or no conclusion at all. What has changed is that the reviewer can see which agreement governs, how its clause supports the answer, where uncertainty remains, and why the assistant cannot authorize the transfer.

Generation creates a candidate. Evidence and policy determine whether it is supportable. Validation and human judgment determine whether it is usable. The action boundary determines how much harm a mistake can cause.

Source Notes

• NIST AI RMF: Generative Artificial Intelligence Profile, NIST AI 600-1, generative-AI risk management guidance released 2024-07-26; verified 2026-07-19.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, foundational RAG research separating retrieval of external evidence from sequence generation; verified 2026-07-19.
• Google Machine Learning Crash Course: Introduction to Large Language Models, current official primer on tokens, context, and LLM training and use; verified 2026-07-19.