Training, Fine-Tuning, Prompting, and Retrieval

The Fine-Tuning Project That Could Not Fix Missing Evidence

An internal policy assistant gives a confident answer about parental leave. The answer was correct last year. It is wrong for the employee’s region today, and the assistant supplies no clause that a reviewer can inspect.

The team proposes fine-tuning on the current employee handbooks. That sounds plausible because the model lacks the right answer. Yet the failure is not one thing. The application did not identify the employee’s region. Its corpus contains superseded and current documents without effective dates. Search returned the older version. The prompt allowed an answer without evidence. Fine-tuning could make the model repeat today’s policy more often, but it would hide a changing fact inside weights while leaving identity, versioning, access, and citation broken.

The useful question is therefore not “Should we fine-tune?” It is: which layer owns the observed failure, and what is the smallest change that can make that layer meet its acceptance rule? Prompts specify a task. Retrieval supplies external evidence. Code and workflow enforce controls. Fine-tuning changes recurring learned behavior. Pretraining creates broad capability. Treating these mechanisms as substitutes produces systems that are expensive to change and difficult to trust.

Follow the Failure to Its Owner

Before changing anything, preserve the failed input, output, available context, retrieved passages, model and prompt versions, user permissions, and evaluator decision. Then state the consequence and the acceptance rule. For the leave question, a useful rule might require the correct effective policy for the employee’s region, clause-level support, no cross-region disclosure, and abstention when authoritative evidence is absent.

Now rerun the trace one boundary at a time.

If the region never reached the application, model adaptation cannot recover it reliably; the interface or workflow owns the gap. If the correct document was absent from the authorized corpus, ingestion owns it. If it was present but search missed it, retrieval owns it. If the passage reached the model but the answer contradicted it, context assembly, task specification, or model behavior may own it. If the answer was right but the application exposed a restricted clause, permissions own the failure. A model can participate in all these paths without owning all their defects.

This trace also prevents a common measurement mistake. “Answer quality rose” is not enough if the new system cites the wrong version, leaks a document, stops abstaining, or exceeds the latency the workflow can tolerate. The acceptance rule must include the severe failures and operating limits that an average score would conceal.

The ladder is a search order, not a claim that every system should climb it. Lower layers are often easier to inspect, patch, and roll back. Move upward only when a measured gap survives the changes below it.

Put Exact Requirements Outside the Model

The assistant must know which policy may be shown to which employee. That requirement belongs in identity, authorization, query filters, and application code—not in a sentence asking the model to respect permissions. The same is true for arithmetic, schema validity, transaction limits, approval gates, and actions that must never occur without human authority.

Deterministic does not mean infallible. Rules can be incomplete, identity data can be stale, and approval queues can fail. The advantage is a clear enforcement point with explicit tests and ownership. For the policy assistant, the application resolves region and employment class, filters sources before retrieval, rejects answers without valid citations, and routes ambiguous cases to HR. Those controls remain in force no matter which prompt or model is active.

Use a tool when the answer depends on a live calculation, database result, status, or transaction rather than a body of documents. A leave-balance service can return the employee’s current entitlement; retrieval should not approximate it from prose. Tool calls still need authorization, input and output validation, timeouts, provenance, and a safe failure path.

Make the Prompt a Task Contract

Once the workflow supplies the right inputs and controls, the prompt can state what the model is being asked to do with them. For this assistant, the contract names the task, the allowed evidence fields, the output schema, the required clause support, the uncertainty rule, and the conditions for abstention or escalation. A few examples may clarify boundaries such as conflicting effective dates or a policy that is silent on the question.

Instruction authority must remain visible. Stable system and application instructions outrank user requests; retrieved policies and user text are untrusted data, even when they contain imperative sentences. Delimit that content, restrict tools independently, and test prompt injection as a system attack. Wording cannot safely grant or revoke permission.

A prompt package is a release artifact. Version its instructions, examples, model settings, schema, owner, evaluation results, and rollback target. Longer prompts are not automatically stronger: irrelevant examples can skew behavior, competing instructions can create ambiguity, and excess context consumes latency and attention. The policy team tests paraphrases, conflicting documents, missing evidence, hostile text inside documents, and requests outside the assistant’s authority.

Prompting is the first choice for a missing or ambiguous task specification because it is cheap to inspect and change. It cannot supply evidence the application never provided, guarantee an exact business rule, or teach a base model a broad capability it does not possess.

Retrieval Keeps Changing Facts Outside the Weights

Retrieval-augmented generation pairs a generator with a separate information-retrieval system. The separation is valuable when facts are current, private, permissioned, frequently revised, or expected to carry citations. The model need not memorize the leave policy if the system can supply the applicable version at answer time.

A production path has two connected flows:

approved sources -> parse -> structure-aware chunks -> metadata -> index
identity + question -> authorization filters -> search -> rerank
-> assemble context -> generate -> verify support -> answer or abstain

Embeddings map content into vectors so that semantically related text can be retrieved even when query and document use different words. Vector search is useful, but it is not authority. Exact terms, dates, jurisdictions, identifiers, and access classes may need keyword search and metadata filters. A reranker can then score a smaller candidate set with richer query-document evidence. The final context should preserve headings, effective dates, parent sections, and exceptions; a paragraph detached from its governing condition may reverse the policy.

For the leave assistant, the corpus records source owner, jurisdiction, employment class, effective interval, superseded status, access class, and deletion state. Authorization filters apply before generation. The team tests cross-region, revoked-access, cache, and index paths rather than hiding forbidden passages after the model has seen them.

Retrieval creates its own chain of possible failures. The authoritative answer may be missing. Parsing may drop a table. Chunking may split an exception from its rule. Search may miss the clause. Reranking may prefer a superseded version. Context assembly may mix jurisdictions. The generator may ignore correct evidence, and a citation may point to a passage that does not support the claim.

Evaluate those boundaries separately: corpus coverage, retrieval recall, ranking, context integrity, answer groundedness, citation support, permission enforcement, freshness, and deletion. A current model over a stale index is still a stale system.

Fine-Tune Only the Behavior That Remains

After workflow, prompt, and retrieval changes, suppose the assistant reliably finds the right policies but inconsistently classifies questions into the organization’s stable routing categories. The failure recurs across paraphrases, materially affects handoff, and remains below threshold with the prompted baseline. This is a credible fine-tuning candidate because the desired change is persistent behavior rather than changing knowledge.

Fine-tuning continues training a pretrained model on task-specific examples. Instruction tuning is fine-tuning on examples expressed as instructions and desired responses so that the model learns to follow that class of task. Parameter-efficient methods, including adapters, update or add a smaller set of parameters while keeping most base weights fixed. They can reduce storage and training burden and make variants easier to manage, but they do not remove the need to evaluate the composed base model, adapter, prompt, and workflow.

Useful targets include stable classification boundaries, specialized transformations, terminology, output behavior, and style at sufficient scale. Poor targets include current facts, user-specific permissions, hard constraints, missing workflow inputs, or a few examples that a prompt or validator fixes directly.

The training set must teach the boundary, not merely display ideal answers. Each example should map to a recurring failure class. Include difficult valid cases, neighboring categories, ambiguous inputs, insufficient-evidence cases, refusals, and escalation behavior in proportions that reflect the deployment risk. Record provenance, rights, annotation guidance, segments, languages, deduplication, and dataset version.

Keep evaluation separate. Split by entity, source, time, and template wherever similar examples could cross the boundary. Near-duplicates in tuning and evaluation can make memorization look like generalization. Maintain a sealed final set that is not repeatedly consumed by prompt and tuning decisions.

The policy team ships the adapted model only if the routing slice improves without degrading policy accuracy, citation support, abstention, access control, important employee segments, latency, or rollback. A better average that creates a severe new failure is not an improvement.

Pretraining Needs a Capability Case

Training from scratch is not the next rung for an application team that has exhausted prompt variations. It is a different undertaking: acquiring broad capability by training on a large corpus, with corresponding demands for data rights, compute, specialist expertise, security, evaluation, deployment, and lifecycle ownership.

It can be justified when available models cannot meet a required modality, language coverage, deployment boundary, intellectual-property position, scale economics, or research objective. Even then, compare a different existing model, open weights, a specialized small model, distillation, retrieval, and fine-tuning. The burden must be earned by a capability and control requirement, not by dissatisfaction with one application’s answers.

Run the Adaptation as an Experiment

The policy assistant now has several plausible changes. To learn which one works, freeze the baseline and the evaluation slices before optimizing:

1. Record the model, prompt, corpus, index, retriever, reranker, tools, policy code, and thresholds.
2. Preserve representative failures and define acceptance rules, severe-error limits, segments, latency, cost, and security gates.
3. Change one owning layer where practical.
4. Test the changed component and the complete path.
5. Compare new failures and operating burden with the measured benefit.
6. Accept, reject, narrow, or combine the intervention with a recorded reason.

An ablation is especially revealing when several changes arrive together. Remove the new prompt, reranker, metadata rule, or adapter in turn. If quality does not change, that layer may be unnecessary. If the result collapses only when the authorization filter is removed, the team has learned which control carries the safety property.

The leave-policy failure ultimately needs no fine-tuning. Region becomes required workflow context; permission and effective-date filters constrain retrieval; hybrid search and reranking find the applicable clause; the prompt requires a supported answer or abstention; schema and citation checks reject malformed output. The model weights remain unchanged because none of the measured defects required a learned behavior change.

That outcome is not a modest form of adaptation. It is the engineering result: each failure has an owner, each change has evidence, and each behavior-changing component can be versioned and rolled back independently.

Record the Decision Another Team Must Inherit

An adaptation record should let a reviewer reconstruct the argument:

1. Failure and consequence: preserved examples, affected slices, severity, frequency, and current owner.
2. Baseline and acceptance: exact system versions, thresholds, severe errors, segments, latency, cost, and security gates.
3. Candidate change: workflow, control, prompt, retrieval, tool, fine-tune, model swap, or pretraining—and the gap it is expected to close.
4. Data and evidence: sources, rights, quality, split, contamination controls, component tests, and end-to-end results.
5. Trade-offs: regressions, new permissions or injection paths, freshness, operational load, uncertainty, and residual risk.
6. Decision and release: accept, reject, combine, or narrow; name owners, approvals, artifact versions, monitoring, rollback, and revisit triggers.

Try the same reasoning on a technical-support assistant that gives a correct remedy for the wrong product version. Determine what evidence would distinguish a missing task field, a retrieval miss, mixed context, ignored evidence, and a persistent sequencing failure. Propose the smallest intervention for each diagnosis, then name one result that would disprove your choice. The point is not to climb the ladder. It is to stop at the layer whose change survives a fair test.

Chapter 13 chose the least complex architecture that could satisfy the production contract. This chapter changes the least costly behavior layer when that architecture misses a measured requirement. The next obligation is to bind the exact model, data, prompt, retrieval, policy, code, and evaluation versions into a release another team can inspect.

Source Notes

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, the original RAG formulation combining parametric and non-parametric memory and motivating provenance and updateability; verified 2026-07-10.
• NIST glossary: retrieval-augmented generation, concise definition of RAG as a generative system paired with a separate retrieval system; verified 2026-07-10.
• Google Cloud: deploy and operate generative AI applications, operational guidance on prompts, retrieval, tuning, artifact versioning, and evaluation-driven iteration; last reviewed by the publisher 2024-11-19 and verified here 2026-07-10.
• NIST AI RMF Generative AI Profile, risk-management actions for generative systems including provenance, adaptation documentation, evaluation, and lifecycle controls; verified 2026-07-10.