From Vague Idea to Testable Use Case

One Idea, Three Different Systems

“Use AI to improve customer support” arrives in a planning document. Product imagines an assistant that drafts replies. Operations imagines automatic ticket routing. Finance imagines fewer agents. Security imagines a system reading every customer record. The apparent agreement is only agreement about a slogan.

Model selection would make the disagreement harder to see. A fluent prototype could look like progress before anyone had agreed on the user, the permissible data, the action after an output, or the cost of a mistake. The team first needs a use case: a contract linking one problem in one workflow to an input, an AI role, an output, an action, measurable consequences, and people with authority to stop.

Begin with the work as it exists. A support ticket enters a queue. An agent identifies the order, decides what the customer is asking, searches policy, checks account state, chooses a permitted remedy, writes a response, and sometimes seeks approval. The team should observe where time and errors actually accumulate, including workarounds and exceptions. Conflicting policy documents create a retrieval problem. Missing intake information creates an input problem. Unclear refund authority creates a governance problem. Faster drafting solves none of them.

That workflow contains at least three possible interventions:

1. Classify an authenticated ticket into the existing service taxonomy and recommend a queue.
2. Retrieve current, permitted policy evidence for an agent’s query.
3. Generate a response draft from the conversation and approved evidence.

Each verb commits the system to a different output and test. Other use cases may predict an outcome, rank options, extract fields, detect anomalies, recommend to a named decision-maker, or act through a bounded tool. “Help” and “improve” conceal these choices.

Now choose the boundary. Which of the three support systems would you release first? Do not answer from the apparent sophistication of the model. Ask which problem has been observed, which baseline is credible, which errors can be detected, and which action is easiest to contain. In this workflow, permission-aware retrieval may be the strongest first candidate: it addresses the search bottleneck without yet drafting a claim or changing a record. Observation might still show that reconciling the policy corpus beats building any AI system.

Follow the Output Until Something Happens

The retrieval candidate does not end when it returns passages. An agent receives them, checks whether they address the request, decides on a remedy, and writes a response. The result therefore needs document identity, version, effective date, and a link back to the source. The agent must be able to reject it and search again. The system must record enough to investigate stale or unauthorized retrieval without copying sensitive content into unnecessary logs.

This action path defines the system more precisely than the label “search assistant.” The same passages inserted directly into an automatically sent answer would create a different use case. So would evidence shown to a refund approver, or evidence used by a tool that changes an order. Every additional action changes the necessary permissions, oversight, evaluation, and recovery.

Trace the path with six questions: What enters? What leaves? Who receives it? What decision or action follows? How can a person correct or contest it? What happens when the output is absent, late, contradictory, or wrong? If the team cannot answer in the same way, it has not yet described one system.

Negative scope is part of the contract. The retrieval system does not draft the answer, approve compensation, alter the account, or search documents outside the agent’s existing access. These exclusions are not promises about a later model. They are boundaries that architecture and tests must enforce now.

Make the Baseline Face the Same Cases

The honest alternative is not “no help.” It may be repaired document ownership, a clearer navigation tree, keyword search, a rules-based router, a better intake form, or a staffed specialist queue. Test the strongest credible alternative on the same cases and outcomes as the AI proposal.

For retrieval, the comparison set should contain ordinary questions, ambiguous phrasing, conflicting documents, recently changed policy, restricted records, uncommon products, and requests for which no approved answer exists. Compare permission-aware retrieval with the current search experience and with a simpler improved search. If corpus repair solves the bottleneck, that is a product success even though the proposed AI system is not built.

Turn Consequences Into Measures

A convenient model metric cannot stand in for the result. Retrieval recall may describe whether the relevant passage appeared among the results, but the business and human question is whether the agent reached a correct resolution without creating new burdens or exposure.

The contract needs four connected kinds of evidence. Task evidence covers relevant and current passages, abstention when evidence is absent, and zero unauthorized results. Workflow evidence covers time to useful evidence, repeated searches, escalation, and rework. User and affected-party evidence covers correct resolution, accessible communication, uneven outcomes across relevant groups, and a usable correction path. Operational evidence covers latency, cost, availability, incident load, corpus drift, and rollback.

Write thresholds before the pilot. A team might require a defined improvement in time to useful evidence while setting unauthorized retrieval as a release blocker and placing limits on stale results and correction load. The values must come from the workflow’s baseline, consequence, and operating capacity; no percentage is universally responsible.

Error costs decide whether the output may advise, requires review, or should never be produced. For each important failure, name the affected party, severity, detectability, reversibility, and response. A missed relevant passage may cause delay. A stale cancellation policy may create an incorrect promise. A cross-account result is a privacy and authorization failure. An apparently complete result when no approved evidence exists invites over-reliance. These events need separate tests because an average quality score can hide every one of them.

Early estimates may remain qualitative when evidence is thin. Precision comes from better cases and observation, not decimal places attached to guesses.

Put Data Rights Inside the Boundary

“Uses support data” does not specify an input. For the retrieval candidate, name the policy collections and field classes it may index, the owner of each, the purpose and permission for use, the effective-date rules, retention, geography, and every downstream processor. Identify what may enter queries, embeddings, logs, evaluation sets, feedback, and human review queues. State what is prohibited.

Available data is not automatically necessary data. A system that needs every customer record to retrieve policy has probably inherited the vagueness of the original idea. Data minimization makes the permission model, threat model, evaluation set, and operating cost easier to understand.

Write the Use Case Contract

The AI Use Case Canvas is a drafting order, not a form to complete with adjectives. Write its fields as a short account another team can challenge:

1. Name one bounded task and the user problem it addresses, including frequency and consequence.
2. Describe the current workflow: actors, decisions, systems, queues, exceptions, workarounds, and appeal.
3. Name direct users and people affected by the eventual action, including those who never see the interface.
4. State the AI role with a precise verb, its negative scope, and the strongest non-AI baseline.
5. Specify inputs and data rights: sources, provenance, permission, sensitivity, coverage, freshness, retention, and prohibited uses.
6. Follow the output to its recipient, decision, action, authority, correction path, record, and failure behavior.
7. Define oversight as real work: when review occurs, what evidence the reviewer sees, expected workload, escalation, override, and quality assurance.
8. Set task, workflow, user, business, risk, and operational measures against representative cases and the baseline.
9. Record important error costs, risk hypotheses, deployment context, preventive and detective controls, fallback, and rollback.
10. Name owners and reviewers by decision right, then state what evidence blocks development, permits a pilot, pauses operation, forces rollback, or retires the system.

“Accurate,” “safe,” “responsible,” and “seamless” do not complete any field. Each needs a context, observable test, threshold, and owner.

The retrieval candidate can now be stated without the original ambiguity:

When an authenticated support agent handles an existing-order question, retrieve current approved policy passages permitted by that agent’s access. Return source identity, version, and effective date for review; do not draft a response, change an account, or search customer records. Compare with repaired keyword search on a versioned evaluation set. Block release on any unauthorized result, and pause when stale-result or correction-load thresholds are crossed. The policy owner governs corpus validity, the technical owner governs retrieval and access enforcement, the support operations owner governs workflow and rollback, and the risk owner accepts residual exposure.

That paragraph is not the whole canvas. It is a compression test: the fuller record should make every sentence defensible.

Ownership must follow decisions. The product owner governs permitted use and outcome. Technical and data owners govern architecture, versions, provenance, access, and retention. The domain owner judges correctness and severity. Operations owns monitoring, incident response, and rollback. A risk owner accepts residual exposure, while the owner of the customer channel ensures correction, complaint, and appeal work. A name without authority, time, and resources is only decoration.

See Whether Four Readers Find the Same System

Give the contract independently to a product reviewer, a builder, a domain expert, and a risk reviewer. Ask each to describe the input, output, next action, prohibited event, pilot evidence, stop evidence, and decision owner. Materially different answers expose unresolved design, not differences in wording.

Then return to the three systems hidden in the opening slogan. Ticket routing can be evaluated against the existing rules and corrected by a coordinator; urgent-case recall and severe misroutes govern release. Evidence retrieval can ship separately if it respects permissions and beats the repaired search baseline. Response drafting begins only after approved evidence exists, reviewer capacity is measured, and unsupported claims and critical omissions have explicit limits. A useful result from one does not validate the others.

The team has not yet proved that retrieval should launch. It has achieved something earlier and necessary: everyone can now point to the same proposed system, test it against the same alternative, recognize its failures, and identify who may stop it. Risk Triage Before Building determines how much evidence and independent authority that contract requires before anyone builds momentum around a prototype.

Source Notes

• NIST AI RMF Core: Map, context, intended purpose, impacts, affected people, and risk identification within continuous AI risk management; verified 2026-07-10.
• Google Rules of Machine Learning, guidance on defining metrics, simple objectives, and comparable baseline behavior; verified 2026-07-10.