The AI Appropriateness Question

When “Add AI” Is the Wrong Requirement

A utility company receives thousands of requests to update customer addresses. Leaders propose a language model because free-text messages arrive through several channels. A workflow review reveals that most customers already enter the new address into a structured form. The delay comes from a policy requiring manual approval even when identity and service-account checks pass.

Classifying messages or drafting updates would make the queue more elaborate without removing its cause. The useful first release changes the policy, improves the form, applies deterministic validation, and sends ambiguous cases to a person. Language processing may later help with the residue of genuinely unstructured requests. It has no honest role until the simpler path is working.

The first AI design decision is therefore a comparison, not a model choice: use AI only when uncertainty contributes value that a credible alternative cannot, and when the resulting errors can be detected, bounded, and owned.

Find the Work Before Choosing the Tool

Describe the work without using the words AI, model, automation, or agent. Who is trying to accomplish what? What starts the work, what information exists then, and which judgment or transformation is difficult? Follow the output into the next decision or action, including its effect on people other than the direct user.

“Summarize a call so an agent can confirm the record” is a task. “Use an LLM to transform service operations” is not. “Estimate next week’s demand for each warehouse” is a task. “Become predictive” is not. If the team cannot name the work and its consequence, it is still negotiating the problem. A prototype will only conceal that disagreement behind impressive output.

The utility team can now see three different jobs. Validating a structured address is ordinary software. Deciding whether a verified request may bypass manual approval is a policy decision. Interpreting an unusual free-text request is a language task with uncertain output. Only the third job supplies even a candidate reason for AI.

Draw the Smallest Uncertainty Boundary

Begin with the strongest simpler option the team can reasonably implement, not with “do nothing badly.” Redesign the workflow when delay comes from handoffs, incentives, policy, or unclear ownership. Use deterministic rules when requirements and exceptions can be stated economically. Use search when people need inspectable facts, and analytics when they need aggregates or trends rather than an instance-level judgment. Keep qualified human service where novelty, empathy, rights, or moral responsibility dominate.

These choices combine. Rules can enforce hard constraints while a model ranks permitted options. Search can retrieve evidence while generation drafts a synthesis. A person can make the consequential decision while a model reduces clerical work. The design question is not whether the product contains AI; it is where learned, probabilistic behavior earns a boundary inside the product.

For the utility, the non-AI path handles identity checks, account matching, address validation, and routine approval. A language component, if it proves useful, receives only messages the structured route cannot resolve. It proposes an interpretation; it does not update the account. That narrower boundary reduces the data exposed to the model, preserves a deterministic path for most requests, and makes errors visible before they become customer-record changes.

Four questions decide whether such a boundary deserves an experiment:

1. Is the task clear? Name the user, current workflow, input, output, action path, affected people, and desired outcome. If these remain disputed, investigate or redesign the work.
2. Is uncertainty useful? Identify the pattern, language, perception, personalization, ranking, or forecasting behavior that stable rules cannot provide economically. If there is none, prefer ordinary software.
3. Can errors be found before or after consequence? Specify evidence, review, fallback, correction, appeal, rollback, or containment in proportion to the harm. A human-review box is not a control if the reviewer lacks time, evidence, or authority.
4. Can the system be governed for as long as it operates? Establish permitted data, measurable advantage, accountable owners, monitoring, incident response, change control, and retirement. If any indispensable condition is missing, stop or repair it before a prototype creates momentum.

Passing these questions permits a comparison, not a launch. The AI option must still beat the baseline on representative cases and predefined acceptance thresholds.

Some Answers End the Inquiry

Not every weakness should be converted into another backlog item. Stop when law, contract, policy, or organizational values prohibit the use; when necessary data cannot be used safely or with appropriate rights; or when the system depends on an unjustified sensitive proxy. Stop when a serious error would be invisible, irreversible, or impossible to contest. Stop when nobody has authority to accept residual risk or pause operation.

Other answers mean “not yet.” The team may need workflow observations, better labels, a rights review, a credible evaluation set, an appeal path, or a measured baseline. These are discoveries, not failed AI projects. They prevent model development from becoming the most expensive way to learn that the decision was never ready.

The Appropriateness Scorecard

Use the scorecard to expose evidence and disagreement. Score each dimension from 0 to 3: 0 means blocked, 1 rests mostly on hope or serious unknowns, 2 has usable evidence with named gaps, and 3 has strong evidence under the proposed conditions.

Do not average away a blocker. A zero for permissibility, data rights, or an intolerable error remains a stop even if every other row scores three. Record the evidence behind each score, the strongest disagreement, and the person authorized to make the decision.

For the utility, the broad “AI address-change” proposal fails because it has no advantage over workflow repair and would place uncertain interpretation too close to a consequential update. The residual free-text assistant might justify a bounded pilot, but only against the repaired workflow, with permitted messages, an evaluation set of ambiguous requests, human confirmation, account-update controls outside the model, and an owner who can withdraw it.

Decide Five Cases Before Reading On

Apply the four questions and the scorecard to these proposals. For each, choose reject, redesign, investigate, pilot, or proceed to use-case definition. Name the strongest non-AI baseline and the evidence that could change your decision.

1. An internal meeting-note assistant drafts a summary for participants to correct.
2. A system issues clinical diagnoses without clinician review.
3. A warehouse uses forecasts to propose reversible inventory adjustments.
4. An employer produces a single employee-performance score for promotion decisions.
5. A document assistant searches an access-controlled policy corpus and generates an answer.

Do not classify them by “low stakes” and “high stakes.” Reduce each proposal to its actual action boundary.

The meeting assistant may support a limited pilot if participants know it is used, sensitive content is controlled, and evaluation covers omissions and attribution. Autonomous diagnosis should be rejected at the proposed boundary; a clinician-facing evidence or documentation aid is a different use case. The warehouse forecast is a credible experiment when it beats seasonal and rules-based baselines and planners can inspect and reverse its effects.

The employee score should stop pending impact and governance review: historical labels may encode unequal opportunity, the target itself is contested, and an appeal may not repair the consequence. The document assistant should begin with permission-aware retrieval and corpus quality. Generated synthesis earns a later stage only if it improves task outcomes without weakening authorization or claim support.

The decisions differ because uncertainty, evidence, action, and consequence differ. A proposal often becomes more defensible by shrinking the action boundary rather than by choosing a better model.

Make Complexity Earn Its Keep

Before funding a prototype, ask someone to defend the strongest non-AI option on the same cases and measures. Compare task and user outcomes, important error costs, latency, accessibility, lifecycle expense, security and privacy exposure, correction and appeal, resilience to change, and the operating skill required. A workflow repair, form, keyword rule, search system, heuristic, or small classifier may win outright or remain the fallback.

Google’s Rules of Machine Learning recommends establishing metrics and beginning with a simple approach. The durable engineering principle is broader: complexity earns its maintenance burden through evidence, not novelty.

Capture the decision in one page: the work and affected people; the proposed output and action boundary; the alternatives and strongest baseline; the reason uncertainty may help; data rights and gaps; consequential errors and recourse; evaluation cases and thresholds; lifecycle costs; accountable owners; and the resulting decision. Approval expires when the task, data, population, action, provider, policy, or consequence changes materially.

The utility team can now say something more useful than “yes to AI” or “no to AI.” It says yes to a repaired deterministic workflow, no to model-controlled account updates, and perhaps to a measured language-assistance experiment for the remaining ambiguous requests. From Vague Idea to Testable Use Case turns that bounded possibility into a contract the team can evaluate.

Source Notes

• Google Rules of Machine Learning, guidance to establish metrics, compare simple heuristics, and keep the first model simple; verified 2026-07-19.
• NIST AI RMF Core, voluntary Govern, Map, Measure, and Manage outcomes for contextual, continuous AI risk management; verified 2026-07-19.