AI Product Requirements

Five Words Conceal the Product

A team proposes an internal assistant that will “answer policy questions accurately.” Before reading further, try to turn that sentence into a test. Which policies count? Which employees may see which documents? Is an answer acceptable when its conclusion is right but its source is obsolete? What should happen when two current policies disagree?

The sentence cannot answer those questions because it names a hope, not a product. Engineering can still build from it, but each participant will build a different imagined system. A product manager may picture convenient search. A security reviewer may picture permission-aware retrieval. An employee may assume that a confident answer is an authoritative decision.

An AI product requirement describes observable system behavior in the conditions where the product will succeed, struggle, refuse, and fail. It connects a user need to evidence, limits, controls, and an operating response.

Begin at the System Boundary

Suppose the user need is narrower: employees need a sourced explanation of the current travel policy without searching several repositories. The AI role is to retrieve authorized passages and draft a concise explanation. It may not approve expenses, interpret individual legal rights, or modify a record.

That division between need and role is the beginning of the requirement. It prevents a useful explanation tool from quietly becoming a decision maker. It also gives the team something observable: the system receives a question and an authenticated user context; it returns an explanation with evidence, abstains, or routes the question elsewhere.

A conventional feature can often be specified as an input and a deterministic result. Here, behavior also depends on the policy corpus, permissions, retrieval, prompts, model version, validation, interface, and human workflow. The requirement must therefore bind the whole interaction, not assign a quality adjective to “the model.”

The following form is deliberately longer than a user story because it joins behavior to the evidence and response that make the behavior governable:

For [user and context], when [input condition],
the AI-enabled system shall [observable behavior]
within [quality, latency, cost, accessibility, and policy limits].
It shall not [unacceptable behavior].
If [uncertainty, missing evidence, dependency failure, or risk trigger],
it shall [abstain, degrade, ask, escalate, or fall back].
We will accept this when [evaluation, threshold, sample, and severity rule].
[Owner] monitors [signal] and responds with [playbook].

Without the last line, “the model” easily becomes the unnamed owner. Without the fallback line, an offline score can masquerade as a product requirement.

Make “Accurate” Answerable

Accuracy is not one property of the policy assistant. Retrieval can find the wrong document. A draft can contradict a retrieved passage. A correct claim can carry a mismatched citation. An answer can omit an exception that changes the employee’s decision. Combining these failures into one quality score makes the score easy to improve and hard to trust.

For this product, require material claims to be supported by displayed passages from the current, authorized policy collection. Measure retrieval coverage, claim support, citation precision, and consequential omission separately. Evaluate them end to end as well as by component: a good retriever cannot rescue an explanation that reverses the policy, and a fluent explanation cannot rescue a permission failure.

The sample must resist the happy path. Include roles, regions, policy categories, stale documents, conflicting documents, unsupported questions, adversarial instructions, and the languages and assistive workflows the product claims to support. Thresholds should follow error cost, baseline performance, user research, and review capacity—not whichever number a benchmark makes convenient.

Some failures should not be averaged at all. A single confirmed disclosure across an authorization boundary can block release even when every aggregate metric is excellent. Severe cases need their own decision rule.

Give Uncertainty Somewhere to Go

The product will encounter questions its sources cannot settle. A requirement is incomplete if it only says that the model should be “confident enough.” Confidence may be poorly calibrated, and it does not create a current policy where none exists.

When sources are absent, stale, or contradictory, the assistant should state that it cannot provide a reliable answer, show the evidence of the conflict when the user is authorized to see it, and route the case to travel operations. The route needs an owner, service level, and capacity. Otherwise abstention merely moves failure from the model into an invisible queue.

This is also where human control becomes concrete. Employees can inspect sources and reject the explanation; travel-operations staff can resolve an ambiguous policy question; neither group is asked to supervise a volume it cannot absorb. Chapter 18 develops the interaction design. The requirement here establishes the authority and operating promise that the interface must honor.

Write the Forbidden Outcome with Its Response

“The assistant must not give legal advice” sounds firm, but it offers no test and no containment. A decision-grade version says what behavior belongs outside the product and what happens when the boundary is approached:

The employee policy assistant must not interpret individual legal rights or recommend a legal course of action.
Evaluation: adversarial and ordinary prompts across approved languages.
Expected behavior: identify the limit, provide the relevant policy source when authorized, and route the user to the named human channel.
Production response: policy-violation alerts enter weekly review; a severe misdirection triggers pause of the affected intent.
Owner: employee-services product owner with legal-policy reviewer.

This does not promise perfect prevention. It creates ordinary and adversarial tests, a safe response for the user, a review path, and authority to pause the affected intent. Use the same structure for fabricated policy, unauthorized action, discriminatory recommendation, dangerous instruction, inaccessible behavior, or output presented without required evidence.

Privacy, security, and safety become useful requirements in this form. Name permitted data classes, enforce authorization during retrieval, exclude sensitive fields from logs, restrict tool permissions, preserve incident evidence, and define notification triggers. “Comply with policy” cannot substitute for behavior that engineers can implement and reviewers can observe.

Budget the Whole Interaction

If the answer must begin within five seconds, do not hand all five seconds to the model. Allocate time to authentication, retrieval, reranking, generation, validation, streaming, and observability. Define dependency timeouts, retry limits, throughput, and the degraded response. A fast invented answer is not graceful degradation.

Cost deserves the same system boundary. Budget evaluation, human review, incident handling, and vendor change—not only inference. Shorter context may save money while omitting an exception. Validation may add latency while preventing a malformed answer. A smaller model may be enough for routing while a larger one performs a narrow drafting step. The requirement should expose those trades rather than hiding them under “non-functional.”

Keep the Contract Alive

At launch review, the team now has a behavior contract rather than a feature description. For supported travel questions, the system displays the policy version, source passage, explanation, and link. It refuses unsupported interpretation and routes unresolved questions. Authorization is enforced during retrieval. Free-form user text is data, never trusted instruction for tools or policy. Launch depends on agreed quality thresholds, zero-tolerance security cases, response budgets, accessibility task success, and a travel-operations queue that can carry the expected load.

Release does not retire these requirements. The product owner monitors source freshness, consequential omissions, high-severity complaints, abstention reasons, escalation backlog, permission-test regressions, latency, and cost. Each signal has an alert or review cadence and a playbook. A failed access-control test or confirmed cross-boundary disclosure pauses the service.

The contract also has an expiry condition. A material change to the model, prompt, policy corpus, permissions, supported population, threshold, or fallback workflow reopens the relevant evidence. A requirement that cannot be observed after launch cannot protect a changing product.

For each material behavior, the requirements record should preserve the user and affected parties; supported and unsupported contexts; expected output and evidence; prohibited outcomes and severity; acceptance sample and thresholds; uncertainty response; human authority and capacity; privacy, security, safety, accessibility, latency, and cost limits; production signals and playbooks; owner and approvers; and the changes that force review.

Return to “answer policy questions accurately.” It has become a bounded promise that design can express, engineering can implement, evaluation can challenge, operations can observe, and an accountable owner can suspend. That is the work of an AI product requirement.

Source Notes

• NIST AI RMF Core, outcomes for mapping context, measuring risk, documenting limits, assigning roles, and managing residual risk across the system lifecycle; verified 2026-07-10.
• Microsoft Guidelines for Human-AI Interaction, evidence-based interaction guidance organized around initial use, interaction, failure, and behavior over time; verified 2026-07-10.
• People + AI Guidebook: Mental Models, guidance on setting expectations for changing AI behavior and explaining the effect of feedback; verified 2026-07-10.