Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 18

Human-Centered AI Design

Design AI interactions that support calibrated reliance, informed control, useful recovery, contestability, and inclusive access.

The Missing Date

A support agent opens a cancellation case. Above the customer’s messages, an AI-generated summary says that the customer cancelled after the refund window closed. The prose is polished, the type is crisp, and a large Accept and reply button sits beside it. The source messages are collapsed.

The summary has omitted one sentence: the customer first asked to cancel three days before the deadline. The agent can uncover it, but only by distrusting an interface designed to make acceptance feel routine.

Adding “AI can make mistakes” below the button would shift responsibility without changing the decision. The useful question is not whether the interface disclosed AI somewhere. It is whether the interaction helps this agent notice the missing date before the system turns an uncertain summary into a consequential reply.

Human-centered AI gives people the evidence, friction, control, and recovery that the decision requires. Its aim is calibrated reliance: effort should rise when evidence is weak or consequences are high, rather than when the interface happens to look uncertain.

Reliance Develops Through the Interaction

Before the agent reads a word, the product has already made claims through its name, placement, defaults, and visual hierarchy. The agent forms an expectation, inspects—or skips—the evidence, chooses an action, observes an outcome, and learns whether to trust similar outputs next time.

A five-step reliance loop moves through Expectation, Evidence, Choice, Outcome, and Learning; each step has a design cue: scope, source and uncertainty, confirm or override, recovery and appeal, and feedback effect.
Reliance develops through use: expectations shape how evidence is read, choices produce outcomes, and those outcomes teach the user what to trust next time.

This loop makes small interface decisions cumulative. A confident label encourages a quick scan. Collapsed evidence makes that scan less informative. A dominant acceptance button makes agreement cheaper than correction. If the resulting error is hard to undo, the product has converted presentation choices into operational risk.

Good design changes the loop at the point where judgment can still improve.

Promise Only the Capability the Product Can Defend

The first version calls the feature Case Intelligence and says it “understands the customer’s history.” Neither phrase tells the agent what was processed or what remains their responsibility.

A defensible introduction is narrower:

Drafts a case summary from the messages and attachments shown. It may miss dates or context. Review the linked evidence before sending a response.

This disclosure names the benefit, source boundary, important limitation, and expected human action. If attachments in some formats are excluded, say so before the agent assumes they were read. If supported languages, account history, permissions, or regional policies change the result, reveal the boundary where it affects the task.

Disclosure should answer practical questions, not merely announce a technology: Why is this output here? Which data shaped it? What will happen if I continue? Is there a non-AI route? Who remains responsible for the action? A sparkle icon answers none of them.

Expectation setting continues after first use. A material change to sources, behavior, or automation can invalidate habits learned from the previous version. Communicate the change where the agent encounters it, and require renewed confirmation when the action boundary has expanded.

Put Evidence Beside the Claim

The redesigned console divides the summary into claims. Selecting “Cancellation requested after deadline” opens the supporting message span with its author and timestamp. A contradictory earlier request appears beside it. The agent can inspect the surrounding conversation rather than a sentence chosen only because it supports the draft.

This is more useful than displaying “Confidence: 62%.” A model score may be miscalibrated, may describe a component rather than the whole answer, and rarely tells the user what action follows. Decision-shaped uncertainty does:

  • “Two attachments could not be read; inspect them before replying.”
  • “The messages contain two different cancellation dates; resolve the conflict.”
  • “This request is outside the policy sources available to the assistant; route it to billing operations.”

Different tasks need different evidence. Extraction can highlight missing fields and link values to their source locations. A forecast can show a range and the assumptions that move it. A recommendation can expose alternatives and trade-offs. Evidence-bound generation can show passages, dates, and contradictions. An unfamiliar input can be marked outside evaluated coverage.

The design test is simple: after seeing the cue, does the user know what changed and what to do next? Uncertainty that merely looks scientific can increase deference without improving judgment.

Make the Safe Choice Genuinely Available

The agent can edit a claim, reject the summary, request missing information, open the full source, or route the case to a specialist. These controls appear beside the claim they affect. The product preserves the agent’s draft when they leave the AI path.

Control must change the outcome, not decorate it. A confirmation dialog is weak when approval is preselected, the alternative is hidden, or repeated prompts wear down refusal. A human route is fictional when no queue, service level, or authority exists behind it. Personalization becomes manipulation when it learns which wording makes a person comply while hiding or weakening their ability to opt out.

Friction should follow consequence. A routine wording change may remain editable and reversible. A reply involving cancellation, refund, safety, or account closure should require the agent to inspect unresolved claims before sending. An irreversible external action may require a second approval. The interface need not slow every case equally; it must not make the riskiest shortcut the easiest path.

The same reasoning applies after the action. Show what was sent, what record changed, and what remains pending. Provide undo or correction within the harm window. For a tool-using assistant, “request failed” is inadequate if a note was saved but a customer message was not sent. Name the completed and incomplete side effects.

Recovery Is Part of the Main Path

Suppose the agent accepts the flawed summary and the customer challenges the denial. Recovery begins with the customer’s ability to learn that automated assistance influenced the reply and to reach a correction channel without reconstructing the system’s internal vocabulary.

The agent needs the original messages, generated claim, evidence shown at review time, action history, and current policy. They must be able to correct the record and the reply. A severe privacy or safety failure needs a distinct incident path; it should not disappear into a thumbs-down counter.

Preserve work when dependencies fail. Explain whether retry is safe. Keep enough evidence for support and incident review without placing unnecessary sensitive data in logs. Avoid error messages that blame a person for input the product claimed to support.

Once recovery is treated as ordinary product behavior, teams can test it. Seed a contradictory date, an unreadable attachment, a stale policy, and a partially completed tool action. Ask whether a user can identify the state, choose a safe next step, and restore the intended outcome.

Feedback, Correction, and Appeal Are Different Services

After the corrected reply, the agent marks the omitted date. That report is evidence about this case; it should not silently become a universal training label. The customer appeal is different again: it asks an accountable person to reconsider an outcome.

Keep the channels distinct:

  • a correction changes a claim or record;
  • a preference changes presentation for a user, such as draft length;
  • an incident report triggers containment and investigation;
  • an appeal requests reconsideration of an affected outcome;
  • product research helps the team understand whether the workflow is useful.

For each channel, state what information is collected, who can see it, what response follows, and whether it affects this case, future product behavior, or neither. A preference is not truth. A quick acceptance is not proof of correctness. An appeal cannot be reduced to model feedback.

Contestability is therefore an end-to-end service. An affected person must be able to discover the AI’s material role, access relevant reasons and evidence, challenge an error, add information, receive human reconsideration, and obtain a recorded result. Identity checks, privacy controls, case routing, response times, reviewer authority, and quality review make that promise real. Chapter 19 develops the oversight system that must carry this work under actual volume and pressure.

Inclusion Changes the Behavior Contract

The console can meet ordinary accessibility checks and still fail the people who need it. Keyboard access, semantic structure, focus order, contrast, zoom, captions, motion controls, and screen-reader output are necessary. The AI behavior must also survive varied language, dialect, assistive input, device, and connection conditions.

In the cancellation case, a speech-recognition error might alter a date; optical character recognition might miss a handwritten note; translation might weaken the distinction between a request and a confirmation. The interface should expose that uncertainty without requiring precise typing, color perception, audio, or visual evidence alone. Time limits must accommodate cognitive and motor needs. Plain-language explanations and alternate source formats must remain available.

Evaluate task success and recovery across the populations, languages, devices, and assistive workflows the product claims to support. Invite affected people into research while requirements and architecture can still change. Participation that cannot alter a release decision is observation, not shared design.

A Review Record for the Reliance Loop

Before release, walk one consequential task from first expectation through a wrong outcome and recovery. Record:

  1. the capability promised and the boundary the user sees;
  2. the claims, sources, omissions, and uncertainty cues presented;
  3. the available non-AI path, override, confirmation, undo, and appeal;
  4. the side effects produced by acceptance, rejection, failure, and retry;
  5. the destinations, owners, response times, and authority behind escalation;
  6. the way corrections, preferences, incidents, appeals, and research data are separated;
  7. task success and recovery across supported languages and access needs;
  8. the measures used after launch: consequential omission, verification time, correction, override, workload, appeal, reversal, abandonment, and affected-person outcomes.

Now return to the original console. The design failure was not an unattractive screen or a missing disclaimer. It was a sequence that made weak evidence hard to inspect, agreement easy, correction costly, and recourse obscure. A human-centered redesign reverses those incentives. The interface cannot remove uncertainty, but it can make responsible judgment possible before trust hardens into habit.

Source Notes