Case Study 2: Customer Support Summarization and Drafting

The Evidence Was One Click Away

A support agent opens a long billing dispute near the end of a busy shift. The copilot fills the center of the screen with a concise summary and a reassuring reply; the original thread sits behind a link. After a quick scan, the agent sends.

The summary omitted an earlier refund promise. The reply says that a credit has been issued, although the account system shows no such action.

The customer now has a buried promise followed by an invented completion. The agent must reopen the case, reconstruct the history, apologize, and determine whether anyone actually authorized a refund.

Calling this a hallucination would place the defect inside the model and leave the workflow untouched. The system made fluent text easier to approve than evidence was to inspect. It also collapsed two different tasks—reconstructing the case and composing the response—into one persuasive paragraph.

The team’s original promise—reduce the work of reading and writing—does not survive the failure. The product must instead prepare a case summary and response draft that an agent can efficiently verify, revise, and approve, while never representing an unverified account action as complete.

Preserve What the Agent Does Before Writing

Without the copilot, an agent reads the conversation, checks customer identity and account state, finds the governing policy, decides what remains unresolved, takes any authorized action, and writes a reply. The process is slow and vulnerable to missed details, but its order contains an important safeguard: the agent investigates before speaking for the company.

The team measures that existing path before automating it. It records handling and review time, reopened conversations, post-send corrections, escalations, policy searches, and the kinds of facts agents commonly reconstruct. These observations expose the clerical burden the copilot might remove without assuming that faster replies are better replies.

The first release assists with long, low-consequence informational cases. Refunds, identity disputes, threats, legal requests, safety reports, and vulnerable-customer workflows remain on the ordinary path. Exclusion is part of the design, not a temporary footnote: each added case family will need its own failure evidence and controls.

Build an Evidence Packet Before a Draft

Summarization and drafting need different evidence. A useful summary preserves the customer’s request, material chronology, prior commitments, completed actions, disputed facts, and unresolved questions. A useful response follows current policy, uses an appropriate tone, limits sensitive disclosure, and describes only actions supported by system state. Combining the two into one score would hide the very omission that began this case.

The application therefore assembles a versioned evidence packet before it asks for prose. It contains the permitted conversation turns, current policy excerpts, verified customer context, and live results from account tools. The model produces structured summary fields with links back to the relevant turns. A second step drafts from that packet and the agent’s confirmed disposition.

Account actions do not come from conversational text. A tool result distinguishes eligible, authorized, submitted, completed, and failed; the interface renders that state beside any sentence about an action. If the refund service is unavailable, the draft may say that the case is being investigated, but it cannot say that a credit was issued. A missing or conflicting commitment is highlighted rather than smoothed into a coherent story.

The design rejects a cheaper alternative: send the whole thread and a policy document to one prompt, then ask the agent to inspect the answer. That approach saves an orchestration step while making provenance, action state, and omission failures harder to see. The case needs inspectable intermediate state more than it needs a single elegant completion.

The Pilot Exposes a Second Failure

During the pilot, the interface places the draft in the center and the evidence behind expandable links. On an ordinary morning, agents open sources and correct the occasional mistake. During a queue spike, source opening falls, approvals accelerate, and edits become superficial. One agent approves a response that overlooks an earlier exception and gives the customer the standard return deadline. The mandatory human click exists; meaningful review does not.

The incident changes the product. The summary and draft appear beside the evidence they depend on, not in front of it. Prior commitments and action-state claims receive explicit review controls. The send button remains unavailable until the agent resolves conflicts and either confirms or removes unsupported actions. Agents can reject the draft, escalate the case, or finish without the copilot, and those routes do not count against them as product failures.

Support operations also caps assisted volume per agent and preserves review time in staffing plans. If peak load makes the required review unrealistic, the system reduces assistance or returns cases to the unassisted workflow. A control that survives only when the queue is quiet is not a production control.

The human oversight plan now assigns concrete authority:

• The agent owns the final message and may edit, reject, escalate, or proceed without AI.
• Support operations owns staffing, training, case eligibility, and the fallback workflow.
• Quality reviewers can quarantine a prompt, model, policy version, or case family when severe errors recur.
• Product owns customer and agent outcomes; engineering owns evidence assembly, tool-state integrity, and release traceability.
• Privacy, security, and policy owners approve data handling and guidance changes within their domains.

“A human reviews every response” is therefore not the control. The control is a reviewable artifact, enough time, visible evidence, real authority, a workable fallback, and an organization willing to pause the feature.

Evaluate the Path to the Customer

The retained test set includes ordinary cases, long threads, conflicting promises, angry language, multilingual content, ambiguous account identity, sensitive data, policy exceptions, and tool outages. It also preserves the original billing dispute and variations in which the earlier promise is indirect, contradicted, or made under an expired policy.

Reviewers score the summary before seeing the response. They check whether it captured the request, chronology, commitments, completed actions, and open questions, and whether each material claim resolves to evidence. They then score the draft for factual support, action-state accuracy, policy compliance, privacy, tone, and correct escalation. This order reveals whether a polished response inherited a bad reconstruction.

A fabricated completed action, disclosure to the wrong customer, materially wrong entitlement, or unsafe instruction blocks release even when average helpfulness improves. The generative evaluation rubric keeps those severe errors separate from fixable defects such as awkward phrasing. It also records whether the agent could detect the defect in the actual interface; offline reviewers with unlimited time are not a substitute for workflow evidence.

The pilot compares assisted and unassisted queues for resolution, repeat contact, complaint, correction, escalation, handling time, and reviewer load. Edit distance and acceptance rate remain diagnostic signals, not success measures. Fewer edits could mean better drafts; it could also mean that agents have stopped scrutinizing them.

Sample for Failures the Average Conceals

Random sampling estimates ordinary quality but rarely catches the cases that justify the controls. The QA sampling plan combines a stable random sample with deliberate oversampling of long threads, policy exceptions, low-confidence summaries, agent overrides, sensitive-data flags, multilingual cases, tool failures, new model or prompt versions, and case families with recent complaints. Results are sliced by language, customer segment, case family, agent cohort, and complete system version.

Quality reviewers inspect both AI output and the final sent response. That distinction shows whether a defect reached the customer, whether the agent repaired it, and whether the interface helped. Repeated edits to the same claim become an evaluation case and a product defect; they are not collected merely as material for future model training.

Every release record binds the prompt, model, evidence-assembly logic, policy corpus, tool schemas, interface, eligibility rules, and evaluation report. When policy changes, the affected retained cases rerun before new guidance enters generation. When an account API fails, action claims disappear and the normal workflow remains available.

The operating view keeps unlike signals apart: unsupported summary claim, missed commitment, inappropriate disclosure, wrong action state, required escalation, agent override, post-send correction, complaint, latency, and review burden. Each has an owner and response. A blended quality score would allow abundant easy cases to conceal the small number that can break customer trust.

Expansion Is a New Decision

After the queue-pressure incident, the team does not abandon the copilot, but it refuses to treat the repaired interface as proof of general safety. The low-consequence pilot resumes only after the original failure family passes retained tests, peak-load review is rehearsed, and agents demonstrate that they can reject and bypass the system without penalty.

Expansion occurs by case family. Refund cases, for example, require reliable action-state integration, entitlement rules, reversal handling, a more severe error taxonomy, and evidence that agents can distinguish authorization from completion. Multilingual cases require their own review capacity and evaluation evidence; success in English does not grant them entry.

The design review records six commitments:

• Use case: reconstruct eligible support cases and draft responses for agent approval, not automate customer communication.
• Primary harm: a plausible message that misstates the case or company action; related harms include privacy leakage, inappropriate tone, and automation bias.
• Key control: evidence-linked summary and draft review, with verified action state and send authority outside the model.
• Acceptance evidence: separate summary and draft evaluations, severe-error gates, peak-load workflow trials, and risk-stratified QA.
• Pause trigger: any confirmed wrong-recipient disclosure, fabricated completed action, or repeated failure that the interface makes difficult to detect.
• Expansion rule: a new case family enters only with its own evidence, owners, fallback, and retained tests.

The strongest implementation reduces clerical reconstruction while preserving judgment. The weak one optimizes acceptance, hides the evidence behind fluent prose, and counts a hurried click as oversight. The billing dispute makes the difference concrete: a person in the loop can protect the customer only when the rest of the system makes careful judgment possible.

Source Notes and Next Step

See Human-Centered AI Design, Evaluating Generative AI, and Observability and Drift.

This case treated the model as a copilot whose output crosses a human review gate. The next case examines a predictive score that changes which transactions are allowed, reviewed, or blocked—and why the operating policy, rather than the score alone, determines the outcome.