Evaluating Agents and Tool-Using Systems

The Trip Was Booked, but the Run Failed

A travel agent receives a request to reserve a refundable morning flight under a fixed budget. Its final message contains a suitable itinerary and a confirmation number. The trace tells a different story: the agent first reserved a nonrefundable ticket, tried to cancel it, retried the payment call after a timeout, and created two pending charges before a human intervened.

The destination is right. The run is not.

An agent earns authority only when it reaches an allowed result by an acceptable path. Evaluation must make actions, permissions, state changes, resource use, side effects, termination, and recovery visible—not merely grade the final answer.

Turn the Request into an Action Contract

“Book me a flight” describes an outcome but leaves the agent’s authority undefined. The evaluator needs a contract that can distinguish a safe alternative from an unauthorized improvisation. For this trip, the contract requires a refundable flight departing before noon, caps the total price, permits search without approval, requires confirmation before purchase, forbids split tickets, and stops the run after two unresolved payment failures. If no itinerary satisfies those terms, a clean handoff is an acceptable result.

The contract also says what evidence must survive: the itinerary shown for approval, its price and refund terms at that moment, the user’s response, the exact purchase request, the tool result, and the resulting reservation and payment state. That evidence turns broad intentions into predicates an evaluator can check.

Three kinds of condition deserve different treatment. Required outcomes describe what must be true when the run ends. Forbidden transitions describe events that must never occur, such as purchasing before approval or reading another traveler’s record. Bounded resources describe how much time, money, and retrying the agent may consume. A high completion score cannot compensate for crossing a forbidden boundary.

Give the Agent a World That Can Change

A prompt and reference answer cannot reveal whether the agent called the wrong API, acted on stale state, or left a charge behind. The travel task therefore begins with an inspectable world: flight inventory, fare rules, a payment account, an empty reservation ledger, tool permissions, and a clock. Hidden evaluator facts include a price change after approval and a payment endpoint that commits a write but times out before acknowledging it.

The harness records each observation, call, result, approval, state transition, and budget change. Its tools should reproduce the awkward semantics of production: pagination, authentication expiry, ambiguous identifiers, eventual consistency, partial success, and concurrent changes. Simulation, test tenants, read-only mirrors, and reversible dry runs are safer starting points than live systems, but a frictionless simulator gives false confidence. If duplicate requests are possible in production, they must be possible in the test world.

Reset matters as much as setup. Every run should start from a known fixture and finish with an inspection of reservations, charges, temporary credentials, drafts, and locks. Otherwise one test can inherit another’s residue and make both the agent and the grader look better or worse than they are.

Follow the Run Through State

At the first decision, the agent finds a refundable flight for $420 and asks the user to approve it. While the user is responding, the fare rises to $465. An approval is not a reusable token for “some flight to this city.” It binds the proposed action, target, price, and material consequences. The changed terms invalidate it, so the correct next action is to present the new price or find another option.

This makes human oversight testable. The agent must ask at the right boundary, expose the facts needed for a decision, wait for an unambiguous response, and apply that response only to the action shown. Silence is not consent. Approval copied from retrieved text is not consent. An earlier approval cannot authorize a changed recipient, amount, scope, or payload. For high-consequence actions, the environment should enforce this binding rather than rely on the model to remember it.

Suppose the user approves the $465 fare and the purchase call times out. The agent now knows that it sent a request; it does not know whether the airline accepted it. Blindly retrying converts uncertainty into a possible duplicate charge. The safe branch is to inspect reservation and payment state using the request’s idempotency key. If the first write succeeded, continue from that state. If its outcome remains unknowable, stop or escalate with the exact residue instead of guessing.

The original failed run took the dangerous branch. It treated a missing response as a failed action, purchased again, and then tried to repair the duplicate by cancelling a ticket whose refund terms it had not verified. The recovery increased the harm. A useful evaluation catches the first invalid transition, then records every consequence that followed.

Score Without Inventing One Ideal Plan

Several trajectories could satisfy the travel contract. One agent may search by schedule and then filter by refundability; another may apply both constraints in the first call. Evaluation should not punish harmless differences in planning. It should ask whether the required final state was reached, whether tools and arguments were correct, whether authority and approval held, whether remembered facts remained current, whether budgets held, whether failures were contained, and whether the agent stopped with a reconstructable state.

Keep those judgments separate. A run may find no valid flight yet deserve a strong recovery judgment because it preserves funds and hands off clearly. Another may book the requested itinerary while failing the authority gate. Severe failures—cross-tenant access, an unapproved purchase, an unreconciled duplicate charge—block release rather than disappearing into a weighted average.

Repeated runs are necessary because the policy may be stable while the model’s choices vary. Record the model, prompt, tool versions, fixture, randomization policy, and grader version with every result. Report both common behavior and severe tails. “Passed once” is weak evidence for a system that will act thousands of times.

Make Memory and Tool Output Earn Trust

Longer runs introduce another kind of drift. A constraint visible at step two may vanish from a summary at step twenty. A traveler with the same surname may be mistaken for the requester. A cached fare may be treated as current, or content returned by a tool may instruct the agent to ignore its governing rules.

Seed those conditions deliberately. Every consequential state variable should retain its source and freshness. User claims, model inferences, and tool-confirmed facts should remain distinguishable. The agent should re-read current state before irreversible action, keep sensitive memory within the right user and task, validate tool output before using it, and finish without orphaned reservations or temporary authority.

Tool tests belong at the boundary where calls become effects. Exercise malformed arguments, wrong identifiers, read-versus-write confusion, permission denial, stale versions, conflicting updates, partial success, and malicious returned content. When a tool is unavailable, the acceptable response depends on the contract: back off within the retry budget, choose an allowed alternative, or hand off. Seeking a privilege bypass is not recovery.

Release Only the Authority the Evidence Supports

The first useful harness release does not need every imaginable journey. It needs representative task families, explicit risk tiers, credible state fixtures, versioned tool behavior, and failures chosen from the real consequence paths. Its record should preserve:

1. the action contract, initial state, and hidden evaluator facts;
2. tool schemas, permissions, versions, and simulation limits;
3. required outcomes, forbidden transitions, approval rules, and budgets;
4. the complete event and side-effect ledger;
5. injected failures and repeated-run policy;
6. grader logic, human adjudication, and severe-failure rules;
7. artifact retention, redaction, replay, and cleanup behavior;
8. the release decision, production canary, kill switch, and recovery owner.

Structured events make a run replayable; prose summaries explain the judgment. Keep identifiers needed to reconcile actions, while redacting secrets and unnecessary personal data.

Release scope should follow demonstrated containment. An agent may earn permission to search live inventory while purchases remain simulated. It may draft an itinerary but require a person to execute payment. A safe partial result can justify more authority than an apparently complete run whose side effects are hidden. Production canaries should preserve the same budgets, approval gates, monitoring, and immediate stop mechanism proven in the harness.

Return to the travel fixture and vary it twice. First, make the approved fare rise before purchase; write the exact event that invalidates approval. Then make the payment call commit and time out; define the inspection that distinguishes a completed purchase from an unknown one, and the state in which the agent must hand off.

Now transfer the method to an IT administrator asked to remove a departing contractor’s access at a specified time. Define which systems may be read, which accounts may be disabled, which destructive actions are forbidden, how a namesake is excluded, and what residue is acceptable when one identity provider is unavailable. If the only success condition is “access removed,” the test is not yet safe enough to run.

Agent evaluation is complete when the team can reconstruct what the system believed, what authority it held, what it changed, what remains uncertain, and why that evidence justifies the next increment of production authority.

Source Notes

• NIST AI RMF Core: Measure, voluntary guidance on deployment-relevant, repeatable, documented evaluation, safe failure, human oversight, and independent assessment; verified 2026-07-20.
• NIST AI RMF Generative AI Profile, guidance relevant to human oversight, information security, incident disclosure, testing, and generative-system risk management; verified 2026-07-20.
• See Agents, Tools, and Autonomy for authority design and Security Architecture for AI Applications for least privilege and sandboxing.