Agents, Tools, and Autonomy

When a Draft Becomes an Action

An operations team gives an assistant access to its ticketing system, cloud console, messaging platform, and internal documentation. Its assignment sounds modest: investigate service alerts and propose remediation. During a test, it finds a runbook, concludes that an old deployment is unhealthy, scales the service, closes the incident ticket, and posts an all-clear message.

The service was slow because of a database lock, not insufficient capacity. Scaling increased load. Closing the ticket suppressed escalation. The message created false confidence.

This was not merely a wrong answer. The system observed state, formed a plan, selected tools, changed infrastructure, altered the record of work, and communicated to other people. Each capability allowed an uncertain model output to travel farther.

A generative system produces content for someone or something else to interpret. An agentic system can choose and use tools across steps to change an environment. The boundary is not a product label. A model that emits one structured lookup for a person to approve may be part of an ordinary tool-assisted interaction; a system that chooses goals, decomposes work, retains state, and acts without immediate confirmation has acquired substantially more autonomy.

Autonomy is a property of permissions, action paths, and supervision—not of the model name. Design it one action at a time.

If a system only produces content, begin with the evidence and validation pipeline in Generative AI, LLMs, and Multimodal Models. The controls here begin where generated output can cause a side effect.

Follow the State, Not the Agent’s Story

An agentic system usually combines a model, a planning policy, tool interfaces, working state or memory, an environment that changes after actions, an evaluator, and termination logic. These names are useful only if they reveal who controls each transition.

At the first alert, the operations assistant has an observation: latency is high. It also has missing facts. It does not yet know whether the bottleneck is compute, storage, a downstream dependency, or a database lock. A free-form prompt such as “fix the incident” invites the planner to convert that uncertainty into action.

A controlled loop makes the uncertainty visible. The assistant may inspect approved metrics and current deployment state. It proposes a diagnosis and a next observation. Policy determines whether that observation is allowed. Only after evidence supports a change does the system prepare an action for approval or execution. It then reads the environment again to verify the effect and decides whether to stop, continue, or hand off.

For the incident, a sound first pass might be:

1. Read latency, saturation, error, and database-lock signals through narrow diagnostic tools.
2. Record the evidence for the leading explanation and the evidence that would disprove it.
3. Propose a reversible action, its expected effect, and its rollback.
4. Check whether policy permits this action in this environment and whether approval is required.
5. Execute once, then verify service health and ticket state through independent reads.
6. Stop or escalate when the evidence does not support another bounded step.

The system should record the goal, observed state, proposed action and arguments, permission used, expected effect, reversibility, approval, tool response, verification result, cost, and next-state decision. This is not a transcript of the model’s confidence. It is an account of what the system knew, was allowed to do, actually did, and observed afterward.

A tool returning success proves only what its contract says. A deployment command returning zero does not prove service health. A messaging API accepting a request does not prove that the right people received accurate information. Verification needs a separate observation wherever the consequences justify one.

Choose Autonomy Per Action

The same product can contain several levels of autonomy.

At the assistive level, the system drafts or recommends while a person decides and acts. At supervised action, it prepares a tool call and shows the exact target and expected effect for confirmation. With constrained automation, it may execute reversible actions inside a narrow policy, budget, and environment. Autonomous operation permits consequential multi-step work without per-action approval and therefore requires much stronger evidence, independent assurance, mature incident control, and hard limits.

Do not assign one level to the assistant as a whole. The incident assistant may search public documentation automatically, inspect production telemetry through read-only capabilities, require approval to restart a worker, and be unable to close a ticket or post an all-clear. These differences are the design.

Multi-agent designs do not escape this analysis. They add ownership and coordination questions: two agents may repeat an action, work from inconsistent state, delegate in a circle, or issue conflicting changes. Add another agent only when a distinct role or environment earns the complexity, and ensure that one coordinator can enforce action ownership, shared-state rules, budgets, and termination.

Turn Permissions into Small Capabilities

Least privilege is more than a restricted service account. A broad credential behind a broad tool still leaves the planner with a dangerous action space.

Replace a cloud shell with explicit operations such as read_service_health, restart_worker, or scale_service_within_limit. Separate create_incident_update_draft from publish_incident_update, and separate both from close_incident. Validate resource identifiers and environments outside the model. Use allowlists, rate and spend limits, dry-run support, idempotency keys, and rollback metadata.

For each capability, answer in prose before implementation:

• What bounded user outcome requires it, and which identity acts on whose behalf?
• Which resources, fields, recipients, regions, and environments can it reach?
• What state, policy, and approval must be present before invocation?
• What changes, who may be affected, and how reversible is the change?
• Which independent observation will verify the intended result?
• What limits apply to time, spend, steps, frequency, and concurrency?
• Which arguments, versions, approvals, results, and failures can be logged safely?
• On partial failure, does the system retry, compensate, roll back, or escalate?

This sequence forces the team to justify the action rather than fill a capability catalog. In the opening incident, no tool should both change infrastructure and declare the incident resolved. Those acts have different evidence, authority, and failure consequences.

Treat every tool result as untrusted input. A runbook, ticket, web page, file, or API response can be stale, misleading, or deliberately hostile. Content from the environment may inform the plan; it must not silently become higher-priority policy or grant itself new permissions.

Approval Must Carry the Decision

Human approval works only when the reviewer can understand the proposed action, has authority to reject it, and has enough time to exercise judgment.

For a production change, the preview should show the target service and environment, exact parameters, evidence for the diagnosis, expected effect, uncertainty, alternatives, blast radius, rollback, and the permission being invoked. “The agent wants to fix the service. Approve?” merely transfers liability to a button press.

Require explicit approval for irreversible actions, external communication, money movement, permission changes, deletion, production deployment, legal or contractual commitments, and high-impact decisions. Exceptional blast radius may justify two-person approval. Narrow, low-risk actions can use pre-approved policy, but a named owner must approve and test that policy.

Approval has two common failure modes. Bundling hides several consequential actions inside one decision. Excessive prompting trains reviewers to approve harmless and dangerous steps with equal inattention. The remedy is not more approval; it is a better action boundary.

Termination and Recovery Complete the Loop

“Continue until done” is not a stop condition. The goal may be ambiguous, the environment may never satisfy it, or each retry may worsen the incident.

Define what success looks like in observable state. Define safe failure and the conditions for handoff. Then impose hard bounds on elapsed time, steps, tokens, tool calls, spend, retries, and concurrent actions. Detect repeated or contradictory actions. Use idempotency keys where a retry might duplicate a side effect. Stop on policy violations, forbidden states, missing permission, unresolved conflict, or lack of progress. Preserve an operator kill switch and a user cancellation path.

Rollback is not a slogan either. Some actions reverse cleanly; others can only be compensated for. Scaling a service down may reverse a capacity change, but it cannot erase load already sent to the locked database. Correcting an all-clear message does not make every recipient forget it. Classify reversibility before action and set approval accordingly.

Persist only the state needed for the task. Long-lived memory introduces privacy, staleness, and instruction-contamination risks. Give retained state a schema, provenance, lifetime, and correction path. Immediately before any consequential action, re-read the facts that could have changed.

Evaluate the Path to the Outcome

Agent evaluation needs a simulator, sandbox, shadow mode, or controlled environment in which side effects can be inspected. A useful task suite includes ordinary work, ambiguous goals, missing permissions, stale state, malicious tool output, partial failure, timeouts, conflicting instructions, duplicate requests, and prohibited actions.

Outcome metrics alone are misleading. The system must complete permitted tasks while selecting the right tools and arguments, obeying policy and approval, avoiding unnecessary actions, terminating well, recovering from failures, and staying within cost and latency bounds. Severe violations—such as a cross-account read or an unauthorized production change—should block release rather than disappear inside an average score.

Safe refusal and escalation are successful outcomes when evidence or authority is missing. Conversely, an agent that reaches the requested end state by violating policy has failed.

The operations team should replay the opening incident before any live trial. Give the assistant high latency with low compute saturation, a visible database lock, an outdated runbook, a tool response containing hostile instructions, and a request to “handle everything.” The expected path is diagnostic observation, rejection of the stale instruction, no scale action, escalation to the database owner, and no ticket closure or public update. Then vary one fact at a time and observe whether the action path changes for the right reason.

Progress from simulation to shadow mode, allowlisted internal tasks, low-blast-radius actions, human confirmation, and limited canary exposure. Each step needs promotion evidence and stop rules. Logs must bind model, policy, tool, permission, approval, action, result, and operator versions so an incident can be reconstructed.

Design Exercise: Bound a Purchasing Assistant

A company wants an assistant to keep office supplies in stock. “Maintain inventory” leaves product choice, quantity, vendor, address, timing, and spending authority unresolved. A malicious catalog description could influence planning; a stale count could create a duplicate order; vendor substitution could violate procurement policy.

Begin with the smallest useful boundary. Let the assistant read an approved inventory view, compare stock with policy-defined reorder bands, and prepare a proposal from allowlisted catalogs. It may not add vendors, change bank details or delivery addresses, exceed category budgets, or place a high-value order. The reviewer sees items, quantities, prices, vendor, inventory evidence, budget impact, and destination. The order uses an idempotency key, and a separate read confirms the resulting record.

Now classify each proposed capability as assistive, supervised, constrained, or prohibited. Decide what happens when inventory is stale, a price spikes, an item is unavailable, approval expires, the vendor times out, or the catalog contains an instruction aimed at the agent. For any action you would automate, write its permission boundary, verification observation, termination condition, and recovery path. If those cannot be stated precisely, move the action down the ladder.

Make the Release Claim Narrow Enough to Defend

Before release, name the allowed goals, prohibited outcomes, affected parties, tool capabilities, permission owners, approval rules, budgets, termination conditions, verification reads, rollback limits, and kill authority. Record which tests cover malicious content, stale state, duplicate actions, tool failure, denied permission, recovery, and human workload. State which failures block launch regardless of the average result.

For the incident assistant, an honest first release might claim only that the system can gather approved diagnostic evidence and draft a remediation plan. It cannot change production, close the incident, or communicate resolution. Later evidence may justify a narrowly supervised restart capability. It does not justify granting every neighboring action.

Return to the original test. The assistant saw a familiar runbook and completed an apparently coherent sequence. What the sequence lacked was the system’s discipline: evidence before diagnosis, policy before action, verification after effect, and authority before closure. The safest useful agent is not the one that can do the most. It is the one whose action boundary is clear enough that every transition can be permitted, observed, stopped, and owned.

Continue with Why AI Fails to trace how broken assumptions propagate across the wider system.

Source Notes

• OWASP GenAI Security Project: LLM06:2025 Excessive Agency, causes, examples, and mitigations for excessive functionality, permissions, and autonomy; verified 2026-07-19.
• MITRE ATLAS, adversarial threat knowledge base for AI-enabled systems; verified 2026-07-19.
• NIST AI RMF: Generative Artificial Intelligence Profile, NIST AI 600-1, generative-AI risk-management actions relevant to agentic systems and human oversight; published 2024 and verified 2026-07-19.