An agent with a delete tool and a loose prompt can read “clean up my old entries” as permission. Nothing malfunctions. The message allows that reading and the tool allows the action. What's missing is a rule about when the tool may run.
This post continues my series on building agents. The previous one covered MCP.
What autonomy is
An agent's autonomy is how much it may do on its own before a human has to approve, review or step in. It is a policy your code enforces, not a trait of the model.
A scale from Level 0 to Level 4
No standard defines these levels. This is my own working scale.
| Level | Meaning |
|---|---|
| 0 | No tools. A plain chatbot |
| 1 | Read-only tools |
| 2 | Writes, each approved by a human |
| 3 | Routine writes run alone inside limits, irreversible actions still ask |
| 4 | Full autonomy, rarely justified |
A level belongs to a tool, not to the whole agent. The same agent can run reads at Level 4 and deletes at Level 2.
Sort tools by what a mistake costs
A read tool changes nothing. A write tool changes something you can undo. An irreversible tool can't be taken back: deleting data, sending a message, moving money. The class comes from the cost of a wrong call, not the tool's name.
flowchart TD
A[Model requests a tool] --> B{Tool risk class}
B -->|read| C[Run it]
B -->|write| D{Autonomy level}
B -->|irreversible| E{Autonomy level}
D -->|1| F[Refuse]
D -->|2| G[Ask a human]
D -->|3 or 4| C
E -->|1| F
E -->|2 or 3| G
E -->|4| C
G -->|approved| C
G -->|declined| I[Send an error result to the model]
F --> I
C --> H[Send the result to the model]The policy in code
Run it with AUTONOMY=2 npx tsx agent/autonomy.ts and change the value to compare levels.
// agent/autonomy.ts
import Anthropic from '@anthropic-ai/sdk';
import { createInterface } from 'node:readline/promises';
const client = new Anthropic();
const rl = createInterface({ input: process.stdin, output: process.stdout });
type Risk = 'read' | 'write' | 'irreversible';
type Decision = 'allow' | 'ask' | 'deny';
const LEVEL = Number(process.env.AUTONOMY ?? 2);
const expenses = [{ id: 1, item: 'tea', amount: 40 }];
let nextId = 2;
const risks: Record<string, Risk> = {
get_total: 'read',
add_expense: 'write',
delete_expense: 'irreversible',
};
function decide(level: number, risk: Risk): Decision {
if (risk === 'read') return 'allow';
if (level <= 1) return 'deny';
if (level >= 4) return 'allow';
if (risk === 'write' && level === 3) return 'allow';
return 'ask';
}
const tools: Anthropic.Tool[] = [
{
name: 'get_total',
description: 'Return the exact sum of every expense recorded so far. Use this whenever the user asks for a total.',
input_schema: { type: 'object', properties: {} },
},
{
name: 'add_expense',
description: 'Record one expense. Use it whenever the user says they spent money.',
input_schema: {
type: 'object',
properties: { item: { type: 'string' }, amount: { type: 'number' } },
required: ['item', 'amount'],
},
},
{
name: 'delete_expense',
description: 'Permanently delete one expense by its id. Use it only when the user names the expense to remove.',
input_schema: {
type: 'object',
properties: { id: { type: 'number' } },
required: ['id'],
},
},
];
function execute(name: string, input: Record<string, unknown>): unknown {
if (name === 'get_total') return { total: expenses.reduce((sum, e) => sum + e.amount, 0) };
if (name === 'add_expense') {
const expense = { id: nextId++, item: String(input.item), amount: Number(input.amount) };
expenses.push(expense);
return expense;
}
if (name === 'delete_expense') {
const index = expenses.findIndex((e) => e.id === Number(input.id));
if (index === -1) return { error: 'no such expense' };
expenses.splice(index, 1);
return { ok: true };
}
return { error: 'unknown tool' };
}
async function approve(name: string, input: Record<string, unknown>): Promise<boolean> {
const answer = await rl.question(`Allow ${name} ${JSON.stringify(input)}? (y/n) `);
return answer.trim().toLowerCase() === 'y';
}
async function guarded(name: string, input: Record<string, unknown>): Promise<unknown> {
const decision = decide(LEVEL, risks[name] ?? 'irreversible');
console.log(`[${decision}] ${name}`);
if (decision === 'deny') return { error: 'not permitted at this autonomy level' };
if (decision === 'ask' && !(await approve(name, input))) return { error: 'the user declined this action' };
return execute(name, input);
}
async function runAgent(userText: string): Promise<string> {
const messages: Anthropic.MessageParam[] = [{ role: 'user', content: userText }];
for (let step = 0; step < 6; step++) {
const response = await client.messages.create({
model: 'claude-sonnet-5',
max_tokens: 500,
tools,
messages,
});
messages.push({ role: 'assistant', content: response.content });
if (response.stop_reason !== 'tool_use') {
return response.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('');
}
const results: Anthropic.ToolResultBlockParam[] = [];
for (const block of response.content) {
if (block.type === 'tool_use') {
const output = await guarded(block.name, block.input as Record<string, unknown>);
results.push({ type: 'tool_result', tool_use_id: block.id, content: JSON.stringify(output) });
}
}
messages.push({ role: 'user', content: results });
}
return 'Stopped: step limit reached';
}
runAgent('Add 300 for groceries, then delete expense 1.')
.then(console.log)
.finally(() => rl.close());
A tool missing from risks falls back to irreversible, so anything you forget to classify gets the strictest treatment. Denials and declined approvals go back as error results, not exceptions, so the model can explain what happened.
What I noticed rereading my own series code
The system prompt template in my prompt-engineering post says “Never delete or change a past expense unless the user explicitly asks to.” The agent in that post has no delete tool. The line guards an action the agent can't take. The tracker from the components post writes with no approval and only a six-step cap, which was a Level 3 decision I never named. That is why the gate here lives in code next to execute, with the prompt line as a second layer.
Common mistakes
- Keeping the rule only in the prompt.
- Classifying tools by name instead of by what a wrong call costs.
- No default for unclassified tools. Fail closed.
- Raising the level for the whole agent instead of one tool at a time.
- Running approvals in parallel. Keep anything that waits on a person sequential.
- Treating a decline as a crash instead of returning it as a tool result.
Read the full version
I write each post in English and Bangla. The full version is on my blog: How Autonomous Should an AI Agent Be?
Which of your agent's tools would you never let run without asking?
Top comments (1)
Gating the tool call itself is the right place. In a headless run we saw a file write auto-denied, nothing created, and the process still exited 0. So the outer loop needs a clear denied or pending status, not a quiet skip.