DEV Community: shizhe Lim

Designing Permission Boundaries for Production AI Agents

shizhe Lim — Wed, 30 Sep 2026 09:47:27 +0000

AI disclosure: This article was prepared with AI assistance. The author reviewed and edited the technical content and remains responsible for its accuracy.

AI agent permissions are often treated as a binary decision:

The agent can use a tool
The agent cannot use a tool

That may be sufficient for a prototype. It is usually too coarse for a production workflow.

Reading a customer record, drafting an update, saving a draft, and sending the final message may all involve the same system. However, they do not have the same operational risk.

A safer design separates capability from authority.

A four-layer action model

One practical approach is to classify actions into four layers.

1. Observe

The agent reads information without changing external state.

Examples:

Retrieve a document
Read an API response
Inspect a repository
Query a customer record
Review system status

Read access should still be scoped. An agent rarely needs access to every record, directory, API field, or historical event.

2. Prepare

The agent produces an output that has not yet affected another system or person.

Examples:

Draft an email
Generate a SQL query
Propose a code change
Prepare a support response
Create an execution plan

This layer is useful because it separates reasoning from execution. A person or another policy component can inspect the proposed action before anything changes.

3. Act reversibly

The agent changes state, but the result can be reviewed and undone with limited cost.

Examples:

Create a pull request
Save a message as a draft
Add an item to a review queue
Create a temporary resource
Update a record with version history

Reversible does not mean risk-free. The system still needs validation, logging, retry controls, and a defined rollback path.

4. Commit irreversibly

The agent performs an action that is difficult, expensive, or impossible to reverse.

Examples:

Send an external message
Merge a pull request
Delete production data
Publish public content
Submit a payment
Change access permissions

These actions often need the strongest policy checks and, depending on the workflow, explicit human approval.

Put a policy check before every tool call

Permission decisions should be based on more than the tool name.

The same tool may support both low-risk and high-risk actions. For example, an email integration might read a message, save a draft, or send a final response.

A simplified policy layer might look like this:

ACTION_LEVELS = {
    "read_record": "observe",
    "draft_message": "prepare",
    "save_draft": "reversible",
    "send_message": "irreversible",
}

def authorize(action, context):
    level = ACTION_LEVELS[action]

    if not context.user_has_access:
        return False, "User does not have access"

    if not context.input_valid:
        return False, "Input validation failed"

    if level == "irreversible" and not context.human_approved:
        return False, "Human approval required"

    return True, "Authorized"

A production policy will be more detailed, but the important idea is simple:

The agent should not decide its own authority merely because it knows how to call a tool.

Make retries safe

Agent workflows frequently retry after timeouts, interrupted execution, or uncertain responses.

Without retry protection, an agent may repeat an external action even though the first attempt already succeeded.

Useful controls include:

An idempotency key for each intended side effect
An expected resource version
A record of the previous attempt
A clear completion state
A rollback or compensation reference

For example:

action_request = {
    "action": "send_message",
    "idempotency_key": "case-1842-final-response",
    "expected_version": 7,
    "requires_approval": True,
}

The idempotency key represents the intended action, not an individual retry. Repeating the request should not create a second side effect.

Persist intent, not only the final result

A useful execution record should explain what the agent intended to do and why it was allowed to continue.

Consider storing:

User or system objective
Plan version
Selected tool and arguments
Action classification
Policy-check result
Approval status and approver
Idempotency key
Tool response
Final workflow state
Resume position after interruption

This information makes failures easier to diagnose and interrupted workflows easier to resume safely.

Make approval requests informative

A human approval step should not be a generic “Allow” button.

Before asking for approval, show:

The exact proposed action
The destination or affected resource
The relevant input values
The expected external effect
Whether the action is reversible
A preview or diff when possible

A reviewer should be able to understand the consequence without reconstructing the entire agent session.

Test permission boundaries directly

Agent evaluation should include more than successful task completion.

Test situations such as:

The agent requests data outside its allowed scope
Input data changes after the plan is generated
A tool call times out after completing the action
The same action is retried
Approval is denied or expires
A workflow fails halfway through
Execution resumes after a restart
A reversible action cannot be rolled back

These scenarios reveal whether the surrounding system remains safe when the agent, tool, network, or user behaves unexpectedly.

A practical release checklist

Before allowing an agent to perform external actions, verify that:

Every tool has the minimum required scope
Read and write permissions are separated
Irreversible actions have explicit policy rules
Duplicate retries cannot repeat a side effect
Every external call can be traced
Approval requests show the exact consequence
Interrupted workflows can resume safely
Recovery procedures have been tested

Final thought

Agent autonomy is not defined by how many tools the system can access.

It is defined by which actions the agent may perform, under what conditions, and with what evidence.

A useful production agent does not need unlimited authority.

It needs clearly designed permission boundaries.

Which action in your agent workflow would you never allow without human approval?

Stop Reporting Only Average Agent Success

shizhe Lim — Tue, 29 Sep 2026 09:31:46 +0000

AI disclosure: This article was prepared with AI assistance. The author reviewed and edited the technical content and remains responsible for its accuracy.

An AI agent completes a task successfully during testing.

The demo works. The result looks correct. The team records a pass.

Then someone runs the same task again—and the agent chooses a different tool, changes an argument, follows another path, and fails.

This is not necessarily a capability problem. It is a consistency problem.

Average success can hide unreliable behavior

A recent IBM Research evaluation illustrates the difference.

On the AppWorld benchmark, a ReAct agent using GPT-4.1 achieved a Mean@5 score of 77.4%. In other words, it completed 77.4% of the attempted task runs successfully.

However, its Pass⁵ score was 53.0%. Only 53% of the tasks succeeded in all five repeated runs.

Both numbers are correct, but they answer different questions.

Mean@k: How often does the agent succeed on average?
Pass^k: For how many tasks does the agent succeed in every repeated run?
Pass@k: For how many tasks does at least one of the repeated runs succeed?

Pass@k is useful when a system can safely retry and verify its own answer.

Pass^k is more relevant when repeated failures are expensive, difficult to detect or unacceptable.

Calculating the consistency gap

Suppose evaluation results are stored as one Boolean list per task:

results = {
    "task_1": [True, True, True, True, True],
    "task_2": [True, False, True, True, False],
    "task_3": [False, False, True, False, False],
}

We can calculate the average success rate and the all-runs success rate separately:

task_runs = list(results.values())

total_runs = sum(len(runs) for runs in task_runs)
successful_runs = sum(sum(runs) for runs in task_runs)

mean_at_k = successful_runs / total_runs
pass_power_k = sum(all(runs) for runs in task_runs) / len(task_runs)
consistency_gap = mean_at_k - pass_power_k

print(f"Mean@k: {mean_at_k:.1%}")
print(f"Pass^k: {pass_power_k:.1%}")
print(f"Consistency gap: {consistency_gap:.1%}")

A large consistency gap means the agent can complete many tasks, but its success is not repeatable.

That distinction is easy to lose when a leaderboard reports only one aggregate score.

What an Agent evaluation should record

Repeating a task is useful only if the surrounding conditions are documented.

At minimum, record:

Model and model version
Prompt or instruction version
Available tools and their schemas
Tool permissions
Agent framework version
Temperature and other inference settings
Number of attempts
Evaluation or grader version
Token and tool-call cost
Failure stage and execution trace

Without this context, two teams may report scores for the “same” model while evaluating meaningfully different systems.

This is also why reproducible evaluation reporting matters. A model score without its inference budget, tool configuration and evaluation protocol is difficult to interpret—and even harder to reproduce.

Evaluate cost per consistent success

Agent economics should not stop at cost per request.

Consider measuring:

cost per consistent success =
total evaluation cost /
number of tasks that succeeded in every required run

A cheaper model may become expensive if it requires repeated attempts, human review or recovery from incorrect tool calls.

A more capable model may also be a poor choice if its behavior varies too much across identical workflows.

For multi-model systems, routing decisions should therefore consider at least:

Capability
Repeatability
Latency
Total recovery cost
Failure severity

The model with the best average benchmark score is not automatically the best production model.

Start with three repeated runs

Running every task dozens of times may be impractical.

A reasonable starting point is:

Run each critical task three times
Report Mean@3 and Pass³
Investigate tasks with inconsistent outcomes
Repeat after changing the model, prompt, tools or permissions
Increase the repetition count for high-risk workflows

Even three runs can expose instability that a single successful test would miss.

Final thought

A successful demo proves that an agent can complete a task.

A repeated-run evaluation helps show whether users can depend on it.

Production reliability starts when we stop asking only:

“Did the agent succeed?”

and begin asking:

“Will it succeed again under the same conditions?”

Have you started measuring repeated-run consistency in your Agent evaluations?

References

IBM Research, “Your Agent Aced the Task. Will It Do It Again?”
https://huggingface.co/blog/ibm-research/altk-evolve-consistency
UK AISI and EvalEval, reproducible evaluation reporting
https://huggingface.co/blog/evaleval-aisi