When people say "AI agent," they usually mean one of two things. Either a chatbot that suggests what you should do, or a script with an API key that just does it. The first is safe and useless. The second is useful right up until the morning it quietly does something expensive.
I work on an agent that operates live ad accounts — real budgets, real money moving every hour. That constraint forces the question early: how do you hand execution to a model without handing over the keys entirely?
The pattern we landed on is graduated autonomy, and I think it generalizes well beyond ads.
The all-or-nothing trap
Most agent frameworks give you a binary. human_in_the_loop=True or False. Approve every tool call, or approve none.
Both settings are wrong most of the time:
- Approve everything turns into rubber-stamping within a week. Nobody reads the 40th confirmation dialog. You've built the illusion of oversight, not oversight.
- Approve nothing works fine until the model does something reasonable-looking and catastrophic, and now you're reconstructing what happened from provider logs.
The real problem is that the binary treats every action as equally risky. Pausing a keyword that spent $12 with zero conversions is not the same decision as restructuring a campaign. Trust is not a boolean, so the switch shouldn't be either.
Four levels instead of two
Level 0 OBSERVE Agent watches, reports findings. Executes nothing.
Level 1 RECOMMEND Agent proposes a concrete change. Human applies it manually.
Level 2 APPROVE Agent stages the change. Human clicks approve. Agent executes.
Level 3 AUTOPILOT Agent executes directly within a declared action envelope.
The important part is not the four levels — it's that the level is per scope, not per system. A new account starts at 0. After two weeks of the operator reading the agent's findings and agreeing with them, it moves to 1. Once staged changes have been approved without edits enough times, it earns 2. And a specific class of action — say, pausing zero-conversion keywords under a spend threshold — can be promoted to 3 while everything else stays at 2.
That last bit matters more than anything. You are not promoting "the agent." You are promoting this action type, in this account, under these bounds. It's closer to granting a permission than to trusting a coworker.
@dataclass(frozen=True)
class ActionEnvelope:
action_type: str # "pause_keyword"
scope: str # "account:1234"
max_level: int # 3
bounds: dict # {"max_daily_spend_delta": 50.0}
def resolve_level(action, envelopes) -> int:
match = next(
(e for e in envelopes
if e.action_type == action.type and e.scope == action.scope),
None,
)
if match is None:
return 0 # unknown action -> observe only
if not within(action, match.bounds):
return min(match.max_level, 2) # out of bounds -> demote to approval
return match.max_level
Note the failure mode: an unrecognized action doesn't get blocked with an error, and it doesn't get executed. It gets demoted. The agent still does its work, the human just sees it first. Unknown means "ask," not "crash."
Autonomy is worthless without an audit trail
Here's the part teams skip, and it's the part that actually makes the rest usable.
Every executed action has to carry the reasoning that produced it — not a post-hoc explanation generated by asking the model "why did you do that," which is confabulation, but the actual inputs the decision was made from.
{
"action": "pause_keyword",
"target": "kw:88213",
"level_used": 3,
"envelope": "pause_keyword@account:1234",
"observed": {
"spend_7d": 43.10,
"conversions_7d": 0,
"impressions_7d": 2210,
"account_cpa_target": 35.0
},
"rule_fired": "zero_conv_over_cpa_target",
"reverted_by": null,
"ts": "2026-08-20T09:14:22Z"
}
Two properties are non-negotiable:
-
Replayable. Given
observed, you can re-run the decision and get the same output. If you can't, you don't have a log — you have a diary. -
Reversible. Every autonomous action names its inverse.
reverted_bystarts null and gets filled if a human disagrees.
The reversion rate is the single most useful metric we track. It's the promotion signal and the demotion signal in one number. If humans revert less than a few percent of a given action type, that action type is ready to move up a level. If reversions spike after a platform change, the action type demotes itself automatically. Trust becomes measured rather than asserted.
What this buys you
The thing I did not expect: graduated autonomy makes the agent more useful at low levels, not just safer.
At Level 1, the agent is forced to produce a change specific enough for a human to apply by hand. That constraint kills vague output. "Consider optimizing your underperforming campaigns" is not applicable. "Pause keyword X, it spent $43 over seven days against a $35 CPA target with zero conversions" is. The discipline required for Level 3 turns out to improve Level 1.
And when something does go wrong, the question is answerable. Not "the AI did something," but: this action, at this level, from this envelope, on this data, at this timestamp — and here's the human who approved it, or here's why no human was in the path.
That's the difference between an agent you can run on production systems and a demo.
We build this into Soku, an agent that runs ad campaigns across Google, Meta, TikTok and ChatGPT Ads. The domain is ads, but nothing above is ads-specific — if your agent touches infrastructure, billing, customer data, or anything else where a bad afternoon is expensive, the same four levels apply.
Curious what others are doing here. If you've shipped an agent with execution rights, how did you scope its permissions?
Top comments (1)
This is a strong permission model. I’d avoid using reversion rate as the only promotion signal, though. It has selection and delay bias: humans only revert harms they notice, ad conversions arrive late, and an action can be locally correct while violating portfolio-level constraints.
I’d pair it with opportunity-weighted metrics: eligible actions, proposed/executed/approved counts, edit rate, time-to-review, delayed business outcomes, policy violations, and incidents per exposure. A shadow/holdout lane helps estimate the counterfactual—what would have happened without execution—instead of equating “not reverted” with “beneficial.”
The envelope itself should be versioned and evaluated at dispatch time against a fresh snapshot, with the action bound to account, target, arguments, data watermark, policy version, and expiry. That prevents approval or autopilot decisions from surviving changed inputs.
Finally, not every action has a true inverse. Pausing can be reversed; deleted learning history, spent budget, sent messages, or leaked data cannot. Each action class needs a recovery contract: inverse where possible, compensating action otherwise, plus blast-radius and cumulative-budget limits across many individually safe actions.