Would You Let an AI Agent Move Your Money?

What human-in-the-loop costs once it stops being a stub

Part 4 of an ongoing experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo grows with the series.

---

The rules say the customer is owed a €512.64 refund. The agent agrees. The API is one method call away. Who presses go?

That one line of code is where "AI-assisted" becomes "AI has authority." An agent can be perfectly capable of deciding that a refund is justified without being allowed to issue the refund. Deciding and doing are different permissions. That's the boundary I wanted to make impossible to blur.

The decision: separate judgement from authority (ADR 002)

I considered two designs:

• Option A: let the agent execute whatever tool it decides to call, constrained by prompts and instructions.

• Option B: assign every action a risk tier in code, then make consequential actions wait for a human regardless of how confident the model is.

I chose B. Not because I think the model is always wrong. Because I don't want model confidence to be an authorization mechanism.

The policy is deterministic:

• LOW -> execute autonomously; record the action in the audit trail
• MEDIUM / HIGH -> create a durable approval request; a human explicitly approves or rejects it
• VERY_HIGH -> propose only; this service has no execution path for it

That last distinction matters. "Please don't do this" is a prompt instruction. "There is no code path that can do this" is an architectural property.

flowchart LR
    P["Agent proposes action"] --> G{"RiskPolicy.tierFor()"}
    G -- "LOW" --> E["Proceeds autonomously<br/>audit recorded"]
    G -- "MEDIUM / HIGH" --> Q["Approval queue<br/>+ audit trail"]
    Q --> H["Human approves or rejects"]
    G -- "VERY_HIGH" --> M["Queued as propose-only<br/>human executes manually"]
    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
    classDef human fill:#ecf2ed,stroke:#93b39d,color:#3d5344
    class P,E,Q,M step
    class G decision
    class H human

The consequence is uncomfortable but intentional: A wrong model decision can produce a wrong proposal. It cannot produce a wrong execution.

The gate is deliberately boring

The enforcement point is one method. Before the switch even runs, there's an important rule: an action with no assigned risk tier is refused. No default. No "probably safe". No fallback to whatever the model requested. The system fails closed.

// dev/tonal/support/application/GatedActionService.java
return switch (RiskPolicy.tierFor(action)) {
    case LOW -> {
        audit.record("PROCEEDED %s (LOW) — %s".formatted(action, 
                description));
        yield new Result(Outcome.PROCEEDED_AUTONOMOUSLY, null,
                "Low-risk action executed with sign-off on output");
    }
    case MEDIUM, HIGH -> {
        PendingApproval proposal = new PendingApproval(
                UUID.randomUUID().toString(), 
                action, 
                tier,
                description, 
                OffsetDateTime.now());
        String id = queue.enqueue(proposal);
        audit.record("QUEUED %s (%s) — %s".formatted(action, tier, id));
        yield new Result(Outcome.QUEUED_FOR_APPROVAL, id,
                "Awaiting human approval");
    }
    case VERY_HIGH -> { /* enqueued as propose-only, flagged manual */ }
};

Three properties do most of the security work.

1. Fail closed

An unknown action is refused and audited. The system doesn't guess that an unclassified action is safe.

2. VERY_HIGH has no execution path

This isn't a configuration flag. The service physically doesn't know how to execute a VERY_HIGH action. You can read the class and verify that property.

3. Every decision is auditable

Refusals. Autonomous executions. Approval requests. The important events all leave an audit record. When someone asks six months later, "What happened to that refund?", the answer should be a query — not an archaeological expedition through logs.

I also pinned the strongest claim with a test:

@Test
void veryHighRiskActionsAreNeverExecutedByTheSystem() {
    GatedActionService.Result result =
            service.propose(ActionType.DELETE_DATA,
                  "purge export artifacts");

    assertThat(result.outcome())
            .isEqualTo(
                  GatedActionService.Outcome.QUEUED_FOR_MANUAL_EXECUTION);
}

If someone later adds an execution path for VERY_HIGH actions, I want the test suite to complain before production does.

But human approval isn't free

This is the part that's easy to hand-wave away. It's tempting to say "just put a human in the loop."

Fine. Which human? Where does the approval request live? How long does it stay valid? How do they know it arrived? What happens if nobody responds? Can the same request be approved twice? What exactly did the agent propose? What did the human actually approve? How do you reconstruct the decision six months later?

Those questions turned "human approval" from a boolean into infrastructure. I needed a durable queue, reviewer notification, and an append-only audit trail.

A gate nobody can actually open and review isn't safety. It's just latency.

The option I deliberately didn't build

There's a middle ground between approving every transaction and manually executing everything: pre-authorized mandates.

For example, an account owner could say:

Refund up to €50 per customer, up to €500 per day, and only to the original payment method.

The agent proposes the refund. The deterministic policy checks the bounds. If it fits, no individual approval is required.

The human still owns the authority — they've just exercised it as policy rather than one transaction at a time. It's attractive.

I still deferred it. Because a loose mandate can authorize a lot of quiet mistakes before anyone notices. A real implementation would need expiry, review cadence, tight bounds, recipient constraints, and its own audit trail. That's worth building if the approval queue becomes a measured bottleneck. Not because it feels elegant.

The cost is latency

A refund the agent could theoretically issue in three seconds might now wait until a human is available at 8 AM.

The customer gets:

"We'll process this within one business day."

That's worse UX. On purpose. The latency is the price of keeping execution authority outside the model. And that's why I don't put everything behind the same gate.

A support agent can answer a question instantly. It can classify a ticket. It can draft a response. It can probably update some low-risk metadata without waking anyone up.

But when the action moves money, changes access, deletes data, or otherwise creates consequences that are difficult to undo, the economics change.

The goal isn't human approval everywhere. The goal is human ownership where the consequences justify it.

The same pattern shows up outside support:

• Loan systems separate scoring from disbursement.
• Clinical systems can assist with triage without prescribing.
• Industrial systems can use automated perception while interlocks retain control of dangerous execution.

Wherever a model meets a consequential action, someone has to own the trigger.

I don't want that someone to be the thing that also guesses.

---

And if you think I'm being too cautious with that €512.64, good. That's exactly the argument I want to have. Because the tiers shouldn't be based on vibes. They should evolve when the evidence says they should.

---