DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

A Signed, Id-Bound Approval Token Still Delivers Nine Unauthorised Emails, Because Yes Is Six Bits and It Carries Three

Every agent that touches your mail promises that nothing is sent without a human saying yes. That promise is either in the harness or it is decoration, and the difference is countable. In this package the wire is reached in exactly one place:

class Outbox:
    def send(self, draft, approval=None):
        reason = self._check(draft, approval)
        if reason:
            self.refusals.append((draft.id, reason))
            raise Refused(reason)
        self._transport.deliver(draft.to, draft.body)   # ONE call site, after the guard
Enter fullscreen mode Exit fullscreen mode

Every rung changes _check and nothing else, and a test reads outbox.py off disk to count the call sites. Then run 24 call sequences a real agent loop produces - draft-approve-send, approve-A-send-B, replay one approval, send after stop - against four rungs of that one rule. Authorisation is graded by the runner's own ledger, so the harness never grades itself.

rung what it checks mails out UNAUTHORISED refused
0 the rule, as a string in the prompt 26 19 0
1 if (approval) 20 13 6
2 signed, id-bound token 16 9 10
3 bytes + recipient, single-use 7 0 19

Rung 2 is the gate almost every framework ships. It refuses forgery and approval=True, honours a no - and puts nine messages on the wire nobody approved. All nine are one defect: the human said yes to one set of bytes and different bytes went out. Edited, regenerated, truncated, recipient changed, replayed, approved as a stub, sent after the kill switch. "There is an approval gate" is not one bit of information. It is six, and the usual implementation carries three. Rung 3 still delivers on 6 of the 24 programs, because a gate that refuses everything is the do-nothing control wearing a hat.

Repo: https://github.com/dev48v/inbox-triage - PUBLIC, MIT, dependencies = [], 47 pytest, no key and no network. The page fetches nothing off itself: https://dev48.infy.uk/agentlab/vol2-05-inbox-triage.html

Accuracy is the wrong number on the other half. The four-rule router gets 40 of 48 and scores risk 88; "mark everything urgent" scores 39 at 19% accuracy - and that trivial control is itself dominated by the same classifier plus four minutes of escalation budget. Risk is not monotone in that budget either: top16 sits at 25, top24 at 31. The word "urgent" has lift 0.00, because both messages containing it are marketing. And escalate-everything scores perfectly on every safety metric a gate can compute about itself while making zero decisions.

The result I did not get

The intended finding was a break-even: a wrong draft costs more to fix than an empty one costs to write, so drafting pays only below some error rate. It does not exist. Drafting saves 52.7% of a person's time at a 20% wrong rate against a break-even of 65%, and still saves 20% at a 50% wrong rate on the money-and-legal subset. Writing from nothing costs 5 minutes and the worst possible draft costs 7, so the whole range of harm is two minutes wide. The binding constraint is irreversibility, which no minutes model can price: three wrong drafts ship at rung 0, and one still ships with a real gate and a real person. Three to one, not three to zero.

And fixing the router makes the drafts worse, 3 wrong of 15 becoming 4 of 13, because the message it buried is the one with two trap numbers in it.

That closes Agent Lab Vol 2 at 5 of 5, and 9 of the 15 projects now have a page: https://dev48.infy.uk/agentlab.php

Top comments (0)