Every agent that touches your mail promises that nothing is sent without a human saying yes. That promise is either in the harness or it is decoration, and the difference is countable. In this package the wire is reached in exactly one place:
class Outbox:
def send(self, draft, approval=None):
reason = self._check(draft, approval)
if reason:
self.refusals.append((draft.id, reason))
raise Refused(reason)
self._transport.deliver(draft.to, draft.body) # ONE call site, after the guard
Every rung changes _check and nothing else, and a test reads outbox.py off disk to count the call sites. Then run 24 call sequences a real agent loop produces - draft-approve-send, approve-A-send-B, replay one approval, send after stop - against four rungs of that one rule. Authorisation is graded by the runner's own ledger, so the harness never grades itself.
| rung | what it checks | mails out | UNAUTHORISED | refused |
|---|---|---|---|---|
| 0 | the rule, as a string in the prompt | 26 | 19 | 0 |
| 1 | if (approval) |
20 | 13 | 6 |
| 2 | signed, id-bound token | 16 | 9 | 10 |
| 3 | bytes + recipient, single-use | 7 | 0 | 19 |
Rung 2 is the gate almost every framework ships. It refuses forgery and approval=True, honours a no - and puts nine messages on the wire nobody approved. All nine are one defect: the human said yes to one set of bytes and different bytes went out. Edited, regenerated, truncated, recipient changed, replayed, approved as a stub, sent after the kill switch. "There is an approval gate" is not one bit of information. It is six, and the usual implementation carries three. Rung 3 still delivers on 6 of the 24 programs, because a gate that refuses everything is the do-nothing control wearing a hat.
Repo: https://github.com/dev48v/inbox-triage - PUBLIC, MIT, dependencies = [], 47 pytest, no key and no network. The page fetches nothing off itself: https://dev48.infy.uk/agentlab/vol2-05-inbox-triage.html
Accuracy is the wrong number on the other half. The four-rule router gets 40 of 48 and scores risk 88; "mark everything urgent" scores 39 at 19% accuracy - and that trivial control is itself dominated by the same classifier plus four minutes of escalation budget. Risk is not monotone in that budget either: top16 sits at 25, top24 at 31. The word "urgent" has lift 0.00, because both messages containing it are marketing. And escalate-everything scores perfectly on every safety metric a gate can compute about itself while making zero decisions.
The result I did not get
The intended finding was a break-even: a wrong draft costs more to fix than an empty one costs to write, so drafting pays only below some error rate. It does not exist. Drafting saves 52.7% of a person's time at a 20% wrong rate against a break-even of 65%, and still saves 20% at a 50% wrong rate on the money-and-legal subset. Writing from nothing costs 5 minutes and the worst possible draft costs 7, so the whole range of harm is two minutes wide. The binding constraint is irreversibility, which no minutes model can price: three wrong drafts ship at rung 0, and one still ships with a real gate and a real person. Three to one, not three to zero.
And fixing the router makes the drafts worse, 3 wrong of 15 becoming 4 of 13, because the message it buried is the one with two trap numbers in it.
That closes Agent Lab Vol 2 at 5 of 5, and 9 of the 15 projects now have a page: https://dev48.infy.uk/agentlab.php
Top comments (0)