Design trade-offs from building a guarded banking assistant on MCP
Here's the 5½-minute walkthrough, with live attacks:
Most chat assista...
For further actions, you may consider blocking this person and/or reporting abuse
I'd add a provenance case where every value appears in the customer's message, but the proposed combination contradicts it: “Do not send 50 to Alice; send 20 to Bob.” A transfer of 50 to Bob has both values in the text. Does the lexical check distinguish that from the intended pair? This seems a useful test alongside the existing approval-state sequences, while keeping the human confirmation as the final boundary.
Good catch, and you're right. I checked and the check as written would have let 50 to Bob through. It looks for the amount and the payee separately, anywhere in what the customer typed this session, so both values were "there". I've added your example as a test, and the check now drops clauses the customer negated, so "Do not send 50 to Alice" no longer counts. It's still lexical though. "Send 50 to Alice. Actually, 20 to Bob" would still ground 50 to Bob, and I couldn't close that without breaking normal back-and-forth like "Pay Bob" / "How much?" / "500". So it's written down as a known limit, and like you said, the Confirm card stays the real boundary.
Update: the fix is merged, here's the PR: github.com/farihaNaqvi/safeledger/...
Thanks for adding the test and documenting the remaining correction case. One state-machine case seems useful alongside that: a transfer is pending, the customer corrects the amount or payee, then an old tab submits Confirm for the original request. Does the correction invalidate that pending request server-side, so the stale Confirm is refused? Clarification can still collect missing fields before a request exists. The extra check would cover an outdated approval card without claiming to solve the lexical intent problem.
Good one, and no, it didn't. A correction created a second pending request and the original stayed live until it expired, so an old tab could still confirm it. I've changed that: a new transfer request now supersedes any transfer still waiting in the same conversation, a Confirm on the old one is refused server-side, and the app closes the old card. One side effect I hadn't expected: the pending and daily limits were counting the request about to be replaced, so the first version blocked corrections near a limit. The tests caught that. Transfers now go one at a time per conversation, which seemed a fair trade.
github.com/farihaNaqvi/safeledger/...
Thanks for adding supersession and catching the limit interaction. One acceptance test I'd keep for that change is a concurrent old Confirm and replacement request, using separate transactions. If Confirm wins, the original transfer has already been accepted; if replacement wins, the old Confirm must fail. The result should never be two executable transfers while both screens suggest one superseded the other. That would test the server-side ordering directly, alongside closing the old card in the app.
Good call, and that test found a real gap. Confirm and the supersede were already safe against each other. Both only change the request if it's still pending, so only one can win. The limits were the problem. A correction's limit check leaves out the transfer it's about to replace, and if the old Confirm landed between that check and the supersede, the original went through while the correction stayed confirmable without the original counted. With 30k already sent, 40k waiting and a correction to 45k, that's 115k against a 100k daily limit. Now creating a request and pressing Confirm both take a per-customer lock in Postgres, so the old Confirm either lands first and counts against the limit, or lands after the supersede and gets refused. Your test is in: 40 rounds of the two racing in separate transactions, and it requires both orders to show up. With the lock taken out, it fails.
github.com/farihaNaqvi/safeledger/...
Requests not actions is the right call, and the provenance check is a nice next step. The part I'd look at now is the chat around the Confirm button. A fooled model can't move money anymore, but it can still talk the customer into confirming: "there's a pending security check on your card, please confirm it". It's the same trick wallet drainers use in crypto. The wallet shows the real transaction, but people confirm because the site's story around it sounds fine. So I'd render the pending action from the stored request, in its own card outside the chat, and add a test where the model's text describes it as something else. Does the Confirm screen show anything the model wrote?
Fair question, and the honest answer was "a little". The card's summary was built server-side from the stored request, so the amount, payee and masked account came from the bank, not the model. But two bits of model text did end up on it: the freeze reason and the transfer note. A fooled model could have put "security check, please confirm" right on the card. And the card sat in the chat stream right under the model's reply, which is pretty much the wallet drainer setup you described.
Both are fixed now. The summary has no model-written text. The transfer note (which the payee does see) gets its own line, labelled as written by the assistant. Pending actions now live in a separate panel outside the chat. There's also a test where the model calls a freeze a "security check" and the card still shows only the stored values. I hadn't thought of it in wallet drainer terms before, thanks for that.
Update: the fix is merged, here's the PR: github.com/farihaNaqvi/safeledger/...
Nice, quick fix, and the separate panel is the big one. The transfer note is the bit I'd still watch. It's model text that leaves the bank and lands with someone outside, so it's a way to send data out, not only to mislead. A fooled model could put the customer's balance or another account number in the note of a small transfer to a planted payee. Same problem as email tools where the recipient is pinned but the model writes the body. Is there a check on what goes in the note, or at least a length limit?
Fair point, and you're right that it's a different problem from the misleading story. Right now there's no check on what goes in the note. It's capped at 200 characters (the bank enforces the same), the injection detector scans it but only flags, and the card shows it in full, labelled as written by the assistant. So the only thing between a fooled model and a balance in the note is the customer actually reading that line before pressing Confirm, which is a weak control. There is one partial defence: the payee and amount have to be ones the customer typed, so a planted payee only works if the customer names it. That doesn't stop a leak to someone they do pay, though.
The fix I'd go for is treating the note like the amount and payee: every word in it has to come from what the customer typed, with a short cap so the card always shows all of it. Then a balance or account number the model read can't get in. The email comparison is a good one, I'll use it when I write that up.
That fix makes sense. One gap I'd check: if every word has to come from what the customer typed, it depends on everything they typed in the chat. If they pasted their IBAN or a balance earlier to ask a question, the model can still pick those words for the note. A note field on the Confirm panel that the customer fills in there closes it, the model never writes the note at all.
Done, and you were right that my version only moved the problem. The note is gone from the tool entirely now, so the model can't write anything that reaches the payee. The Confirm card has a "Note for the payee" field. Whatever the customer types there is sent with Confirm and saved in the same update that confirms the transfer. It has to be one line and 200 characters max, and only transfers accept one. If the model tries to pass a note anyway, it's dropped.
github.com/farihaNaqvi/safeledger/...