DEV Community: arun rajkumar

Adding Pay by Bank to WooCommerce: setup, sandbox testing, and the webhook setting people miss

arun rajkumar — Wed, 14 Oct 2026 05:51:19 +0000

A merchant installs a gateway plugin, takes one test payment, goes live, and a few weeks later notices some orders sitting on "Pending payment". The plugin usually isn't the problem. What gets missed is the case where the customer pays but their browser never makes it back to the shop.

This is how I'd add Pay by Bank to a UK WooCommerce store. Setup takes about ten minutes. The webhook section is where the time actually goes.

Is it right for your store?

Pay by Bank sends the customer to their banking app to approve, then back to you. Compared with Apple Pay that's an extra hop, and on a £12 impulse buy it can cost you a sale. Fena's write-up on this suggests the fee difference starts to matter above roughly £100 and more so above £250 (source). That fits what we see: bigger baskets, trade buyers, categories that suffer chargebacks, and mostly UK customers.

I wouldn't replace cards. Offer both at checkout and let people choose. Bank payments cost about half of what cards do, and there's no chargeback route because the customer approved the payment inside their own bank's app.

Setup

You need an Atoa Business account that has finished onboarding and KYB.

In the Atoa Business app go to Settings → API Access and generate a Sandbox key. Add your shop's homepage as the redirect URL.
In WordPress, install the plugin from Plugins → Add New (search "Atoa Pay", slug atoa-pay). It needs WordPress 6.0+ and PHP 8.0+.
Go to WooCommerce → Settings → Payments, click Manage next to "Atoa – Pay by Bank App", paste the key, choose your Store Location, turn on Sandbox Mode, enable the method and save.

Both block and classic checkout work. Card, Apple Pay and Google Pay can be switched on separately under the same account.

Testing

At the bank picker, choose Atoa Test Bank. You then get a screen where you pick the result yourself:

COMPLETED: payment succeeded, the order should move to processing.
FAILED: the customer goes back to checkout to try again.
PENDING: the bank hasn't confirmed yet.

Run all three. PENDING is the closest thing to a customer closing the tab halfway through, so it tells you the most.

The webhook setting people miss

The plugin registers its webhook URL for you and re-syncs it each time you save settings. It also checks payment status 5 minutes, 30 minutes, 1 hour, 4 hours and 24 hours after checkout, in case a webhook never arrives.

The problem I'd watch for: if you enable V2 webhook signing in the Atoa Dashboard (Settings → Webhooks), the signing secret has to be pasted into the plugin's Webhook Signing Key field. If it isn't, the docs say plugin webhooks will fail signature verification, and orders stay in Pending until one of those status checks catches up.

V2 signs the raw request body with HMAC-SHA256 and sends the result in an X-Atoa-Signature header. If you have your own endpoint (fulfilment, ERP sync), this is the check in PHP:

<?php
$secret = getenv('ATOA_WEBHOOK_SIGNING_SECRET');
$raw    = file_get_contents('php://input');            // raw body, not decoded
$header = $_SERVER['HTTP_X_ATOA_SIGNATURE'] ?? '';
$given  = preg_replace('/^v1=/', '', $header);         // strip the v1= prefix
$expect = hash_hmac('sha256', $raw, $secret);

if (!hash_equals($expect, $given)) {                   // timing-safe compare
    http_response_code(401);
    exit;
}

$event = json_decode($raw, true);
// handle PAYMENTS_STATUS / REFUND_STATUS idempotently, then:
http_response_code(200);                               // anything but 200 counts as a failure

Hash the raw body, because re-encoded JSON won't produce the same bytes. Use hash_equals for the comparison. And make the handler safe to run twice: Atoa retries with exponential back-off for up to 24 hours on any non-200 response, so duplicates will happen.

Going live

Swap in a Production key, turn off Sandbox Mode, and make one real £1 payment from your own bank account. Check the order status changes, then work through the go-live checklist.

The sandbox guide and API reference are at docs.paywithatoa.co.uk, and the plugin is on WordPress.org.

If you've had orders stick in odd states on your own store, I'd like to hear what caused it.

I Have Four Controls for AI Agents. Each One Stops Measuring the Moment People Know It's There.

arun rajkumar — Mon, 12 Oct 2026 06:45:24 +0000

Three weeks ago I wrote that nobody checks whether the guardrail is running. Four people have spent those three weeks proving the post stopped one step too early.

The post was about a control that exists and isn't executing. What they found is worse, and it arrived four separate times in four separate threads, from people who were not talking to each other. A control can be running, monitored, read by a human on a schedule, and still stop measuring the thing it was built to measure. Not because it broke. Because people learned it was there.

I want to go through the four, because each one is a real instrument with real numbers attached, and each one dies the same way.

The rejection rate, broken by the person who proposed it

The thread started with exemption approvals. Somebody has to sign off when a control gets a carve-out, and the question was how you tell a real signer from a rubber stamp.

@peterbuildssecure made the move I'd been failing to make. Sitting outside the requester's chain is not what makes a signer real. What makes them real is bearing a cost for being wrong that is larger than the cost of asking the requester an awkward question. In payments that is why a bank's check on a mandate survives for decades and an internal second signature often does not survive a busy quarter: the bank's own money moves.

So I offered the cheap instrument. Rejection rate. A signer outside the chain who has approved every exemption for a year is not a control, and you can decide that without judging any individual approval or knowing anything about the domain.

He then took it apart, which is the part worth reading:

A zero rejection rate can also mean requesters learned what passes and stopped sending anything else.

That is not a measurement error. The signer is working. The requesters have adapted. And the number reads identically in both cases, so the instrument cannot tell a control that is deterring bad requests from a control that waves everything through.

His fix is to seed. Submit a known-bad exemption quarterly and see whether it gets stopped. That separates "signer never rejects" from "nothing worth rejecting arrives", which the rate alone cannot.

I'll take it, with one limit. An obviously bad exemption that gets caught tells you the obvious case is caught, which nobody doubted. The informative seed is a marginal one, close enough to the real submissions that a tired signer might wave it through. Building that needs somebody who can judge marginal cases well, which is the capability you were trying to test. The seed inherits the problem one hop back.

The same thing, in production, with dates on it

While that was theoretical, @road511 was running it.

He maintains a list of exempted data feeds — feeds allowed to produce nothing without raising an alert. In September he came back with numbers: 897 polled endpoints across 286 upstream servers, where an upstream server means one with at least one enabled resource,, re-pulled on 21 September so the figures had a date on them rather than being three days stale. Ten exemptions on the North American deployment. Zero on the EU one, and his correction on that was more useful than the thing it corrected: the EU list is not better disciplined, it is younger. Which makes the count useless as a health signal in both directions. A long list looks like rot, a zero list looks like discipline, and the length is mostly a clock.

Seven of the ten shared one renewal reason. The same 153-character string, the same 13 October expiry.

I suggested a dedup rule: reject a renewal whose reason matches the previous one for that row. He was careful not to promise it would ship, and said the real value was that it converts a batch sweep back into seven separate decisions.

He came back on 11 October, before the date, with the result.

They looked at the seven one by one. Three did not need an exemption at all — two feeds were producing again, and a third had produced six rows during its own exemption. Four were renewed, each with its own numbers and its own date: two feeds that have published 1 and 2 items in their entire existence run to 31 December, two episodic ones run to 30 November.

Nothing forced honesty. No rule fired. A date forced seven decisions, and somebody looked at them one at a time. The rule I proposed would have been the cheaper version of the thing that actually worked.

And then the part that made me start writing this.

I'd offered him a test: a template reason is one that doesn't name the feed it's about. A real one does. Run it against the ten this afternoon rather than waiting for October. He ran it on the result, and it passed — every renewed reason names its feed and carries its own counts, the batch sentence named none. Length says nothing, which was the other thing I'd have reached for: the honest reasons run 151 to 159 characters, and so did the batch one.

The list now holds six exemptions, no two sharing a sentence.

That is what a careful list looks like. It is also exactly what a list looks like once the author knows the sentence gets read.

Counting instead of thresholds, until the rounding kills it

A different thread, same shape. @build996 and I had a ratio to watch: a value an upstream reports against the value we apply, where a divergence means one side quietly moved.

He wanted to alert on the ratio crossing a threshold. I argued for the step instead — a ratio that walks down over a quarter crosses any line you set on a day when nothing in particular is wrong, and by then the thing you'd compare against has been drifting the whole time.

Then the actual failure. If the upstream stops returning its measured value, a sensible implementation falls back to the applied value, and the ratio pins to exactly 1.0 forever. Healthy, constant, and computed from one input divided by itself.

The check with no threshold and no baseline: alert when the variance of the ratio over the window is zero and the sample size is not. A ratio built from two independently measured numbers jitters, because clocks and rounding and retries move both sides a little.

His correction is better than my check:

The false positive I'd expect first isn't low traffic, it's quantisation.

If the upstream reports its window in whole seconds and we store ours rounded the same way, an honest ratio sits at exactly 1.0 for long stretches. And it is worse than he put it, because there is no raw value for the upstream's half. One second is the measurement. Nothing downstream recovers a finer one.

@_firelinks landed on counting from a different direction entirely — a partner whose settlement calendar we're guessing at, and the question of when a guess has been tested rather than merely not yet disproved. His fix was two counts: periods matched, and periods that could have disproved the guess. The second one is the whole thing. Three months where both candidate rules predict the same date test nothing.

Two people, two problems, same escape route: when a threshold needs a baseline nobody recorded, count something instead.

The word outlives the code

@rafael_asor has been building the cryptographic version of this, in public, with runs posted rather than claims made. A supervised process writes a chain of entries; a second key signs one of them; a reader scores that entry witness-signed if the key is in a trust set the reader supplies.

I asked him to falsify the obvious hole — both keys generated in one script means the witness and the subject share a custodian. He did it the same afternoon. The witness runs as its own process, generates its key in memory, and the supervised process searched 1,277 files, its own argv, its environment and its own heap for the seed, asked the witness directly, and tried signing with a borrowed key id. Four ways, all refused.

The most useful result in that run is the one he called a limitation. A forged allow, appended in chain order, came back witness-signed. The witness attests that it received that entry, in that order, at that time. It does not attest that the action was permitted.

witness-signed reads, to anyone who did not write the code, as "a second party vouched for this action."

He filed it as an issue, along with two others, off one comment. And the naming one is the only one of the three that gets harder with time. The other two are code. Once a word is in a changelog, people build readers on it.

Payments spent real money learning this. Authorised, captured and settled are three words for what looks like one event, and they stay separate because a system that treats an authorisation as a settlement loses money in a way nobody notices for a month. No scheme lets a timestamp authority stand in for an authoriser. They are two roles with two names for exactly this reason.

The thing all four have in common

Every control in this post is only worth what somebody independently reads of it. That part I already believed, and three of these threads are better arguments for it than the article I wrote.

The part I did not have is what happens next.

A visible control changes what arrives at it. So each of these instruments has a shelf life rather than a false positive rate.

The rejection rate worked until requesters learned what passes. The feed-naming sentence worked once, and the six distinct sentences on that list today are partly a careful author and partly an author who knows the sentence gets checked. The zero-variance check works until an upstream rounds to the second. witness-signed was accurate on the day it shipped and will be misread by every reader who arrives after it.

None of that is a reason to stop building them. It is a reason to date them, the way @road511 dates his figures, because a figure with a date on it stays true for a useful while and a figure without one quietly stops being true. An instrument needs the same treatment. Write down when you started trusting it and what it was measuring then, or you will keep reading a number whose meaning has moved.

Which gives one concrete preference over everything above. Prefer the instrument whose input comes from somewhere the measured party cannot adapt: @road511's renewal dates carry a number from the feed, not a sentence about it. "Published 1 item ever, runs to 31 December" is checkable on 1 January by someone who has never read the reason, and the feed cannot learn to look busier.

Where I can't finish the argument

Reconciliation survives decades in payments, and not because it is clever. It survives because the counterparty runs their half for their own reasons and will not stop because our number is inconvenient. Our ledger and the bank's statement are two records with two authors and two sets of incentives, and the gap between them is the control.

None of the agent-side controls in this post have a counterparty. The signer is in the same company. The reader supplies their own trust set. The ratio is computed from two numbers that both arrive over the same wire. The exemption list is read by the team that writes it. Every one of them is one party in two coats, wearing the second coat honestly.

I don't know where the counterparty comes from for an agent. A regulator is one answer, and it's the answer I reached for in an earlier post, and it isn't available to most people reading this. The nearest thing I reached for is a published revision history that the measured party cannot quietly change. That is not a second author, but it removes the ability to rewrite the first one. To be clear about whose idea that is: it is mine, nobody in these threads proposed it, and nobody including me is running one.

If you have a real second reader in an agent system — somebody outside, who checks because it costs them when it's wrong — I would like to know what they are and why they bother.

Thanks to @peterbuildssecure, @road511, @rafael_asor, @_firelinks, @build996, @tom_jones_230c4659491adcd, @scarab-systems, @alexshev and @anp2network, who between them have taken out more of my arguments in three weeks than I managed to build.

Two Strangers Built an Agent Mandate Protocol in My Comments. It Still Needs a Regulator.

arun rajkumar — Wed, 16 Sep 2026 05:45:09 +0000

I got something wrong in my own comments section nine days ago, and two strangers spent a week showing me how wrong.

The article was about guardrails that quietly stop running. Somebody in the thread described a scheme where an agent's authority is checked against a list of state revisions. I said the trouble with a list is that it rots, and the fix is expiry. Short lifetime, list never grows.

@anp2network said no:

A payment authorisation does not get its lifetime from a clock. It gets it from naming the effect [...] The question stops being "is this still fresh" and becomes "has this already been spent".

He is right, and I should have got there first, because it is what I do for a living. A card authorisation for £40 cannot be re-presented for £4,000. Not because it expires quickly. Because it names the amount and the payee, and the side that moves the money is the side that checks.

What happened after that ran twenty replies deep, mostly without me. @peterbuildssecure turned up and the two of them built an authorisation protocol for AI agents in my comments section over about a week. I want to write down what they built, because the thing it runs into at the bottom is not an engineering problem, and I do not think the agent-safety conversation has noticed.

What they built

@peterbuildssecure pushed the single-use idea further. A six-hour agent task is not one transaction. It is forty, or however many tool calls touch state, and each one has its own boundary. One mandate per operation, single use, burned by whoever performs the effect.

Then @anp2network found the hole. Some calls never resolve from the caller's side. A timeout that lands after the effect owner has already committed leaves you not knowing whether it happened, and treating that as failure lets the clock back in through the retry path.

@peterbuildssecure: fencing, not pending. On timeout the issuer writes a cancellation instead of waiting. But the tombstone has to live at the effect owner, because a delayed original still arrives there, and the issuer's records are not in the room when it does.

@anp2network: the cancellation write needs the same channel that just failed. So fold it into the replacement. M2 carries "supersedes M1", and the effect owner admits M2 and fences M1 in one commit.

@peterbuildssecure: and you do not keep the tombstone forever. Two tiers. A blocking record for as long as an honest late message could still turn up, then a cheap id marker after that.

That is a week of argument compressed into five paragraphs, and I have lost most of it in the compression. The thread is better than this summary.

The pattern nobody named

Read the sequence again and one thing repeats.

Every move deletes a store and creates one somewhere else.

The revision list rots, so use expiry. Expiry is a clock, so use single-use. Single-use needs a burn record. The burn record has to live at the effect owner. The effect owner cannot hold it forever, so split it in two and keep the cheap half.

The state never goes away. It gets smaller and it changes address.

And at the end of it there is a number. How long the blocking record lives.

Nobody in that thread could derive that number. Not for lack of ability. Twenty replies of extremely careful reasoning got to it and then stopped, because it is not the kind of thing reasoning produces.

Payments did not solve this either

This is the part I kept circling instead of answering.

Payments has the same number and does not compute it. It gets handed one. A card scheme's settlement window is a retention policy with a regulator attached, and the reason it works is not that the number is right. It is that the argument was ended by somebody with the authority to end it.

Both sides read the same rulebook, at the same revision, and neither one can move it afterwards and call it a clarification.

@anp2network pushed back when I said that, and fairly. You do not strictly need a regulator. An engineer-set window with a public revision history the effect owner cannot write to buys most of the same property.

Most of it. Publication makes a unilateral change visible. It does not make it expensive. A card scheme can throw a member out. A commit history cannot. That gap is the difference between a rule and a strongly worded preference, and plenty of systems run fine on the second one, provided everybody knows which one they are standing on.

Why this is not just a payments story

Every agent-safety mechanism I have read this year bottoms out in a number like this.

How long the audit trail stays queryable. How long an idempotency key blocks a replay. How long a revoked credential stays revoked in a cache. How stale a policy snapshot may be before a check refuses to run.

The mechanisms are good. Some are better designed than what payments was running on fifteen years ago. But the number underneath always shows up as a configuration default, and a configuration default is what you write when nobody has decided who owns the question.

If the owner is "the platform team", the number is whatever seems reasonable in the sprint where storage costs come up. That is not a dig at platform teams. It is what happens to any number with no counterparty on the other side of it.

Where the argument stops working

I do not have a clean ending, which is why this is a post and not a proposal.

The regulator analogy breaks as soon as you ask who the regulator would be. Payments got one because money moving in the wrong direction is legible to a state. An agent deleting the wrong S3 prefix is not, and I do not want an FCA for tool calls. I doubt anyone does.

The honest version is smaller than the analogy. Inside one company the caller and the effect owner are usually the same organisation, so the rulebook does not need a regulator. It needs a written-down owner and a change process that is not a pull request approved in forty seconds.

That is boring. It is also roughly what payments ran on before it had regulators, and it held for a while.

The question

Go and find the retention number in your own system.

The TTL on your idempotency keys. The window your dedupe cache actually covers. The age at which outbox rows get vacuumed. The lifetime of a revocation entry.

You have one. Somebody typed it.

Who was it, what did they know when they typed it, and what happens to them if it turns out to be too short?

If the answer to the last one is "nothing", you have a mechanism rather than a rule. Worth knowing that before an agent finds out on your behalf.

The thinking here is @anp2network's and @peterbuildssecure's, not mine. Thanks also to @salparvez, whose roof-and-foundation version of the two-tier idea is the one I actually remember, and @_firelinks, who showed me that a negative control run inside the thing it is testing is circular in exactly the state you built it to catch.

This is the third article I have written out of my own comments section. At some point that stops being a content strategy and starts being a confession.

Original thread: Nobody Checks Whether the Guardrail Is Running

Four People Rebuilt the Payment Authorisation in My Comments Section

arun rajkumar — Thu, 10 Sep 2026 05:22:57 +0000

I published two articles three weeks apart. They were not about the same thing.

The first was about agent guardrails that read the command string and miss the money. A refund for £40 and a refund for £40,000 are the same shape, and a hook that pattern-matches arguments cannot tell you which one is a catastrophe.

The second was about guardrails nobody checks the liveness of. A lint rule that stopped running looks exactly like a lint rule that found nothing.

Different subjects. Different threads. As far as I can tell, no overlap at all in who turned up to argue.

Both threads ended up building the same object. Nobody involved said the word "payments".

What the first thread built

Three weeks ago @peterbuildssecure left this on the guardrails post:

Preview resolves the proposed effect and returns an operation ID plus the subject, target, amount, currency, relevant state version and expiry. Human approval produces a single-use mandate bound to that exact operation. Execute consumes it atomically while rechecking the authoritative state and idempotency key.

Then the part I think is the most important sentence in either thread:

the server should require the mandate whether or not the client noticed that metadata. Otherwise the convention protects careful clients while direct callers retain the bypass.

What the second thread built

Yesterday, on a completely different post, @anp2network argued that you should stop trying to carry a reachability guarantee across a process boundary, because it cannot survive the trip:

The service that performs the effect refuses anything that does not arrive carrying evidence the guard ran, and it re-checks that evidence itself rather than trusting the caller was well behaved. Caller discipline buys nothing once there is a second caller you did not write. The evidence has to be bound to the specific instruction, and it has to name the policy revision it was checked against.

Read those two next to each other.

Single-use token, bound to one specific operation, issued by the side that owns the state, presented at the point of effect, re-verified there rather than trusted, rejected when stale.

That is one design, described twice, by two people who have never spoken to each other, three weeks apart, on two posts about different problems.

It is also a payment authorisation. It has been a payment authorisation since roughly 1979.

The other two got further than I did

@jon_at_backboardio went straight past the single-operation model to its weak point:

five refunds of £8,000 each, every one under every cap, same payee, four minutes. every individual call is boring. the mandate model catches that only if mandates are scoped to a window rather than to an operation

He also settled the sync-versus-async argument in one observation: the calls where blocking hurts are high-volume and low-value, and the calls where blocking is fine are the ones you want to stop. Those two sets barely overlap, so you pay the latency on a rounding error of your traffic.

And @max_quimby, who started the whole thing, put his finger on why none of this is really an AI problem:

None of that is agent-specific, which I think is the point — the agent just removes the human who used to eyeball the number.

Payments did not design this. It lost money until it existed.

Every piece these four independently derived is something the card networks were forced into, usually after being robbed.

Bound to the instruction. An authorisation is for an amount and a payee. You cannot get one for £40 and present it for £4,000. That is not elegance, it is scar tissue from people doing exactly that.

Verified by the side holding the state. The merchant does not decide whether an authorisation is good. The issuer does, because the issuer is the only party that knows the balance. Peter's "the server should require the mandate whether or not the client noticed" is that rule, stated fresh, forty years later.

Single-use. Replay is the oldest attack there is.

Expiry. This one gets skipped in every clean-room design and it is load-bearing. Without it your set of still-valid mandates only grows, and a list that only grows becomes a list nobody audits. Expiry is what keeps that set small enough to reason about. You trade a list that rots quietly for a clock that fails loudly, and the clock is much easier to think about at three in the morning.

Three more things the threads are about to discover

Since payments got here first and paid for it, here is what is waiting further down the road.

Velocity, not caps. Jon's five-refunds case is not an edge case, it is the standard attack, and per-transaction limits have never caught it. What catches it is cumulative exposure per counterparty per window. The check is easy. Choosing the window is not, and whatever window you choose, somebody can straddle it. Four minutes clears an hourly window if you start at 59 minutes past.

Preview is a surface too. Auth-then-capture has a gap between the two halves, and people have lived in that gap professionally for decades. If preview simulates the effect and execute performs it, any divergence between those two code paths is somewhere a difference can hide. Much smaller than the hole it closes. Not zero.

Approval screens launder decisions. Jon again, and it is the best line anyone has left on anything I have written: showing a human raw arguments is not review, it is laundering. If the human cannot see what the number means relative to the account it is hitting, you have not added a control. You have added a signature to blame later.

The one part that does not transfer

I do not want to oversell the analogy, because there is a real gap in it.

A payment has a natural boundary. There is a moment the transaction starts and a moment it settles, and every lifetime in the system hangs off those two points. Expiry is easy to reason about because there is an obvious thing for it to be shorter than.

An agent halfway through a long task has no such boundary. The task might run for six hours. It might spawn sub-tasks. It might pause overnight and resume. Scope the mandate to the operation and Jon's five-refund case walks straight through. Scope it to the session and you have issued a bearer token for a session whose end you cannot define.

So: what is the agent equivalent of a transaction boundary?

I do not have an answer. Payments handed us the shape of the solution for free, and then kept the one thing that made it work.

Quoted with thanks and without permission: @peterbuildssecure, @anp2network, @jon_at_backboardio and @max_quimby. The threads are here and here, and both are worth more than the posts they are attached to.

Your Audit Log Agrees With Itself. That's the Problem.

arun rajkumar — Wed, 09 Sep 2026 10:15:14 +0000

@remdore ran an experiment this week that I have not stopped thinking about.

One account, balance 100. Ten workers, each withdrawing 10. Every worker does the obvious thing: read the balance, subtract, write it back, log what happened.

Final balance: 90. Nine withdrawals gone.

That part is a race condition and every backend engineer has met it. The part I want to talk about is the audit log.

Ten rows. Every one of them says the balance went from 100 to 90. No gaps. No nulls. No orphaned foreign keys. Nothing a schema validator, a checksum, or a nightly reconciliation job would flag.

Hand me that table during an incident and I would read it as one withdrawal retried ten times. I would go and look at the retry logic. The retry logic is fine. I would spend the afternoon there.

The log is not wrong. That's what makes it dangerous.

We are in regulated payments. The audit log is not a debugging convenience for us. It is the artefact. It is what an auditor reads, what a dispute resolves against, what we would put in front of the FCA if someone asked how a payment reached the state it did.

So the standard we hold it to is not "does it look sensible". It is "is this the record of what happened".

And the failure in that experiment passes every test I know how to write against a log.

Think about what you can actually check automatically:

Schema conformance. Passes.
No nulls in required columns. Passes.
Referential integrity to the account. Passes.
Monotonic timestamps. Passes.
Balance arithmetic internally consistent per row. Passes, ten times.
Row count matches the number of operations attempted. Passes. Ten operations, ten rows.

Every one of those questions is about whether the log is coherent. None of them is about whether the log is true. Ten processes each believed they were alone, and each one wrote down a faithful account of the world as it saw it. Coherence is exactly what you get when ten honest narrators all have the same wrong information.

The thing you would need to detect it is a row that disagrees with another row. There isn't one. Disagreement is the signal, and the bug removed the signal.

This is the same failure I wrote about two days ago

I published something on Monday about guardrails nobody checks the liveness of. A lint rule that stopped running, a policy check that matches nothing, an eval that skips half its corpus and reports the half it ran. The thread on it got long, and the argument that came out of it was sharper than the article.

The shape is the same here.

A check that never fires and a log that never contradicts itself both produce the same reading: nothing is wrong. In both cases the absence of a complaint is being taken as evidence of health, and in both cases the mechanism that would produce the complaint is the thing that broke.

You cannot fix that by inspecting the output more carefully. The output is clean. That is the whole point.

The one-statement version, and why it matters more than it looks

Postgres 18 lets you name the row before and after a change in the same statement that makes it:

UPDATE accounts SET balance = balance - 10 WHERE id = 1
RETURNING old.balance AS was, new.balance AS now;

The syntax is nice. It is not the interesting part.

The interesting part is that the log entry is now derived from the write instead of being a second write that describes the first. Before this, the audit row was a separate statement carrying a separate copy of the truth, and two copies of the truth can drift. After it, there is one statement, and the record and the change are the same event. There is no version of the run where the balance moves and the log says something else, because there is no seam between them for a lie to get into.

That is the general principle and it is worth more than the syntax. Any time the record and the thing being recorded are two separate writes, you have two versions of the truth, and only one of them is the one that moves money.

You can apply that without Postgres 18. The change is to stop thinking of the audit log as something you write about an operation and start treating it as something the operation emits. If a code path can complete without producing its record, the record is optional, and optional loses to deadline pressure every time.

Where this stops working

I want to be honest about the limit, because the single-statement version solves a single-database problem and most of us do not have a single-database problem.

The moment the operation crosses a service boundary, the seam is back. The money moves in one place and the record is written in another, and now you are relying on an outbox, or a queue with delivery guarantees you have read about but not tested, or in the worst case a developer remembering to call the logging function. Convention again.

The only thing I have found that helps at that scale is narrowing the number of routes. Anything touching money goes through one path with the record emitted inside it, so there is no version of the call that reaches the effect without producing the evidence. Everything outside that path is convention, and I am not comfortable about it, but I would rather have one path I can defend than six I have to trust.

The question I would actually like answered

Here is what I do not have a good answer for.

Every detection method I know for this class of bug is a disagreement detector. Two sources that should match and don't. A count that should equal another count. A hash that should still verify. All of them need two things to compare.

The read-modify-write bug produces one thing, ten times. There is nothing to compare it against, because the second source was never independent.

So: has anyone got a check that catches a log which is internally consistent and collectively false, without needing an independent second record to diff against? Not a stronger schema. Not more constraints on the row. Something that notices that ten narrators agreeing is itself suspicious.

I think the honest answer might be that there isn't one, and the only real defence is making the seam impossible rather than detecting what crawls through it. But I would like to be wrong about that, because "make the seam impossible" is expensive and does not retrofit.

Credit where it is due: the experiment and the Postgres 18 walkthrough are @remdore's. Go and read the original. I have only taken the part that scared me and pointed it at my own industry.

Nobody Checks Whether the Guardrail Is Running

arun rajkumar — Mon, 07 Sep 2026 14:07:25 +0000

Every AI-and-engineering post right now is about adding a guardrail.

Lint rules the agent can't bypass. Evals before you ship a prompt change. A review bot on every PR. A regression suite built from real production failures. Architecture rules checked into the repo so the model reads them on the way in.

I have written some of those posts. I still believe in them. This is the work that makes AI usable by senior engineers instead of a liability.

But I have been in three separate conversations on this platform in the last week that were all, underneath, about the same thing, and it isn't about which guardrails to add.

A guardrail that has never fired and a guardrail that silently stopped running produce identical output.

Green.

The pipeline that had never been green

Vicente Reyes had a GitHub Actions workflow called Deploy to DigitalOcean. Fully wired. SSH action, secrets, the works. And every time he shipped a backend change he still SSH'd into the droplet and ran git pull by hand.

The deploy job was gated on CI passing.

on:
  workflow_run:
    workflows: ['CI']
    branches: ['main']
    types: [completed]

jobs:
  deploy:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

Sensible enough. The problem was that CI had never once gone green on main. Not flaky. Never. So every deploy run showed skipped, forever, and the pipeline sat permanently behind a gate that could not open.

His line about it is the one worth keeping. A pipeline that is silently and permanently blocked looks, from a distance, exactly like a pipeline that doesn't exist.

Nothing in the Actions UI says this workflow has not succeeded in forty runs. You have to go and ask.

The grader that can't say no

On a thread about designing trustworthy AI evals, Heinrich Neb made the point that every grader needs a known-bad twin. An input it is supposed to reject, plus a recorded date of when it last actually rejected something.

His framing is the sharpest version of this I have seen: a grader that has never failed and a grader that silently stopped running print the same green.

Same failure as Vicente's pipeline, one layer up. In his case the gate was stuck closed. In an eval suite the gate is stuck open, which is worse, because a stuck-closed gate is annoying enough that somebody eventually investigates. A stuck-open gate just keeps saying yes.

Think about how an eval suite actually rots. Somebody changes a prompt template and the grader's regex stops matching, so everything scores as pass. Somebody renames a dataset field, the loader returns an empty list, the suite runs zero cases in 0.4 seconds and reports 100%. A provider changes a default and your grader model gets more agreeable.

All three look like success.

The score with no provenance

The third one is mine, from the same thread.

An eval result with no harness version, no dataset snapshot and no prompt revision attached to it is not evidence. It is a self-reported claim.

Which is fine right up until the number moves. Then somebody asks whether the model got better or the suite got easier, and if you can't answer, you never had a measurement. You had a vibe with a decimal point on it.

That reflex comes from working in payments. In a regulated system nobody asks you to trust that a control ran. They ask you to demonstrate which control ran, on what input, at what time, under which version of the rules. Months later. To somebody who wasn't there and isn't inclined to take your word for it.

Engineering has quietly inherited that burden. Evals are increasingly the artefact a shipping decision rests on. They just haven't inherited the paperwork.

What actually ties these together

When you automate a check, you swap one question for another and don't notice.

Before automation the question is did someone look at this? You know the answer, because you can see the person and ask them.

After automation the question you think you are asking is still did the check pass? But the question you are now depending on is is the check alive?

Almost nobody instruments the second one.

This is well understood in operations. You don't only alert on errors, you alert on the absence of a heartbeat, because a monitoring system that dies looks exactly like a system with no problems. Dead man's switches exist for precisely this reason.

We have somehow not carried it across to the checks that gate our code. A CI workflow, a lint rule, an eval suite, an agent policy. These are all monitoring systems for correctness, and we run them with no heartbeat at all.

What a guardrail needs before you trust it

None of this is exotic.

A known-bad input it must reject. Every check needs a case it is supposed to fail on, running alongside the real ones. If your lint rule can't catch its own canary, the lint rule isn't running. If your eval's negative control scores as a pass, the grader is broken and every other number in that run is noise. Same idea as a negative control in a lab. Nobody trusts an assay that only ever comes back clean.

A recorded date of last rejection. Not when it last ran. When it last said no. A guardrail that hasn't rejected anything in four months is either protecting an unusually disciplined team or it broke in May, and those look identical on a dashboard. Put it in a column somewhere. If nobody can answer "when did this last catch something", it is decoration.

A run count somebody occasionally looks at. The zero-cases failure is the sneakiest one, because a suite that loads an empty dataset passes fast with a perfect score. Assert on the count. If it expects 240 cases and got 0, that is a hard failure, not a 100%.

Provenance on the result. Harness commit, dataset hash, prompt revision, model version, timestamp. Attached to the score, not sitting in a CI log with thirty-day retention. The test is simple. Six months from now, can you reproduce this exact number? If not, you can't use it to defend a decision, which means it was never really the reason for the decision.

Why this gets worse with agents, not better

The reason I keep coming back to this is that AI shifts the ratio.

The argument for agents in engineering, the one I actually believe, is that you get something close to an army of near-zero-mistake juniors. The constraint stops being how fast people can write code and becomes how fast people can review it. So you compensate by pushing more review into automation. More lint rules, more tests, more evals, more policy checks.

That is the right move. I would make it again.

But it means the fraction of your correctness resting on unattended machinery goes up sharply. When a human reviewed everything, a broken lint rule was a small hole in a large net. When automation reviews everything, the broken lint rule is the net.

The guardrails become load-bearing at exactly the moment nobody is watching them closely enough to notice they stopped.

Go and check one

Pick the guardrail you would be most upset to lose. The eval suite that gates prompt changes, the rule that stops an agent touching the payment path, whatever yours is.

Then answer one question about it. When did it last say no?

If you can find that out in under a minute, good. If you can't find it at all, you don't have a guardrail. You have a green light with nothing behind it, and you have been treating it as evidence.

Credit where it's due. The "never failed and stopped running print the same green" framing is Heinrich Neb's, and the permanently-gated pipeline is Vicente G. Reyes'. I just noticed they were the same bug.

I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about AI, payments, and the messy parts of running engineering systems. @mickyarun.

Your Health Check Is Lying to You

arun rajkumar — Fri, 04 Sep 2026 12:30:27 +0000

Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it.

The last act is the one I want to pick up.

Two minutes after his first fully automated deploy, the site returned a 502. Nothing had crashed. The old container was gone, the new one hadn't finished booting, and he happened to refresh inside a window about one second wide.

That window is in almost every Docker Compose deploy I have seen. It is in mine.

The interesting part isn't the window. It's that the obvious fix only closes half of it, and the other half fails much more quietly.

The proxy has no idea

A static Traefik route looks like this:

http:
  services:
    api:
      loadBalancer:
        servers:
          - url: http://api:3000

That config says: send traffic to api:3000. It does not say: send traffic to api:3000 when something is actually listening there. Traefik has no opinion on the matter. It resolves the name, opens a connection, reports whatever comes back.

So every time docker compose up -d stops the old container and starts the new one, there is a stretch where the proxy is confidently routing at a process that is still running migrations. Nobody hits the app during that stretch, you never find out.

The fix everyone reaches for is an active health check.

http:
  services:
    api:
      loadBalancer:
        servers:
          - url: http://api:3000
        healthCheck:
          path: /healthz
          hostname: api.example.com
          interval: "5s"
          timeout: "3s"

Correct, and it's where most write-ups stop.

The hostname line is not optional

Leave hostname out and Traefik sends the check with Host: api, the internal Docker service name. Anything with a host allowlist in front of it will reject that. Django gives you a DisallowedHost 400. Express with a host check does the same. So does NestJS behind most gateway setups.

From Traefik's side a 400 is a failed check, and a container that fails every check is permanently unhealthy.

So you spend an afternoon debugging why your app is down. Your app is fine. The health check is asking the question in a form the app is contractually obliged to refuse.

Set hostname to the real public domain. Then the check looks like real traffic, which is the whole point of it.

Liveness and readiness are different questions

The advice attached to that config is usually "keep /healthz trivial". No database, no auth, just return 200. That advice is right, and for a good reason: if your health endpoint touches a business dependency, a permissions change or one slow query quietly pulls the container out of rotation.

But a trivial endpoint answers a trivial question. It answers is this process alive. It does not answer can this process serve a request.

Those two come apart constantly. A container that boots cleanly but can't reach its database will return 200 from a trivial check all day. Pool empty. Queue consumer never attached. The proxy sees green, sends real traffic, and instead of one second of clean 502s you get sustained 500s from a container that is reporting healthy.

So split them. Liveness stays trivial and wires to your restart policy. Readiness checks what the container needs to do its job and wires to the load balancer. Collapse the two and you pick one failure mode to catch and get blindsided by the other.

The part nobody writes down

This is the bit I actually sat down to write.

The natural readiness check for a service that depends on Postgres is a SELECT 1. Cheap, honest, and it's what I'd reach for too.

Think about what it proves. It proves the pool can hand you a connection. It says nothing about whether the pool is nearly empty.

Under load those diverge, and the divergence is ugly. If readiness borrows from the same pool the application serves from, then when that pool is close to exhausted the check either succeeds by taking the last free connection, which is the moment you least wanted a health probe competing with real requests, or it fails outright because there was nothing left to take.

The second one is what hurts.

A container fails readiness. The load balancer pulls it. Its traffic redistributes to its neighbours. Those neighbours now carry more load, their pools saturate, they fail readiness too, and out they go.

You have built a check whose entire job is to protect the system, and under exactly the conditions it was meant to protect against, it removes the system.

Three cheap ways out, pick one:

Give readiness a dedicated connection outside the application pool. Then a saturated pool is a slow app rather than an unhealthy one.

Or keep it on the shared pool and be honest about what it asserts, which is "Postgres is reachable from this container". Fine thing to check, as long as nobody downstream believes it means "has capacity".

Either way, never let a single failed probe evict. Require consecutive failures over a window so a momentary squeeze can't cascade.

The general principle is that a readiness check consuming the same scarce resource it measures will always give you its worst answer at the worst possible moment.

Readiness is a dependency graph you're signing up for

Everything you put in a readiness check becomes something that can take you out of rotation.

Put a downstream HTTP call in there and you have coupled your availability to somebody else's. They have a bad ten minutes, every one of your containers marks itself unready simultaneously, and now you have zero healthy instances of a service that was working perfectly well.

So the scoping rule is narrow. Check only what this container needs to serve its own traffic.

Vicente landed on exactly this for his stack. Postgres in, Redis and Celery out, on the grounds that background jobs degrading is not the same thing as the web tier being unable to answer. Redis going down should page someone. It should not empty the load balancer.

That's a design decision, not a config line, and it's worth the ten minutes.

Start-first doesn't get you to zero either

Say you fix all of that. The window shrinks from "boot plus migrations plus static assets" down to "process fork time".

It shrinks. It does not close. docker compose up -d stops the old container before the new one is ready, and no health check changes that ordering.

The usual next step is a start-first rollout. Bring up the new container, wait for it to pass readiness, kill the old one. The docker-rollout plugin does this, Swarm and Kubernetes do it natively.

The bit that gets skipped: start-first closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight.

Two things have to be true for that second window to be harmless. The proxy has to notice the old server left and stop sending it new work before the container dies, and Traefik's config reload is not instant. And the application has to be allowed to finish what it is already holding. In gunicorn that's graceful-timeout. In Node it's whatever you wired into your SIGTERM handler, and if you wired nothing, the answer is that it doesn't.

A rollout without a drain window doesn't eliminate dropped requests. It moves them from the front of the deploy to the back, where they are harder to spot because the deploy reports success.

If your app is anywhere near money that distinction matters a lot. A request dropped at the front is a failed connection and the client retries. A request dropped at the back was accepted, was being processed, and then wasn't.

What to actually do

The answer is different at each size, so:

Single box, Compose, low traffic. Split liveness and readiness, scope readiness to your own datastore, set hostname, live with a health-gated couple of seconds. Completely reasonable place to stop. Reaching for an orchestrator here costs you more than the window does.

Multiple instances behind a proxy. All of the above, plus a failure threshold so one bad probe can't evict, plus readiness on its own connection. This is where the cascade becomes possible, so this is where it's worth paying for.

Deploys that cost you money when they drop requests. Start-first rollout with an explicit drain. Proxy deregistration before SIGTERM, graceful timeout longer than your slowest normal request.

The tools matter less than the question.

Every health check is an assertion, and the failure is always the same one. The assertion is narrower than what everyone downstream believes it to be. Somebody reads a trivial 200 as "can serve". Somebody reads a SELECT 1 as "has capacity". Somebody reads a green deploy as "nobody got a 500".

Write down what your check actually proves. Then go and find out who is relying on it to prove more than that.

Thanks to Vicente G. Reyes for the original write-up and for a good argument in the comments. Most of this is that argument.

I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about the messy parts of running engineering systems. @mickyarun.

Your Agent's Guardrails Can't See the Money

arun rajkumar — Fri, 21 Aug 2026 10:53:11 +0000

There's a post going round this week about agent guardrails that opens with a good story. The author's agent wanted to force-push to main. Not because it was confused. The rebase was stuck, force-pushing would unstick it, and every step in that chain of reasoning was sound. Locally correct, non-locally expensive.

What makes that example teachable is that the command carries its own consequence. git push --force origin main has the danger written into it. You can pattern-match it. You can put it on a list. A hook can read the string and stop.

Most of what I worry about doesn't look like that at all.

Two calls, one of them a disaster

Here are two refund calls.

await payments.refund({
  paymentId: "pay_9f2c14",
  amount: 4000,
  currency: "GBP",
  reason: "customer_request",
});

await payments.refund({
  paymentId: "pay_9f2c14",
  amount: 4000000,
  currency: "GBP",
  reason: "customer_request",
});

One is forty pounds. The other is forty thousand. Same endpoint, same argument names, same shape, same reason code, same everything a pre-execution hook can read off the text.

Now the part that actually matters: neither of them is inherently wrong.

Forty thousand might be a perfectly good refund against a perfectly good invoice. Forty might be a refund on a payment that was already refunded an hour ago, which in some ways is the worse of the two. You cannot sort these by looking at them, because the thing that makes one of them a mistake isn't in the call.

Blast radius is a property of the target, not the command

The force-push case works because the danger lives in the verb. Force-push is dangerous in nearly every context. The set of situations where you genuinely want it is small enough to enumerate, so a rule can cover it and the rule stays roughly true.

An amount-bearing API call is dangerous as a property of the state on the other side. Whether that refund is safe depends on things the caller simply doesn't have:

Has this payment already been refunded, fully or partially?
Does the merchant's balance cover it, or does this pull them negative?
How much has this actor already moved today?
Is the original payment under dispute, where a refund does something quite different to who ends up liable?
Is pay_9f2c14 even this merchant's payment?

A hook sitting in the agent process knows none of this. It can read intent. Cost is not a property of intent.

"Fine, have the hook go and check"

This is the obvious next move and it isn't stupid. Let the guard call the API, fetch the payment, look at the balance, then decide.

Try building it and you find out what you signed up for.

Your hook now needs credentials to read payment state, which means the agent host is holding read access to your ledger in order to protect you from the agent host. It needs to understand refund semantics, dispute states and settlement timing well enough to form a judgement, so your refund rules now live in two codebases that have to agree forever. And it has a window between checking and executing, which is exactly where the interesting failures live. The balance was fine when you looked. Something else landed. Yours goes through anyway.

You haven't built a guardrail. You've built a second, worse copy of your authorisation service, running in the least trusted process you own, with a cache.

Put the check where the state is

The version that survives contact is boring. The check goes at the rail, at the point of authorisation, because that's the only place that knows what this actor may move, what it has already moved, and what the target looks like right now.

We're regulated, so we already had to have that spine. Every movement of money goes through an authorisation step with the full picture and an audit record on the other side. Wiring agents into it didn't mean building something new. It mostly meant resisting the urge to build something in front of it and call that safety.

The agent-side hook still has a job. Just a smaller one than people want to give it.

// The hook classifies and asks. It does not decide.
async function preToolUse(call) {
  if (!MOVES_MONEY.has(call.tool)) return allow(call);

  // Ask the system that owns the state what this would actually cost.
  const effect = await rail.preflight(call.args);

  return confirmWithHuman({ intent: call.args, effect });
}

rail.preflight is where the work happens, and it lives on the side that has the ledger. Its whole purpose is so a human sees "this refunds £40,000 against an invoice already refunded in full on the 3rd" rather than "the agent would like to call refund." One of those is a decision. The other is a rubber stamp with extra steps.

It's also explicitly advisory. Between preflight and execute the world moves. Enforcement stays server side, under a mandate scoped to an amount, a payee and a clock. If preflight and the rail ever disagree, the rail wins.

I wrote more about mandates versus API keys in an earlier post if that thread interests you.

The uncomfortable bit

Your agent framework cannot own your safety story for anything that moves money, and it doesn't matter how good its hooks get.

That's not a knock on the hooks. Command-level guards are useful and I'd rather have them than not. For destructive verbs, where the danger is in the word, they're the right tool and they'll catch real mistakes that would otherwise cost someone a weekend.

They just sit at the wrong altitude for a whole class of call that reads as completely unremarkable and is defined entirely by context the caller doesn't hold.

So here's the test I'd apply to any guard before trusting it. Could this exact call, byte for byte, be both correct and catastrophic depending on something the guard cannot see? If the answer is yes, that guard is a linter. Useful. Not load-bearing. Put the real check where the state lives.

Has anyone landed on a decent convention for a tool declaring "you can't infer my blast radius from my arguments, come and ask"? I haven't seen it in the MCP spec, and it feels like the piece that's missing.

Your MCP Eval Checklist Has an Auth Row. In Payments It's the Whole Table.

arun rajkumar — Mon, 10 Aug 2026 14:16:19 +0000

There's a good checklist going around dev.to for vetting an MCP server before you wire it into an agent. Four things to look at. Tool surface area: how many tools, and are they atomic or coarse. Auth model: API key, OAuth, token scope. Maintenance: last commit, open issues, is anyone home. Token profile: does it dump a full document when a summary would do.

It's a genuinely good checklist. I've used a version of it. For most tooling categories it's exactly the right lens.

Then you point an agent at something that moves money, and three of those four rows go quiet.

Not because they stop mattering. Because one of them grows until it's the only thing you're really deciding.

A read tool and a write tool are not the same animal

Here's the thing that took us a while to say out loud.

If an agent calls a code-search tool twice, you get the same answer twice and waste a few tokens. If it reads a git diff twice, nobody notices. Reads are safe to repeat. That's the whole reason retries are the default everywhere in the agent stack. A tool call times out, the client tries again, you move on.

A payment tool call is not a read. Retry it once and you've billed someone twice.

We build open banking payments. The failure that actually keeps me up isn't a hallucinated argument or a server returning a forged result. It's the boring one. The connection blips mid-checkout, the client does what clients do and retries, and now there are two payment intents where the user meant one.

So the eval question for a money-moving tool isn't "what's the auth model." It's "what happens on the second call I didn't mean to make."

Idempotency lives on the intent, not the turn

The fix is old and unglamorous. Idempotency keys. Everyone in payments already knows them. The part people get wrong with agents is where the key lives.

The instinct is to make the agent turn idempotent. Same prompt, same result. That's the wrong seam. The agent turn is fuzzy by design, and you don't want it to be the thing carrying the guarantee.

Put the key on the payment intent. The client generates it once, before the tool is ever called, and it travels with the money, not with the conversation.

// The key is minted where the intent is born, not inside the agent loop.
const intent = {
  idempotencyKey: crypto.randomUUID(), // one per real-world payment
  amount: 4200,
  currency: "GBP",
  payee: "merchant_8842",
};

// Retries of the SAME intent collapse to one charge.
// A genuinely new payment gets a new key, on purpose.
await paymentsTool.charge(intent);

Now a dropped connection is harmless. The retry carries the same key, the server recognises it, and the second call returns the first result instead of moving money again. The agent can be as jittery as it likes. The guarantee sits below it, where the stakes are.

Read paths open, write paths gated

The other move is to stop treating "tools" as one category.

On the read side we let the agent run. Fetch balances, list transactions, pull an account's status, look up a payout. If it over-calls, it wastes tokens and we tune it later. Low blast radius, no gate.

On the write side, anything that changes state or moves money goes behind a human confirmation. Not the agent confirming to itself. A person, or a service acting under an explicit, narrow mandate, in the loop before the call executes.

If you're on Claude Code or a similar setup, the cheap version of this is a pre-call hook that classifies the tool and decides whether it needs a gate.

// Classify by side effect, not by name.
const NON_RETRYABLE = new Set(["charge", "refund", "payout", "mandate.create"]);

function preToolUse(call) {
  if (NON_RETRYABLE.has(call.tool)) {
    return requireHumanApproval(call); // blocks until a person says yes
  }
  return allow(call); // reads sail through
}

The point isn't the code. It's the split. Reads and writes want different defaults, and a checklist that scores a server as one thing misses that the same server can hold both.

The credential is a mandate, not a key

The auth row on the checklist usually asks whether it's an API key or OAuth. Fine question. Wrong altitude for payments.

What you actually want to hand an agent is a mandate. Scoped to an amount and a payee. Time-boxed, so it expires whether or not anyone remembers to revoke it. Revocable mid-flight. And auditable after the fact, so when someone asks "why did this money move" there's a straight answer that doesn't depend on trusting the model.

A key says "this caller is allowed." A mandate says "this caller is allowed to move this much, to this party, until this time, and here's the record." The second one is the only thing I'd let near a live payment rail.

We already keep that audit spine for money movement, because we're regulated and there's no version of this job where you don't. The work with agents wasn't inventing it. It was extending the same discipline to tool calls, so an agent's action leaves the same trail a human's would.

So, the checklist

Keep all four rows. For a knowledge base or a code-search server, run the standard lens and move on.

But the moment a tool can move money, promote one question above the rest and answer it first: is this call retryable, and if it isn't, what stops the second one? Everything else on the checklist is downstream of that.

A read tool can be replayed all day. A payment tool replayed once bills a real person real money.

Are you seeing any of the community MCP servers treat retryable and non-retryable tools as different classes yet, or is that still left entirely to whoever's calling them?

Gartner Says 40% of Apps Will Have AI Agents by December. Here's the Plumbing Nobody Puts on the Slide.

arun rajkumar — Mon, 03 Aug 2026 08:09:14 +0000

There's a number going around dev.to this week. Gartner says 40% of enterprise apps will ship a task-specific AI agent by the end of 2026. Last year it was under 5%.

Every deck quotes it. Every thread argues about it. Fine.

I run engineering at a UK payments company. We're FCA-authorised, SOC2, the whole regulated stack. On the side I build an open-source agent framework called Bodhiorchard, where a dozen agents do real work on a real codebase. So I've shipped the thing the slide is describing. And I can tell you the 40% isn't the hard part.

The hard part is everything under the slide. And the biggest piece of it is money.

The demo is not the deployment

An agent demo is easy. You give it a prompt, it writes some code or drafts a report, everyone claps.

Then you try to put it near a live system and the questions start. What can this thing actually call? What happens when it's confidently wrong? Who gets paged?

None of that shows up in a projection. All of it shows up in production.

The real boundary isn't "no money." It's a signed mandate.

The reflex guardrail is to forbid the scary thing. Don't let the agent move money. Read-only, propose-only, a human clicks the button.

That reflex is already history. Agents are going to move money, because paying is half of what it means to finish a task. An agent that can research a supplier, compare options and fill a cart, then stops and waits for a human to tap "confirm," isn't an agent. It's an intern with a shopping list.

This is the part of the 40% that actually rewires the economy. Not agents writing code. Agents that transact. The moment an agent can pay, it stops being an assistant and becomes an economic actor, and the rails for that are being built right now.

That's what we're building at Atoa: a regulated bank rail for AI agents. An agent settles a payment bank to bank, over an FCA-authorised rail, with a signed record on every move. Built for the protocols this is standardising on, AP2 for mandates, x402 for pay-per-request, MCP as the interface. So "never let it touch money" was never going to be our answer. The answer is where the authority lives.

An agent's permission to spend can't be a line in a system prompt that a clever input talks its way around. It has to be a signed, scoped mandate: explicit about how much, to whom, within what limits, and verifiable on its own. Every payment gets a real-time affordability check. Every payment leaves a signed, provable record, who authorised it, what the funds check returned, where it settled.

That's the shift worth internalising. The boundary moved from "can the agent act?" to "is the agent's authority explicit, scoped, and provable?" Money moving for an autonomous agent needs more oversight, not less. So you make the authority a hard artifact and the audit trail non-optional, and then you let it pay.

Context in, not hope in

Here's a real one from Bodhiorchard.

An agent was asked to produce a payout report. Left to its own reading of the task, it started building a brand new service to generate that report. The report already existed. It was about to rebuild something we already had, in a slightly different shape, as new surface area to maintain.

The fix wasn't a smarter prompt. It was context. We feed agents structured context through MCP before they write a line, what we call a BUD in Bodhiorchard. Once the agent could see the existing report and the decisions behind it, it did the sane thing. It extended what was there instead of spawning a duplicate.

That's the difference between an agent that helps and one that quietly grows your tech debt. Not intelligence. Context.

Deterministic checks come before the probabilistic step

An agent's output is a guess. A good guess, often. Still a guess.

So the guess doesn't get to be the last word. Before anything an agent produces goes near a real code path, it runs the checks a human would have to pass. Types. Schema validation. The test suite. The design-pattern lints that encode decisions no off-the-shelf linter ships with. In the payments flow, the affordability check plays the same role: a deterministic gate the probabilistic step has to clear before anything settles.

If that deterministic layer isn't there first, you haven't deployed an agent. You've deployed a very fast intern with commit access and no code review.

Someone owns the failure

This is the part nobody wants on the slide, because it's a headcount question, not a technology one.

When an agent fails, it usually doesn't crash. It fails plausibly. The report looks right. The code compiles. The number is just wrong. That kind of failure needs a human owner who knows the domain well enough to smell it, and a record clean enough to trace it back.

My mental model: an agent is an army of near-zero-mistake juniors. That's a gift to a senior engineer and a trap for a team without one. Enabling seniors with agents is the right move. Replacing seniors with agents is how you find out what plausible failure costs.

So

Yes, 40% of apps will probably have an agent by December. The slide will be right.

But the agent isn't the work. The scoped mandate, the context feed, the deterministic gate, the human who owns the failure. That's the work. That's the 60% under the waterline.

And the biggest piece of that iceberg is payments. The agent economy doesn't start when models get smarter. It starts when agents can pay, safely, over rails that were built for them. That's not a 2030 story. It's live now, and we're building one of the rails.

If you're shipping an agent this year, which of those four do you already have, and which are you hoping the model handles for you?

AI Agents Ship Bugs When They're Blind. So We Stopped Building Blind.

arun rajkumar — Fri, 31 Jul 2026 06:51:09 +0000

Most AI coding mistakes don't look like mistakes.

They compile. The tests pass. CI is green. The diff reads fine at 6pm on a Friday. Sometimes it's a bug that surfaces two weeks later when money moves where it shouldn't. And sometimes it's not a bug at all — the agent quietly rebuilds something you already had, and you don't notice until you're maintaining two versions of the same thing.

Same root cause both times. The agent couldn't see. It had my prompt. It didn't have the codebase.

I'm the CTO of a payments company. We're FCA-authorised, we move real money for real merchants, and we've leaned on AI agents hard for over a year. So I've watched this happen enough to stop blaming the model for it. Let me show you exactly what I mean, because it happened to me last week.

The report that didn't need to exist

We were adding multi-business support to payouts. One parent account, several businesses underneath it, each needing its own slice of the flow.

The agent did the core work well. I want to be clear about that. On the multi-business payout logic itself, the code was correct. No complaints.

Then I asked for a small follow-on: add callback parameters and a few dynamic fields into the payout report.

The agent went and built a new reporting service. New repo, fresh scaffolding, a whole new report pipeline. It worked. It compiled, it ran, it produced exactly the report I'd asked for.

One problem. We already had a payout report — the single-business one, live in production, tested against real settlements for months. The right move was to extend it to handle multiple businesses and the new fields. Instead I now had two report implementations for the same domain. One battle-tested, one freshly minted. And every future change to payout reporting would either be made twice or silently drift apart.

That's not a bug. Nothing broke. It's worse in a quieter way. It's code-quality debt the agent created because it couldn't see that the thing it was building already existed.

This is the part I keep coming back to. It's not only about bugs. It's about code quality. A blind agent doesn't just get logic wrong. It reinvents, it duplicates, it walks past the shared helper and writes a fourth one. Every instance of that is a small tax you pay forever.

Why a senior wouldn't have done this

A coding agent is the best junior developer money can buy. Tireless, cheap, no ego, near-zero mistakes on routine work. An army of them at your terminal.

But juniors write code. They don't ship products.

A senior handed that report task would have opened with a different question. Not "how do I build a report" but "don't we already have one of these?" That instinct isn't in training data. It comes from having maintained a codebase long enough to know that a second implementation of anything is a liability, not a feature.

The agent never asked that question. Not because it's dumb. Because it only had my sentence. It had no way to know the single-business report existed, where it lived, or that extending it was the whole job.

So the problem stopped being "how do I get a smarter agent" and became "how do I put the judgment a senior carries in their bones somewhere the agent can read it — before it writes a line."

That question is why I built Bodhiorchard. It's open source, Apache 2.0, self-hosted. We run it at Atoa now.

We stopped handing agents a prompt. We hand them a BUD.

The unit of work in Bodhiorchard isn't a Jira ticket. It's a BUD — a Business Understanding Document. One living source of truth for a feature: what it is, why it exists, who asked for it, the design, the tech spec, the decisions made along the way. Vector-indexed, full history, no scatter across Confluence and Slack and someone's memory.

The point of a BUD is that the context is attached to the work, not floating around it. And crucially, Bodhiorchard traces a feature down to the actual code that implements it. A feature isn't a paragraph. It's a paragraph wired to the files, services, and procedures that make it real.

Work moves through seven phases, each its own tab on the BUD:

Requirements → Design → Tech Spec → Development → Code Review → Testing → Prod

Every phase has an agent. Every phase also has a toggle. Flip the agents off and it says "you're driving this BUD" — human in the loop, by design. This was never meant to run while you sleep.

The same task, inside Bodhiorchard

So I gave it the same job. Add callback params and dynamic fields to the payout report. Same class of model. Same one-line request.

This time it didn't start blind. The request came in against the payout feature's BUD, and because the BUD is wired to code, the agent pulled the context over MCP and found the existing single-business report — the real procedure, in the real repo — before it wrote anything.

# The agent reads context before writing — and finds what already exists
POST /mcp   → get_bud_context(bud_id: "PAYOUTS-217")

{
  "feature": "Payouts — multi-business support",
  "request": "add callback params + dynamic fields to the payout report",
  "precedent": {
    "report": "services/reporting → PayoutReport (single-business, live in prod)",
    "guidance": "extend this. do not stand up a second report pipeline."
  },
  "domain_rules": [
    "One report implementation per domain",
    "Reporting reads the ledger projection, never writes to it"
  ]
}

It extended the existing report. Multi-business, dynamic fields, callback params, added to the code that was already there. No new service. No second pipeline. One report that now does more.

The difference wasn't intelligence. It was sight. One agent could see the codebase and one couldn't.

And notice what the agent gets to do versus what it doesn't. The MCP write tools are bounded to the creative phases — it can create and update a BUD, draft the spec, propose the code. Nothing that moves money is a tool an agent can call directly. Read and propose across everything. Execute nothing with a blast radius. That line is drawn on purpose.

Behind the context sits a mechanical gate. Design-pattern lints run in CI, and if an agent reaches for a banned pattern or stands up a duplicate where an ADR says there should be one, the build fails. Not "a reviewer might catch it." The gate fails, every time. And the test suite leans hard on the negative cases agents love to skip — the illegal state transition, the duplicate event, the report that should have been an extension.

The scoreboard is pointed at the thing AI is worst at

Bodhiorchard has a scoreboard, and it does not reward output. It rewards shipped, working value.

You earn points when a BUD reaches production and when your code-review quality holds above a threshold. You lose a point for a bug caught in testing, and more for one that reaches production. The payout only lands when the feature is closed end-to-end, not when the PR merges.

Read that incentive again. It pays for correctness that survives production and for quality that holds up in review — exactly the things a fast, blind agent erodes when it's optimising for a green checkmark. Volume of code earns you nothing. A duplicate report that passes its own tests earns you nothing. Shipping the right thing does.

The honest status

I'm not going to tell you the agents run the whole SDLC while we sleep. They don't. The platform, the BUD lifecycle, the MCP context path, the feature-to-code indexing, the skill profiling — that's live and we use it daily. A fully autonomous execution loop is still being built. Today it's agent-assisted with a human in the loop at the phases that carry weight, and for anything touching payments that's exactly where I want it.

Where do agents own a boundary? Reading context, finding precedent, proposing a spec, drafting code, linting against recorded decisions. Where do they never own one? Anything where the failure mode is money in the wrong account. That's not a limitation I'm apologising for. That's the design.

So, back to blindness

The new report was never a model failure. Neither was the duplicate-refund class of bug before it. Both came from the same place: handing something a blank prompt and hoping it would guess the code you'd already written and the decisions you'd already made.

Give it the context a senior would carry into the review — the domain rules, the precedent, the boundaries, the code that already exists — and put a mechanical gate behind that context so nobody, human or agent, ships past it. The agent stops reinventing your codebase and starts extending it. It stops being a liability and starts being the army your seniors always wanted.

The 80% was already solved. This is how you stop the other 20% from shipping bugs — and from quietly rotting your code quality — while you're not looking.

What's the last thing an agent rebuilt in your codebase that already existed — and where was the context it needed to find it?

I'm Arun, CTO and co-founder at Atoa, building open banking payments for the UK. I built Bodhiorchard, it's open source, and we run it in our workflow. Repo's here if you want to poke at it — feedback over stars: github.com/mickyarun/bodhiorchard. Find me on X @mickyarun.

I Got Obsessed With Killing Scrum. So I Built a World Out of 12 AI Agents.

arun rajkumar — Sun, 12 Jul 2026 16:09:21 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

I built Bodhiorchard — an open-source platform that replaces Scrum, Jira, and the team wiki with 12 specialised AI agents and a living 3D world you can actually walk around in.

I call the methodology Agent-Driven Development (ADD). Instead of story points, planning poker, and a graveyard of stale Confluence pages, the work lives in one place: a BUD (Business Understanding Document) — a single, vector-indexed source of truth that carries a feature from a Slack conversation all the way to production.

The agents handle the busywork I never wanted humans doing: triage, design drafts, tech plans, standups, test plans, estimation, retros. Your repos show up as trees in an orchard. Features are branches. The whole team walks around as avatars — WASD to move, emotes, houses you upgrade with points you only earn when something actually ships to prod and stays healthy.

The scoreboard is the part I'm proudest of: it rewards quality, not output. Ship a BUD to production: +1. A bug escapes to prod: -1. It scores the things AI is bad at and humans are good at.

Honest status: the platform, the BUD lifecycle, the code-dependency graph, skill profiling, and the 3D world are all live. The fully autonomous execution loop is still being built — today it's agents-assisted, human-in-the-loop. I'd rather undersell it than lie to you.

Demo

Live site + walkthroughs: https://bodhiorchard.ai
Teaser — inside the virtual world: https://www.youtube.com/watch?v=OxoqBI7BNxU

Two views ship today: GARDEN (the orchard) and GRAPH (a cross-repo dependency graph with Bus Factor, Threats, and BUD-stage lenses). Two Slack bots sit on top — one triages new requests and kills duplicates before they become tickets; the other answers plain-English questions like "are we on track for go-live?" straight from the live BUD.

Code

Open source, Apache 2.0, self-hosted — it runs on a Mac mini.

mickyarun / bodhiorchard

Bodhiorchard is the open-source, self-hosted alternative to Jira and Linear for AI-native software teams. Specialised agents draft every spec, forecast cycle times, and reason over your real repositories, commits, and pull requests — replacing story points, standups, and stale tickets so engineers can focus on building.

Bodhiorchard™

Ship software, not Scrum ceremonies.

The open-source, self-hosted alternative to Jira & Scrum — AI agents run the process end-to-end, developers earn XP for what actually ships, and your data never leaves your machine.

Website · Quick Start · Why Bodhiorchard · The Twelve Agents · Docs · FAQ

▶ Watch the demo — a Slack message becomes a scoped, estimated BUD. No sprints, no story points, no standups.

Bodhiorchard replaces sprint, scrum, and Jira ceremony with Agent-Driven Development (ADD) — twelve specialised AI agents handle the busywork (triage, specs, estimates, test plans, retrospectives) while humans keep the decisions that matter, and developers earn XP for the work that actually reaches production. It's a self-hosted Jira alternative for the full lifecycle: intake → spec → design → development → testing → deploy → retrospective. The data plane stays on your hardware; inference runs through your choice of agent CLI…

View on GitHub

How I Built It

This is a solo project. Nights and weekends, on my own, because I couldn't leave it alone.

The stack: FastAPI + Python 3.12 on the backend, Vue 3 + PlayCanvas for the 3D world, Postgres + pgvector for the BUD memory, and Redis underneath. Inference runs through Claude Code today, with Ollama and OpenAI on the roadmap so you can bring your own model.

The hard part wasn't the agents. It was the ontology — deciding that a BUD, not a ticket, is the atom of work, and that everything (estimates, design, tests, the retro's estimated-vs-actual drift table) hangs off that one object. Once that clicked, the agents became small. Each one reads BUD context over a local MCP server and writes back only in the three creative phases where a human is still driving.

The 3D layer looks like the toy. It's actually the point. A dependency graph as a spreadsheet is a chore. A dependency graph as an orchard you can lose an afternoon in — that's the thing that made me want to open it every day.

The Passion Angle

I've spent years watching good engineers spend half their week on Jira instead of building. That quietly made me angry. This is the project I built to get that time back — mine first, then anyone else's who wants it.

"Build well. Then go outside." That's the whole idea.

Bodhi means awakening. An orchard is a grove you tend on purpose. The name is the pitch.

If you've ever felt sprints were broken and wanted to do something about it instead of just complaining in retro — clone it, break it, tell me where I'm wrong. That feedback is the only prize I'm actually chasing.