DEV Community: arun rajkumar

Two Strangers Built an Agent Mandate Protocol in My Comments. It Still Needs a Regulator.

arun rajkumar — Wed, 16 Sep 2026 05:45:09 +0000

I got something wrong in my own comments section nine days ago, and two strangers spent a week showing me how wrong.

The article was about guardrails that quietly stop running. Somebody in the thread described a scheme where an agent's authority is checked against a list of state revisions. I said the trouble with a list is that it rots, and the fix is expiry. Short lifetime, list never grows.

@anp2network said no:

A payment authorisation does not get its lifetime from a clock. It gets it from naming the effect [...] The question stops being "is this still fresh" and becomes "has this already been spent".

He is right, and I should have got there first, because it is what I do for a living. A card authorisation for £40 cannot be re-presented for £4,000. Not because it expires quickly. Because it names the amount and the payee, and the side that moves the money is the side that checks.

What happened after that ran twenty replies deep, mostly without me. @peterbuildssecure turned up and the two of them built an authorisation protocol for AI agents in my comments section over about a week. I want to write down what they built, because the thing it runs into at the bottom is not an engineering problem, and I do not think the agent-safety conversation has noticed.

What they built

@peterbuildssecure pushed the single-use idea further. A six-hour agent task is not one transaction. It is forty, or however many tool calls touch state, and each one has its own boundary. One mandate per operation, single use, burned by whoever performs the effect.

Then @anp2network found the hole. Some calls never resolve from the caller's side. A timeout that lands after the effect owner has already committed leaves you not knowing whether it happened, and treating that as failure lets the clock back in through the retry path.

@peterbuildssecure: fencing, not pending. On timeout the issuer writes a cancellation instead of waiting. But the tombstone has to live at the effect owner, because a delayed original still arrives there, and the issuer's records are not in the room when it does.

@anp2network: the cancellation write needs the same channel that just failed. So fold it into the replacement. M2 carries "supersedes M1", and the effect owner admits M2 and fences M1 in one commit.

@peterbuildssecure: and you do not keep the tombstone forever. Two tiers. A blocking record for as long as an honest late message could still turn up, then a cheap id marker after that.

That is a week of argument compressed into five paragraphs, and I have lost most of it in the compression. The thread is better than this summary.

The pattern nobody named

Read the sequence again and one thing repeats.

Every move deletes a store and creates one somewhere else.

The revision list rots, so use expiry. Expiry is a clock, so use single-use. Single-use needs a burn record. The burn record has to live at the effect owner. The effect owner cannot hold it forever, so split it in two and keep the cheap half.

The state never goes away. It gets smaller and it changes address.

And at the end of it there is a number. How long the blocking record lives.

Nobody in that thread could derive that number. Not for lack of ability. Twenty replies of extremely careful reasoning got to it and then stopped, because it is not the kind of thing reasoning produces.

Payments did not solve this either

This is the part I kept circling instead of answering.

Payments has the same number and does not compute it. It gets handed one. A card scheme's settlement window is a retention policy with a regulator attached, and the reason it works is not that the number is right. It is that the argument was ended by somebody with the authority to end it.

Both sides read the same rulebook, at the same revision, and neither one can move it afterwards and call it a clarification.

@anp2network pushed back when I said that, and fairly. You do not strictly need a regulator. An engineer-set window with a public revision history the effect owner cannot write to buys most of the same property.

Most of it. Publication makes a unilateral change visible. It does not make it expensive. A card scheme can throw a member out. A commit history cannot. That gap is the difference between a rule and a strongly worded preference, and plenty of systems run fine on the second one, provided everybody knows which one they are standing on.

Why this is not just a payments story

Every agent-safety mechanism I have read this year bottoms out in a number like this.

How long the audit trail stays queryable. How long an idempotency key blocks a replay. How long a revoked credential stays revoked in a cache. How stale a policy snapshot may be before a check refuses to run.

The mechanisms are good. Some are better designed than what payments was running on fifteen years ago. But the number underneath always shows up as a configuration default, and a configuration default is what you write when nobody has decided who owns the question.

If the owner is "the platform team", the number is whatever seems reasonable in the sprint where storage costs come up. That is not a dig at platform teams. It is what happens to any number with no counterparty on the other side of it.

Where the argument stops working

I do not have a clean ending, which is why this is a post and not a proposal.

The regulator analogy breaks as soon as you ask who the regulator would be. Payments got one because money moving in the wrong direction is legible to a state. An agent deleting the wrong S3 prefix is not, and I do not want an FCA for tool calls. I doubt anyone does.

The honest version is smaller than the analogy. Inside one company the caller and the effect owner are usually the same organisation, so the rulebook does not need a regulator. It needs a written-down owner and a change process that is not a pull request approved in forty seconds.

That is boring. It is also roughly what payments ran on before it had regulators, and it held for a while.

The question

Go and find the retention number in your own system.

The TTL on your idempotency keys. The window your dedupe cache actually covers. The age at which outbox rows get vacuumed. The lifetime of a revocation entry.

You have one. Somebody typed it.

Who was it, what did they know when they typed it, and what happens to them if it turns out to be too short?

If the answer to the last one is "nothing", you have a mechanism rather than a rule. Worth knowing that before an agent finds out on your behalf.

The thinking here is @anp2network's and @peterbuildssecure's, not mine. Thanks also to @salparvez, whose roof-and-foundation version of the two-tier idea is the one I actually remember, and @_firelinks, who showed me that a negative control run inside the thing it is testing is circular in exactly the state you built it to catch.

This is the third article I have written out of my own comments section. At some point that stops being a content strategy and starts being a confession.

Original thread: Nobody Checks Whether the Guardrail Is Running

Four People Rebuilt the Payment Authorisation in My Comments Section

arun rajkumar — Thu, 10 Sep 2026 05:22:57 +0000

I published two articles three weeks apart. They were not about the same thing.

The first was about agent guardrails that read the command string and miss the money. A refund for £40 and a refund for £40,000 are the same shape, and a hook that pattern-matches arguments cannot tell you which one is a catastrophe.

The second was about guardrails nobody checks the liveness of. A lint rule that stopped running looks exactly like a lint rule that found nothing.

Different subjects. Different threads. As far as I can tell, no overlap at all in who turned up to argue.

Both threads ended up building the same object. Nobody involved said the word "payments".

What the first thread built

Three weeks ago @peterbuildssecure left this on the guardrails post:

Preview resolves the proposed effect and returns an operation ID plus the subject, target, amount, currency, relevant state version and expiry. Human approval produces a single-use mandate bound to that exact operation. Execute consumes it atomically while rechecking the authoritative state and idempotency key.

Then the part I think is the most important sentence in either thread:

the server should require the mandate whether or not the client noticed that metadata. Otherwise the convention protects careful clients while direct callers retain the bypass.

What the second thread built

Yesterday, on a completely different post, @anp2network argued that you should stop trying to carry a reachability guarantee across a process boundary, because it cannot survive the trip:

The service that performs the effect refuses anything that does not arrive carrying evidence the guard ran, and it re-checks that evidence itself rather than trusting the caller was well behaved. Caller discipline buys nothing once there is a second caller you did not write. The evidence has to be bound to the specific instruction, and it has to name the policy revision it was checked against.

Read those two next to each other.

Single-use token, bound to one specific operation, issued by the side that owns the state, presented at the point of effect, re-verified there rather than trusted, rejected when stale.

That is one design, described twice, by two people who have never spoken to each other, three weeks apart, on two posts about different problems.

It is also a payment authorisation. It has been a payment authorisation since roughly 1979.

The other two got further than I did

@jon_at_backboardio went straight past the single-operation model to its weak point:

five refunds of £8,000 each, every one under every cap, same payee, four minutes. every individual call is boring. the mandate model catches that only if mandates are scoped to a window rather than to an operation

He also settled the sync-versus-async argument in one observation: the calls where blocking hurts are high-volume and low-value, and the calls where blocking is fine are the ones you want to stop. Those two sets barely overlap, so you pay the latency on a rounding error of your traffic.

And @max_quimby, who started the whole thing, put his finger on why none of this is really an AI problem:

None of that is agent-specific, which I think is the point — the agent just removes the human who used to eyeball the number.

Payments did not design this. It lost money until it existed.

Every piece these four independently derived is something the card networks were forced into, usually after being robbed.

Bound to the instruction. An authorisation is for an amount and a payee. You cannot get one for £40 and present it for £4,000. That is not elegance, it is scar tissue from people doing exactly that.

Verified by the side holding the state. The merchant does not decide whether an authorisation is good. The issuer does, because the issuer is the only party that knows the balance. Peter's "the server should require the mandate whether or not the client noticed" is that rule, stated fresh, forty years later.

Single-use. Replay is the oldest attack there is.

Expiry. This one gets skipped in every clean-room design and it is load-bearing. Without it your set of still-valid mandates only grows, and a list that only grows becomes a list nobody audits. Expiry is what keeps that set small enough to reason about. You trade a list that rots quietly for a clock that fails loudly, and the clock is much easier to think about at three in the morning.

Three more things the threads are about to discover

Since payments got here first and paid for it, here is what is waiting further down the road.

Velocity, not caps. Jon's five-refunds case is not an edge case, it is the standard attack, and per-transaction limits have never caught it. What catches it is cumulative exposure per counterparty per window. The check is easy. Choosing the window is not, and whatever window you choose, somebody can straddle it. Four minutes clears an hourly window if you start at 59 minutes past.

Preview is a surface too. Auth-then-capture has a gap between the two halves, and people have lived in that gap professionally for decades. If preview simulates the effect and execute performs it, any divergence between those two code paths is somewhere a difference can hide. Much smaller than the hole it closes. Not zero.

Approval screens launder decisions. Jon again, and it is the best line anyone has left on anything I have written: showing a human raw arguments is not review, it is laundering. If the human cannot see what the number means relative to the account it is hitting, you have not added a control. You have added a signature to blame later.

The one part that does not transfer

I do not want to oversell the analogy, because there is a real gap in it.

A payment has a natural boundary. There is a moment the transaction starts and a moment it settles, and every lifetime in the system hangs off those two points. Expiry is easy to reason about because there is an obvious thing for it to be shorter than.

An agent halfway through a long task has no such boundary. The task might run for six hours. It might spawn sub-tasks. It might pause overnight and resume. Scope the mandate to the operation and Jon's five-refund case walks straight through. Scope it to the session and you have issued a bearer token for a session whose end you cannot define.

So: what is the agent equivalent of a transaction boundary?

I do not have an answer. Payments handed us the shape of the solution for free, and then kept the one thing that made it work.

Quoted with thanks and without permission: @peterbuildssecure, @anp2network, @jon_at_backboardio and @max_quimby. The threads are here and here, and both are worth more than the posts they are attached to.

Your Audit Log Agrees With Itself. That's the Problem.

arun rajkumar — Wed, 09 Sep 2026 10:15:14 +0000

@remdore ran an experiment this week that I have not stopped thinking about.

One account, balance 100. Ten workers, each withdrawing 10. Every worker does the obvious thing: read the balance, subtract, write it back, log what happened.

Final balance: 90. Nine withdrawals gone.

That part is a race condition and every backend engineer has met it. The part I want to talk about is the audit log.

Ten rows. Every one of them says the balance went from 100 to 90. No gaps. No nulls. No orphaned foreign keys. Nothing a schema validator, a checksum, or a nightly reconciliation job would flag.

Hand me that table during an incident and I would read it as one withdrawal retried ten times. I would go and look at the retry logic. The retry logic is fine. I would spend the afternoon there.

The log is not wrong. That's what makes it dangerous.

We are in regulated payments. The audit log is not a debugging convenience for us. It is the artefact. It is what an auditor reads, what a dispute resolves against, what we would put in front of the FCA if someone asked how a payment reached the state it did.

So the standard we hold it to is not "does it look sensible". It is "is this the record of what happened".

And the failure in that experiment passes every test I know how to write against a log.

Think about what you can actually check automatically:

Schema conformance. Passes.
No nulls in required columns. Passes.
Referential integrity to the account. Passes.
Monotonic timestamps. Passes.
Balance arithmetic internally consistent per row. Passes, ten times.
Row count matches the number of operations attempted. Passes. Ten operations, ten rows.

Every one of those questions is about whether the log is coherent. None of them is about whether the log is true. Ten processes each believed they were alone, and each one wrote down a faithful account of the world as it saw it. Coherence is exactly what you get when ten honest narrators all have the same wrong information.

The thing you would need to detect it is a row that disagrees with another row. There isn't one. Disagreement is the signal, and the bug removed the signal.

This is the same failure I wrote about two days ago

I published something on Monday about guardrails nobody checks the liveness of. A lint rule that stopped running, a policy check that matches nothing, an eval that skips half its corpus and reports the half it ran. The thread on it got long, and the argument that came out of it was sharper than the article.

The shape is the same here.

A check that never fires and a log that never contradicts itself both produce the same reading: nothing is wrong. In both cases the absence of a complaint is being taken as evidence of health, and in both cases the mechanism that would produce the complaint is the thing that broke.

You cannot fix that by inspecting the output more carefully. The output is clean. That is the whole point.

The one-statement version, and why it matters more than it looks

Postgres 18 lets you name the row before and after a change in the same statement that makes it:

UPDATE accounts SET balance = balance - 10 WHERE id = 1
RETURNING old.balance AS was, new.balance AS now;

The syntax is nice. It is not the interesting part.

The interesting part is that the log entry is now derived from the write instead of being a second write that describes the first. Before this, the audit row was a separate statement carrying a separate copy of the truth, and two copies of the truth can drift. After it, there is one statement, and the record and the change are the same event. There is no version of the run where the balance moves and the log says something else, because there is no seam between them for a lie to get into.

That is the general principle and it is worth more than the syntax. Any time the record and the thing being recorded are two separate writes, you have two versions of the truth, and only one of them is the one that moves money.

You can apply that without Postgres 18. The change is to stop thinking of the audit log as something you write about an operation and start treating it as something the operation emits. If a code path can complete without producing its record, the record is optional, and optional loses to deadline pressure every time.

Where this stops working

I want to be honest about the limit, because the single-statement version solves a single-database problem and most of us do not have a single-database problem.

The moment the operation crosses a service boundary, the seam is back. The money moves in one place and the record is written in another, and now you are relying on an outbox, or a queue with delivery guarantees you have read about but not tested, or in the worst case a developer remembering to call the logging function. Convention again.

The only thing I have found that helps at that scale is narrowing the number of routes. Anything touching money goes through one path with the record emitted inside it, so there is no version of the call that reaches the effect without producing the evidence. Everything outside that path is convention, and I am not comfortable about it, but I would rather have one path I can defend than six I have to trust.

The question I would actually like answered

Here is what I do not have a good answer for.

Every detection method I know for this class of bug is a disagreement detector. Two sources that should match and don't. A count that should equal another count. A hash that should still verify. All of them need two things to compare.

The read-modify-write bug produces one thing, ten times. There is nothing to compare it against, because the second source was never independent.

So: has anyone got a check that catches a log which is internally consistent and collectively false, without needing an independent second record to diff against? Not a stronger schema. Not more constraints on the row. Something that notices that ten narrators agreeing is itself suspicious.

I think the honest answer might be that there isn't one, and the only real defence is making the seam impossible rather than detecting what crawls through it. But I would like to be wrong about that, because "make the seam impossible" is expensive and does not retrofit.

Credit where it is due: the experiment and the Postgres 18 walkthrough are @remdore's. Go and read the original. I have only taken the part that scared me and pointed it at my own industry.

Nobody Checks Whether the Guardrail Is Running

arun rajkumar — Mon, 07 Sep 2026 14:07:25 +0000

Every AI-and-engineering post right now is about adding a guardrail.

Lint rules the agent can't bypass. Evals before you ship a prompt change. A review bot on every PR. A regression suite built from real production failures. Architecture rules checked into the repo so the model reads them on the way in.

I have written some of those posts. I still believe in them. This is the work that makes AI usable by senior engineers instead of a liability.

But I have been in three separate conversations on this platform in the last week that were all, underneath, about the same thing, and it isn't about which guardrails to add.

A guardrail that has never fired and a guardrail that silently stopped running produce identical output.

Green.

The pipeline that had never been green

Vicente Reyes had a GitHub Actions workflow called Deploy to DigitalOcean. Fully wired. SSH action, secrets, the works. And every time he shipped a backend change he still SSH'd into the droplet and ran git pull by hand.

The deploy job was gated on CI passing.

on:
  workflow_run:
    workflows: ['CI']
    branches: ['main']
    types: [completed]

jobs:
  deploy:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

Sensible enough. The problem was that CI had never once gone green on main. Not flaky. Never. So every deploy run showed skipped, forever, and the pipeline sat permanently behind a gate that could not open.

His line about it is the one worth keeping. A pipeline that is silently and permanently blocked looks, from a distance, exactly like a pipeline that doesn't exist.

Nothing in the Actions UI says this workflow has not succeeded in forty runs. You have to go and ask.

The grader that can't say no

On a thread about designing trustworthy AI evals, Heinrich Neb made the point that every grader needs a known-bad twin. An input it is supposed to reject, plus a recorded date of when it last actually rejected something.

His framing is the sharpest version of this I have seen: a grader that has never failed and a grader that silently stopped running print the same green.

Same failure as Vicente's pipeline, one layer up. In his case the gate was stuck closed. In an eval suite the gate is stuck open, which is worse, because a stuck-closed gate is annoying enough that somebody eventually investigates. A stuck-open gate just keeps saying yes.

Think about how an eval suite actually rots. Somebody changes a prompt template and the grader's regex stops matching, so everything scores as pass. Somebody renames a dataset field, the loader returns an empty list, the suite runs zero cases in 0.4 seconds and reports 100%. A provider changes a default and your grader model gets more agreeable.

All three look like success.

The score with no provenance

The third one is mine, from the same thread.

An eval result with no harness version, no dataset snapshot and no prompt revision attached to it is not evidence. It is a self-reported claim.

Which is fine right up until the number moves. Then somebody asks whether the model got better or the suite got easier, and if you can't answer, you never had a measurement. You had a vibe with a decimal point on it.

That reflex comes from working in payments. In a regulated system nobody asks you to trust that a control ran. They ask you to demonstrate which control ran, on what input, at what time, under which version of the rules. Months later. To somebody who wasn't there and isn't inclined to take your word for it.

Engineering has quietly inherited that burden. Evals are increasingly the artefact a shipping decision rests on. They just haven't inherited the paperwork.

What actually ties these together

When you automate a check, you swap one question for another and don't notice.

Before automation the question is did someone look at this? You know the answer, because you can see the person and ask them.

After automation the question you think you are asking is still did the check pass? But the question you are now depending on is is the check alive?

Almost nobody instruments the second one.

This is well understood in operations. You don't only alert on errors, you alert on the absence of a heartbeat, because a monitoring system that dies looks exactly like a system with no problems. Dead man's switches exist for precisely this reason.

We have somehow not carried it across to the checks that gate our code. A CI workflow, a lint rule, an eval suite, an agent policy. These are all monitoring systems for correctness, and we run them with no heartbeat at all.

What a guardrail needs before you trust it

None of this is exotic.

A known-bad input it must reject. Every check needs a case it is supposed to fail on, running alongside the real ones. If your lint rule can't catch its own canary, the lint rule isn't running. If your eval's negative control scores as a pass, the grader is broken and every other number in that run is noise. Same idea as a negative control in a lab. Nobody trusts an assay that only ever comes back clean.

A recorded date of last rejection. Not when it last ran. When it last said no. A guardrail that hasn't rejected anything in four months is either protecting an unusually disciplined team or it broke in May, and those look identical on a dashboard. Put it in a column somewhere. If nobody can answer "when did this last catch something", it is decoration.

A run count somebody occasionally looks at. The zero-cases failure is the sneakiest one, because a suite that loads an empty dataset passes fast with a perfect score. Assert on the count. If it expects 240 cases and got 0, that is a hard failure, not a 100%.

Provenance on the result. Harness commit, dataset hash, prompt revision, model version, timestamp. Attached to the score, not sitting in a CI log with thirty-day retention. The test is simple. Six months from now, can you reproduce this exact number? If not, you can't use it to defend a decision, which means it was never really the reason for the decision.

Why this gets worse with agents, not better

The reason I keep coming back to this is that AI shifts the ratio.

The argument for agents in engineering, the one I actually believe, is that you get something close to an army of near-zero-mistake juniors. The constraint stops being how fast people can write code and becomes how fast people can review it. So you compensate by pushing more review into automation. More lint rules, more tests, more evals, more policy checks.

That is the right move. I would make it again.

But it means the fraction of your correctness resting on unattended machinery goes up sharply. When a human reviewed everything, a broken lint rule was a small hole in a large net. When automation reviews everything, the broken lint rule is the net.

The guardrails become load-bearing at exactly the moment nobody is watching them closely enough to notice they stopped.

Go and check one

Pick the guardrail you would be most upset to lose. The eval suite that gates prompt changes, the rule that stops an agent touching the payment path, whatever yours is.

Then answer one question about it. When did it last say no?

If you can find that out in under a minute, good. If you can't find it at all, you don't have a guardrail. You have a green light with nothing behind it, and you have been treating it as evidence.

Credit where it's due. The "never failed and stopped running print the same green" framing is Heinrich Neb's, and the permanently-gated pipeline is Vicente G. Reyes'. I just noticed they were the same bug.

I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about AI, payments, and the messy parts of running engineering systems. @mickyarun.

Your Health Check Is Lying to You

arun rajkumar — Fri, 04 Sep 2026 12:30:27 +0000

Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it.

The last act is the one I want to pick up.

Two minutes after his first fully automated deploy, the site returned a 502. Nothing had crashed. The old container was gone, the new one hadn't finished booting, and he happened to refresh inside a window about one second wide.

That window is in almost every Docker Compose deploy I have seen. It is in mine.

The interesting part isn't the window. It's that the obvious fix only closes half of it, and the other half fails much more quietly.

The proxy has no idea

A static Traefik route looks like this:

http:
  services:
    api:
      loadBalancer:
        servers:
          - url: http://api:3000

That config says: send traffic to api:3000. It does not say: send traffic to api:3000 when something is actually listening there. Traefik has no opinion on the matter. It resolves the name, opens a connection, reports whatever comes back.

So every time docker compose up -d stops the old container and starts the new one, there is a stretch where the proxy is confidently routing at a process that is still running migrations. Nobody hits the app during that stretch, you never find out.

The fix everyone reaches for is an active health check.

http:
  services:
    api:
      loadBalancer:
        servers:
          - url: http://api:3000
        healthCheck:
          path: /healthz
          hostname: api.example.com
          interval: "5s"
          timeout: "3s"

Correct, and it's where most write-ups stop.

The hostname line is not optional

Leave hostname out and Traefik sends the check with Host: api, the internal Docker service name. Anything with a host allowlist in front of it will reject that. Django gives you a DisallowedHost 400. Express with a host check does the same. So does NestJS behind most gateway setups.

From Traefik's side a 400 is a failed check, and a container that fails every check is permanently unhealthy.

So you spend an afternoon debugging why your app is down. Your app is fine. The health check is asking the question in a form the app is contractually obliged to refuse.

Set hostname to the real public domain. Then the check looks like real traffic, which is the whole point of it.

Liveness and readiness are different questions

The advice attached to that config is usually "keep /healthz trivial". No database, no auth, just return 200. That advice is right, and for a good reason: if your health endpoint touches a business dependency, a permissions change or one slow query quietly pulls the container out of rotation.

But a trivial endpoint answers a trivial question. It answers is this process alive. It does not answer can this process serve a request.

Those two come apart constantly. A container that boots cleanly but can't reach its database will return 200 from a trivial check all day. Pool empty. Queue consumer never attached. The proxy sees green, sends real traffic, and instead of one second of clean 502s you get sustained 500s from a container that is reporting healthy.

So split them. Liveness stays trivial and wires to your restart policy. Readiness checks what the container needs to do its job and wires to the load balancer. Collapse the two and you pick one failure mode to catch and get blindsided by the other.

The part nobody writes down

This is the bit I actually sat down to write.

The natural readiness check for a service that depends on Postgres is a SELECT 1. Cheap, honest, and it's what I'd reach for too.

Think about what it proves. It proves the pool can hand you a connection. It says nothing about whether the pool is nearly empty.

Under load those diverge, and the divergence is ugly. If readiness borrows from the same pool the application serves from, then when that pool is close to exhausted the check either succeeds by taking the last free connection, which is the moment you least wanted a health probe competing with real requests, or it fails outright because there was nothing left to take.

The second one is what hurts.

A container fails readiness. The load balancer pulls it. Its traffic redistributes to its neighbours. Those neighbours now carry more load, their pools saturate, they fail readiness too, and out they go.

You have built a check whose entire job is to protect the system, and under exactly the conditions it was meant to protect against, it removes the system.

Three cheap ways out, pick one:

Give readiness a dedicated connection outside the application pool. Then a saturated pool is a slow app rather than an unhealthy one.

Or keep it on the shared pool and be honest about what it asserts, which is "Postgres is reachable from this container". Fine thing to check, as long as nobody downstream believes it means "has capacity".

Either way, never let a single failed probe evict. Require consecutive failures over a window so a momentary squeeze can't cascade.

The general principle is that a readiness check consuming the same scarce resource it measures will always give you its worst answer at the worst possible moment.

Readiness is a dependency graph you're signing up for

Everything you put in a readiness check becomes something that can take you out of rotation.

Put a downstream HTTP call in there and you have coupled your availability to somebody else's. They have a bad ten minutes, every one of your containers marks itself unready simultaneously, and now you have zero healthy instances of a service that was working perfectly well.

So the scoping rule is narrow. Check only what this container needs to serve its own traffic.

Vicente landed on exactly this for his stack. Postgres in, Redis and Celery out, on the grounds that background jobs degrading is not the same thing as the web tier being unable to answer. Redis going down should page someone. It should not empty the load balancer.

That's a design decision, not a config line, and it's worth the ten minutes.

Start-first doesn't get you to zero either

Say you fix all of that. The window shrinks from "boot plus migrations plus static assets" down to "process fork time".

It shrinks. It does not close. docker compose up -d stops the old container before the new one is ready, and no health check changes that ordering.

The usual next step is a start-first rollout. Bring up the new container, wait for it to pass readiness, kill the old one. The docker-rollout plugin does this, Swarm and Kubernetes do it natively.

The bit that gets skipped: start-first closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight.

Two things have to be true for that second window to be harmless. The proxy has to notice the old server left and stop sending it new work before the container dies, and Traefik's config reload is not instant. And the application has to be allowed to finish what it is already holding. In gunicorn that's graceful-timeout. In Node it's whatever you wired into your SIGTERM handler, and if you wired nothing, the answer is that it doesn't.

A rollout without a drain window doesn't eliminate dropped requests. It moves them from the front of the deploy to the back, where they are harder to spot because the deploy reports success.

If your app is anywhere near money that distinction matters a lot. A request dropped at the front is a failed connection and the client retries. A request dropped at the back was accepted, was being processed, and then wasn't.

What to actually do

The answer is different at each size, so:

Single box, Compose, low traffic. Split liveness and readiness, scope readiness to your own datastore, set hostname, live with a health-gated couple of seconds. Completely reasonable place to stop. Reaching for an orchestrator here costs you more than the window does.

Multiple instances behind a proxy. All of the above, plus a failure threshold so one bad probe can't evict, plus readiness on its own connection. This is where the cascade becomes possible, so this is where it's worth paying for.

Deploys that cost you money when they drop requests. Start-first rollout with an explicit drain. Proxy deregistration before SIGTERM, graceful timeout longer than your slowest normal request.

The tools matter less than the question.

Every health check is an assertion, and the failure is always the same one. The assertion is narrower than what everyone downstream believes it to be. Somebody reads a trivial 200 as "can serve". Somebody reads a SELECT 1 as "has capacity". Somebody reads a green deploy as "nobody got a 500".

Write down what your check actually proves. Then go and find out who is relying on it to prove more than that.

Thanks to Vicente G. Reyes for the original write-up and for a good argument in the comments. Most of this is that argument.

I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about the messy parts of running engineering systems. @mickyarun.

Your Agent's Guardrails Can't See the Money

arun rajkumar — Fri, 21 Aug 2026 10:53:11 +0000

There's a post going round this week about agent guardrails that opens with a good story. The author's agent wanted to force-push to main. Not because it was confused. The rebase was stuck, force-pushing would unstick it, and every step in that chain of reasoning was sound. Locally correct, non-locally expensive.

What makes that example teachable is that the command carries its own consequence. git push --force origin main has the danger written into it. You can pattern-match it. You can put it on a list. A hook can read the string and stop.

Most of what I worry about doesn't look like that at all.

Two calls, one of them a disaster

Here are two refund calls.

await payments.refund({
  paymentId: "pay_9f2c14",
  amount: 4000,
  currency: "GBP",
  reason: "customer_request",
});

await payments.refund({
  paymentId: "pay_9f2c14",
  amount: 4000000,
  currency: "GBP",
  reason: "customer_request",
});

One is forty pounds. The other is forty thousand. Same endpoint, same argument names, same shape, same reason code, same everything a pre-execution hook can read off the text.

Now the part that actually matters: neither of them is inherently wrong.

Forty thousand might be a perfectly good refund against a perfectly good invoice. Forty might be a refund on a payment that was already refunded an hour ago, which in some ways is the worse of the two. You cannot sort these by looking at them, because the thing that makes one of them a mistake isn't in the call.

Blast radius is a property of the target, not the command

The force-push case works because the danger lives in the verb. Force-push is dangerous in nearly every context. The set of situations where you genuinely want it is small enough to enumerate, so a rule can cover it and the rule stays roughly true.

An amount-bearing API call is dangerous as a property of the state on the other side. Whether that refund is safe depends on things the caller simply doesn't have:

Has this payment already been refunded, fully or partially?
Does the merchant's balance cover it, or does this pull them negative?
How much has this actor already moved today?
Is the original payment under dispute, where a refund does something quite different to who ends up liable?
Is pay_9f2c14 even this merchant's payment?

A hook sitting in the agent process knows none of this. It can read intent. Cost is not a property of intent.

"Fine, have the hook go and check"

This is the obvious next move and it isn't stupid. Let the guard call the API, fetch the payment, look at the balance, then decide.

Try building it and you find out what you signed up for.

Your hook now needs credentials to read payment state, which means the agent host is holding read access to your ledger in order to protect you from the agent host. It needs to understand refund semantics, dispute states and settlement timing well enough to form a judgement, so your refund rules now live in two codebases that have to agree forever. And it has a window between checking and executing, which is exactly where the interesting failures live. The balance was fine when you looked. Something else landed. Yours goes through anyway.

You haven't built a guardrail. You've built a second, worse copy of your authorisation service, running in the least trusted process you own, with a cache.

Put the check where the state is

The version that survives contact is boring. The check goes at the rail, at the point of authorisation, because that's the only place that knows what this actor may move, what it has already moved, and what the target looks like right now.

We're regulated, so we already had to have that spine. Every movement of money goes through an authorisation step with the full picture and an audit record on the other side. Wiring agents into it didn't mean building something new. It mostly meant resisting the urge to build something in front of it and call that safety.

The agent-side hook still has a job. Just a smaller one than people want to give it.

// The hook classifies and asks. It does not decide.
async function preToolUse(call) {
  if (!MOVES_MONEY.has(call.tool)) return allow(call);

  // Ask the system that owns the state what this would actually cost.
  const effect = await rail.preflight(call.args);

  return confirmWithHuman({ intent: call.args, effect });
}

rail.preflight is where the work happens, and it lives on the side that has the ledger. Its whole purpose is so a human sees "this refunds £40,000 against an invoice already refunded in full on the 3rd" rather than "the agent would like to call refund." One of those is a decision. The other is a rubber stamp with extra steps.

It's also explicitly advisory. Between preflight and execute the world moves. Enforcement stays server side, under a mandate scoped to an amount, a payee and a clock. If preflight and the rail ever disagree, the rail wins.

I wrote more about mandates versus API keys in an earlier post if that thread interests you.

The uncomfortable bit

Your agent framework cannot own your safety story for anything that moves money, and it doesn't matter how good its hooks get.

That's not a knock on the hooks. Command-level guards are useful and I'd rather have them than not. For destructive verbs, where the danger is in the word, they're the right tool and they'll catch real mistakes that would otherwise cost someone a weekend.

They just sit at the wrong altitude for a whole class of call that reads as completely unremarkable and is defined entirely by context the caller doesn't hold.

So here's the test I'd apply to any guard before trusting it. Could this exact call, byte for byte, be both correct and catastrophic depending on something the guard cannot see? If the answer is yes, that guard is a linter. Useful. Not load-bearing. Put the real check where the state lives.

Has anyone landed on a decent convention for a tool declaring "you can't infer my blast radius from my arguments, come and ask"? I haven't seen it in the MCP spec, and it feels like the piece that's missing.

Your MCP Eval Checklist Has an Auth Row. In Payments It's the Whole Table.

arun rajkumar — Mon, 10 Aug 2026 14:16:19 +0000

There's a good checklist going around dev.to for vetting an MCP server before you wire it into an agent. Four things to look at. Tool surface area: how many tools, and are they atomic or coarse. Auth model: API key, OAuth, token scope. Maintenance: last commit, open issues, is anyone home. Token profile: does it dump a full document when a summary would do.

It's a genuinely good checklist. I've used a version of it. For most tooling categories it's exactly the right lens.

Then you point an agent at something that moves money, and three of those four rows go quiet.

Not because they stop mattering. Because one of them grows until it's the only thing you're really deciding.

A read tool and a write tool are not the same animal

Here's the thing that took us a while to say out loud.

If an agent calls a code-search tool twice, you get the same answer twice and waste a few tokens. If it reads a git diff twice, nobody notices. Reads are safe to repeat. That's the whole reason retries are the default everywhere in the agent stack. A tool call times out, the client tries again, you move on.

A payment tool call is not a read. Retry it once and you've billed someone twice.

We build open banking payments. The failure that actually keeps me up isn't a hallucinated argument or a server returning a forged result. It's the boring one. The connection blips mid-checkout, the client does what clients do and retries, and now there are two payment intents where the user meant one.

So the eval question for a money-moving tool isn't "what's the auth model." It's "what happens on the second call I didn't mean to make."

Idempotency lives on the intent, not the turn

The fix is old and unglamorous. Idempotency keys. Everyone in payments already knows them. The part people get wrong with agents is where the key lives.

The instinct is to make the agent turn idempotent. Same prompt, same result. That's the wrong seam. The agent turn is fuzzy by design, and you don't want it to be the thing carrying the guarantee.

Put the key on the payment intent. The client generates it once, before the tool is ever called, and it travels with the money, not with the conversation.

// The key is minted where the intent is born, not inside the agent loop.
const intent = {
  idempotencyKey: crypto.randomUUID(), // one per real-world payment
  amount: 4200,
  currency: "GBP",
  payee: "merchant_8842",
};

// Retries of the SAME intent collapse to one charge.
// A genuinely new payment gets a new key, on purpose.
await paymentsTool.charge(intent);

Now a dropped connection is harmless. The retry carries the same key, the server recognises it, and the second call returns the first result instead of moving money again. The agent can be as jittery as it likes. The guarantee sits below it, where the stakes are.

Read paths open, write paths gated

The other move is to stop treating "tools" as one category.

On the read side we let the agent run. Fetch balances, list transactions, pull an account's status, look up a payout. If it over-calls, it wastes tokens and we tune it later. Low blast radius, no gate.

On the write side, anything that changes state or moves money goes behind a human confirmation. Not the agent confirming to itself. A person, or a service acting under an explicit, narrow mandate, in the loop before the call executes.

If you're on Claude Code or a similar setup, the cheap version of this is a pre-call hook that classifies the tool and decides whether it needs a gate.

// Classify by side effect, not by name.
const NON_RETRYABLE = new Set(["charge", "refund", "payout", "mandate.create"]);

function preToolUse(call) {
  if (NON_RETRYABLE.has(call.tool)) {
    return requireHumanApproval(call); // blocks until a person says yes
  }
  return allow(call); // reads sail through
}

The point isn't the code. It's the split. Reads and writes want different defaults, and a checklist that scores a server as one thing misses that the same server can hold both.

The credential is a mandate, not a key

The auth row on the checklist usually asks whether it's an API key or OAuth. Fine question. Wrong altitude for payments.

What you actually want to hand an agent is a mandate. Scoped to an amount and a payee. Time-boxed, so it expires whether or not anyone remembers to revoke it. Revocable mid-flight. And auditable after the fact, so when someone asks "why did this money move" there's a straight answer that doesn't depend on trusting the model.

A key says "this caller is allowed." A mandate says "this caller is allowed to move this much, to this party, until this time, and here's the record." The second one is the only thing I'd let near a live payment rail.

We already keep that audit spine for money movement, because we're regulated and there's no version of this job where you don't. The work with agents wasn't inventing it. It was extending the same discipline to tool calls, so an agent's action leaves the same trail a human's would.

So, the checklist

Keep all four rows. For a knowledge base or a code-search server, run the standard lens and move on.

But the moment a tool can move money, promote one question above the rest and answer it first: is this call retryable, and if it isn't, what stops the second one? Everything else on the checklist is downstream of that.

A read tool can be replayed all day. A payment tool replayed once bills a real person real money.

Are you seeing any of the community MCP servers treat retryable and non-retryable tools as different classes yet, or is that still left entirely to whoever's calling them?

Gartner Says 40% of Apps Will Have AI Agents by December. Here's the Plumbing Nobody Puts on the Slide.

arun rajkumar — Mon, 03 Aug 2026 08:09:14 +0000

There's a number going around dev.to this week. Gartner says 40% of enterprise apps will ship a task-specific AI agent by the end of 2026. Last year it was under 5%.

Every deck quotes it. Every thread argues about it. Fine.

I run engineering at a UK payments company. We're FCA-authorised, SOC2, the whole regulated stack. On the side I build an open-source agent framework called Bodhiorchard, where a dozen agents do real work on a real codebase. So I've shipped the thing the slide is describing. And I can tell you the 40% isn't the hard part.

The hard part is everything under the slide. And the biggest piece of it is money.

The demo is not the deployment

An agent demo is easy. You give it a prompt, it writes some code or drafts a report, everyone claps.

Then you try to put it near a live system and the questions start. What can this thing actually call? What happens when it's confidently wrong? Who gets paged?

None of that shows up in a projection. All of it shows up in production.

The real boundary isn't "no money." It's a signed mandate.

The reflex guardrail is to forbid the scary thing. Don't let the agent move money. Read-only, propose-only, a human clicks the button.

That reflex is already history. Agents are going to move money, because paying is half of what it means to finish a task. An agent that can research a supplier, compare options and fill a cart, then stops and waits for a human to tap "confirm," isn't an agent. It's an intern with a shopping list.

This is the part of the 40% that actually rewires the economy. Not agents writing code. Agents that transact. The moment an agent can pay, it stops being an assistant and becomes an economic actor, and the rails for that are being built right now.

That's what we're building at Atoa: a regulated bank rail for AI agents. An agent settles a payment bank to bank, over an FCA-authorised rail, with a signed record on every move. Built for the protocols this is standardising on, AP2 for mandates, x402 for pay-per-request, MCP as the interface. So "never let it touch money" was never going to be our answer. The answer is where the authority lives.

An agent's permission to spend can't be a line in a system prompt that a clever input talks its way around. It has to be a signed, scoped mandate: explicit about how much, to whom, within what limits, and verifiable on its own. Every payment gets a real-time affordability check. Every payment leaves a signed, provable record, who authorised it, what the funds check returned, where it settled.

That's the shift worth internalising. The boundary moved from "can the agent act?" to "is the agent's authority explicit, scoped, and provable?" Money moving for an autonomous agent needs more oversight, not less. So you make the authority a hard artifact and the audit trail non-optional, and then you let it pay.

Context in, not hope in

Here's a real one from Bodhiorchard.

An agent was asked to produce a payout report. Left to its own reading of the task, it started building a brand new service to generate that report. The report already existed. It was about to rebuild something we already had, in a slightly different shape, as new surface area to maintain.

The fix wasn't a smarter prompt. It was context. We feed agents structured context through MCP before they write a line, what we call a BUD in Bodhiorchard. Once the agent could see the existing report and the decisions behind it, it did the sane thing. It extended what was there instead of spawning a duplicate.

That's the difference between an agent that helps and one that quietly grows your tech debt. Not intelligence. Context.

Deterministic checks come before the probabilistic step

An agent's output is a guess. A good guess, often. Still a guess.

So the guess doesn't get to be the last word. Before anything an agent produces goes near a real code path, it runs the checks a human would have to pass. Types. Schema validation. The test suite. The design-pattern lints that encode decisions no off-the-shelf linter ships with. In the payments flow, the affordability check plays the same role: a deterministic gate the probabilistic step has to clear before anything settles.

If that deterministic layer isn't there first, you haven't deployed an agent. You've deployed a very fast intern with commit access and no code review.

Someone owns the failure

This is the part nobody wants on the slide, because it's a headcount question, not a technology one.

When an agent fails, it usually doesn't crash. It fails plausibly. The report looks right. The code compiles. The number is just wrong. That kind of failure needs a human owner who knows the domain well enough to smell it, and a record clean enough to trace it back.

My mental model: an agent is an army of near-zero-mistake juniors. That's a gift to a senior engineer and a trap for a team without one. Enabling seniors with agents is the right move. Replacing seniors with agents is how you find out what plausible failure costs.

So

Yes, 40% of apps will probably have an agent by December. The slide will be right.

But the agent isn't the work. The scoped mandate, the context feed, the deterministic gate, the human who owns the failure. That's the work. That's the 60% under the waterline.

And the biggest piece of that iceberg is payments. The agent economy doesn't start when models get smarter. It starts when agents can pay, safely, over rails that were built for them. That's not a 2030 story. It's live now, and we're building one of the rails.

If you're shipping an agent this year, which of those four do you already have, and which are you hoping the model handles for you?

AI Agents Ship Bugs When They're Blind. So We Stopped Building Blind.

arun rajkumar — Fri, 31 Jul 2026 06:51:09 +0000

Most AI coding mistakes don't look like mistakes.

They compile. The tests pass. CI is green. The diff reads fine at 6pm on a Friday. Sometimes it's a bug that surfaces two weeks later when money moves where it shouldn't. And sometimes it's not a bug at all — the agent quietly rebuilds something you already had, and you don't notice until you're maintaining two versions of the same thing.

Same root cause both times. The agent couldn't see. It had my prompt. It didn't have the codebase.

I'm the CTO of a payments company. We're FCA-authorised, we move real money for real merchants, and we've leaned on AI agents hard for over a year. So I've watched this happen enough to stop blaming the model for it. Let me show you exactly what I mean, because it happened to me last week.

The report that didn't need to exist

We were adding multi-business support to payouts. One parent account, several businesses underneath it, each needing its own slice of the flow.

The agent did the core work well. I want to be clear about that. On the multi-business payout logic itself, the code was correct. No complaints.

Then I asked for a small follow-on: add callback parameters and a few dynamic fields into the payout report.

The agent went and built a new reporting service. New repo, fresh scaffolding, a whole new report pipeline. It worked. It compiled, it ran, it produced exactly the report I'd asked for.

One problem. We already had a payout report — the single-business one, live in production, tested against real settlements for months. The right move was to extend it to handle multiple businesses and the new fields. Instead I now had two report implementations for the same domain. One battle-tested, one freshly minted. And every future change to payout reporting would either be made twice or silently drift apart.

That's not a bug. Nothing broke. It's worse in a quieter way. It's code-quality debt the agent created because it couldn't see that the thing it was building already existed.

This is the part I keep coming back to. It's not only about bugs. It's about code quality. A blind agent doesn't just get logic wrong. It reinvents, it duplicates, it walks past the shared helper and writes a fourth one. Every instance of that is a small tax you pay forever.

Why a senior wouldn't have done this

A coding agent is the best junior developer money can buy. Tireless, cheap, no ego, near-zero mistakes on routine work. An army of them at your terminal.

But juniors write code. They don't ship products.

A senior handed that report task would have opened with a different question. Not "how do I build a report" but "don't we already have one of these?" That instinct isn't in training data. It comes from having maintained a codebase long enough to know that a second implementation of anything is a liability, not a feature.

The agent never asked that question. Not because it's dumb. Because it only had my sentence. It had no way to know the single-business report existed, where it lived, or that extending it was the whole job.

So the problem stopped being "how do I get a smarter agent" and became "how do I put the judgment a senior carries in their bones somewhere the agent can read it — before it writes a line."

That question is why I built Bodhiorchard. It's open source, Apache 2.0, self-hosted. We run it at Atoa now.

We stopped handing agents a prompt. We hand them a BUD.

The unit of work in Bodhiorchard isn't a Jira ticket. It's a BUD — a Business Understanding Document. One living source of truth for a feature: what it is, why it exists, who asked for it, the design, the tech spec, the decisions made along the way. Vector-indexed, full history, no scatter across Confluence and Slack and someone's memory.

The point of a BUD is that the context is attached to the work, not floating around it. And crucially, Bodhiorchard traces a feature down to the actual code that implements it. A feature isn't a paragraph. It's a paragraph wired to the files, services, and procedures that make it real.

Work moves through seven phases, each its own tab on the BUD:

Requirements → Design → Tech Spec → Development → Code Review → Testing → Prod

Every phase has an agent. Every phase also has a toggle. Flip the agents off and it says "you're driving this BUD" — human in the loop, by design. This was never meant to run while you sleep.

The same task, inside Bodhiorchard

So I gave it the same job. Add callback params and dynamic fields to the payout report. Same class of model. Same one-line request.

This time it didn't start blind. The request came in against the payout feature's BUD, and because the BUD is wired to code, the agent pulled the context over MCP and found the existing single-business report — the real procedure, in the real repo — before it wrote anything.

# The agent reads context before writing — and finds what already exists
POST /mcp   → get_bud_context(bud_id: "PAYOUTS-217")

{
  "feature": "Payouts — multi-business support",
  "request": "add callback params + dynamic fields to the payout report",
  "precedent": {
    "report": "services/reporting → PayoutReport (single-business, live in prod)",
    "guidance": "extend this. do not stand up a second report pipeline."
  },
  "domain_rules": [
    "One report implementation per domain",
    "Reporting reads the ledger projection, never writes to it"
  ]
}

It extended the existing report. Multi-business, dynamic fields, callback params, added to the code that was already there. No new service. No second pipeline. One report that now does more.

The difference wasn't intelligence. It was sight. One agent could see the codebase and one couldn't.

And notice what the agent gets to do versus what it doesn't. The MCP write tools are bounded to the creative phases — it can create and update a BUD, draft the spec, propose the code. Nothing that moves money is a tool an agent can call directly. Read and propose across everything. Execute nothing with a blast radius. That line is drawn on purpose.

Behind the context sits a mechanical gate. Design-pattern lints run in CI, and if an agent reaches for a banned pattern or stands up a duplicate where an ADR says there should be one, the build fails. Not "a reviewer might catch it." The gate fails, every time. And the test suite leans hard on the negative cases agents love to skip — the illegal state transition, the duplicate event, the report that should have been an extension.

The scoreboard is pointed at the thing AI is worst at

Bodhiorchard has a scoreboard, and it does not reward output. It rewards shipped, working value.

You earn points when a BUD reaches production and when your code-review quality holds above a threshold. You lose a point for a bug caught in testing, and more for one that reaches production. The payout only lands when the feature is closed end-to-end, not when the PR merges.

Read that incentive again. It pays for correctness that survives production and for quality that holds up in review — exactly the things a fast, blind agent erodes when it's optimising for a green checkmark. Volume of code earns you nothing. A duplicate report that passes its own tests earns you nothing. Shipping the right thing does.

The honest status

I'm not going to tell you the agents run the whole SDLC while we sleep. They don't. The platform, the BUD lifecycle, the MCP context path, the feature-to-code indexing, the skill profiling — that's live and we use it daily. A fully autonomous execution loop is still being built. Today it's agent-assisted with a human in the loop at the phases that carry weight, and for anything touching payments that's exactly where I want it.

Where do agents own a boundary? Reading context, finding precedent, proposing a spec, drafting code, linting against recorded decisions. Where do they never own one? Anything where the failure mode is money in the wrong account. That's not a limitation I'm apologising for. That's the design.

So, back to blindness

The new report was never a model failure. Neither was the duplicate-refund class of bug before it. Both came from the same place: handing something a blank prompt and hoping it would guess the code you'd already written and the decisions you'd already made.

Give it the context a senior would carry into the review — the domain rules, the precedent, the boundaries, the code that already exists — and put a mechanical gate behind that context so nobody, human or agent, ships past it. The agent stops reinventing your codebase and starts extending it. It stops being a liability and starts being the army your seniors always wanted.

The 80% was already solved. This is how you stop the other 20% from shipping bugs — and from quietly rotting your code quality — while you're not looking.

What's the last thing an agent rebuilt in your codebase that already existed — and where was the context it needed to find it?

I'm Arun, CTO and co-founder at Atoa, building open banking payments for the UK. I built Bodhiorchard, it's open source, and we run it in our workflow. Repo's here if you want to poke at it — feedback over stars: github.com/mickyarun/bodhiorchard. Find me on X @mickyarun.

I Got Obsessed With Killing Scrum. So I Built a World Out of 12 AI Agents.

arun rajkumar — Sun, 12 Jul 2026 16:09:21 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

I built Bodhiorchard — an open-source platform that replaces Scrum, Jira, and the team wiki with 12 specialised AI agents and a living 3D world you can actually walk around in.

I call the methodology Agent-Driven Development (ADD). Instead of story points, planning poker, and a graveyard of stale Confluence pages, the work lives in one place: a BUD (Business Understanding Document) — a single, vector-indexed source of truth that carries a feature from a Slack conversation all the way to production.

The agents handle the busywork I never wanted humans doing: triage, design drafts, tech plans, standups, test plans, estimation, retros. Your repos show up as trees in an orchard. Features are branches. The whole team walks around as avatars — WASD to move, emotes, houses you upgrade with points you only earn when something actually ships to prod and stays healthy.

The scoreboard is the part I'm proudest of: it rewards quality, not output. Ship a BUD to production: +1. A bug escapes to prod: -1. It scores the things AI is bad at and humans are good at.

Honest status: the platform, the BUD lifecycle, the code-dependency graph, skill profiling, and the 3D world are all live. The fully autonomous execution loop is still being built — today it's agents-assisted, human-in-the-loop. I'd rather undersell it than lie to you.

Demo

Live site + walkthroughs: https://bodhiorchard.ai
Teaser — inside the virtual world: https://www.youtube.com/watch?v=OxoqBI7BNxU

Two views ship today: GARDEN (the orchard) and GRAPH (a cross-repo dependency graph with Bus Factor, Threats, and BUD-stage lenses). Two Slack bots sit on top — one triages new requests and kills duplicates before they become tickets; the other answers plain-English questions like "are we on track for go-live?" straight from the live BUD.

Code

Open source, Apache 2.0, self-hosted — it runs on a Mac mini.

mickyarun / bodhiorchard

Bodhiorchard is the open-source, self-hosted alternative to Jira and Linear for AI-native software teams. Specialised agents draft every spec, forecast cycle times, and reason over your real repositories, commits, and pull requests — replacing story points, standups, and stale tickets so engineers can focus on building.

Bodhiorchard™

Ship software, not Scrum ceremonies.

The open-source, self-hosted alternative to Jira & Scrum — AI agents run the process end-to-end, developers earn XP for what actually ships, and your data never leaves your machine.

Website · Quick Start · Why Bodhiorchard · The Twelve Agents · Docs · FAQ

▶ Watch the demo — a Slack message becomes a scoped, estimated BUD. No sprints, no story points, no standups.

Bodhiorchard replaces sprint, scrum, and Jira ceremony with Agent-Driven Development (ADD) — twelve specialised AI agents handle the busywork (triage, specs, estimates, test plans, retrospectives) while humans keep the decisions that matter, and developers earn XP for the work that actually reaches production. It's a self-hosted Jira alternative for the full lifecycle: intake → spec → design → development → testing → deploy → retrospective. The data plane stays on your hardware; inference runs through your choice of agent CLI…

View on GitHub

How I Built It

This is a solo project. Nights and weekends, on my own, because I couldn't leave it alone.

The stack: FastAPI + Python 3.12 on the backend, Vue 3 + PlayCanvas for the 3D world, Postgres + pgvector for the BUD memory, and Redis underneath. Inference runs through Claude Code today, with Ollama and OpenAI on the roadmap so you can bring your own model.

The hard part wasn't the agents. It was the ontology — deciding that a BUD, not a ticket, is the atom of work, and that everything (estimates, design, tests, the retro's estimated-vs-actual drift table) hangs off that one object. Once that clicked, the agents became small. Each one reads BUD context over a local MCP server and writes back only in the three creative phases where a human is still driving.

The 3D layer looks like the toy. It's actually the point. A dependency graph as a spreadsheet is a chore. A dependency graph as an orchard you can lose an afternoon in — that's the thing that made me want to open it every day.

The Passion Angle

I've spent years watching good engineers spend half their week on Jira instead of building. That quietly made me angry. This is the project I built to get that time back — mine first, then anyone else's who wants it.

"Build well. Then go outside." That's the whole idea.

Bodhi means awakening. An orchard is a grove you tend on purpose. The name is the pitch.

If you've ever felt sprints were broken and wanted to do something about it instead of just complaining in retro — clone it, break it, tell me where I'm wrong. That feedback is the only prize I'm actually chasing.

Are We in an AI Bubble? We've Built This Exact One Twice Before.

arun rajkumar — Thu, 09 Jul 2026 13:37:41 +0000

Railroads and dark fiber ran this exact playbook. Both times the technology won and the builders died. Here is why the AI build-out looks identical, and why your RAM got more expensive because of it.

I build on this stuff for a living. My company runs on cloud infrastructure. We pay for AI tokens every month. And a few weeks ago I looked at a quote to add memory to a machine and did a double take. The price had nearly doubled in a year.

That RAM quote is where this story starts. Because the reason memory got expensive is the same reason people keep asking if AI is a bubble. And to answer that honestly, you cannot just look at AI. You have to look at the two times we did almost exactly this before.

The pattern is older than the internet

Infrastructure bubbles all rhyme. Someone spots a genuinely world-changing technology. Money floods in. And then comes the move that defines every one of these episodes: companies build capacity far ahead of real demand, betting that demand will show up to fill it.

Sometimes it does. Usually it does not, at least not on the schedule the spending assumed. The gap between capacity built and capacity used is where the money dies.

We have run this experiment twice at national scale. First with railroads. Then with fiber. We are now running it a third time, with data centers.

Railroads: the original overbuild

Britain went first. During the Railway Mania of the 1840s, Parliament passed 263 acts in the single year of 1846 to set up new railway companies, for routes totalling around 9,500 miles. A lot of that track was planned on the assumption that traffic would arrive later. Much of it never earned back what it cost.

America did it bigger. Between 1866 and 1873, roughly 35,000 miles of new track were laid across the country, a lot of it financed on optimism and shaky debt. Then the bill came due. In the Panic of 1873, 89 of the country’s 364 railroads went bankrupt, and around 18,000 businesses failed in two years.

We did not even learn from it. Twenty years later, the Panic of 1893 hit for the same reason: too much railroad, financed too loosely. By mid-1894 a quarter of all US railroads had failed, more than 40,000 miles of them.

The technology was real. Railroads did reshape the world. But being right about the technology did not save the people who overbuilt it.

Fiber: capacity nobody could use

Fast forward a century. During the late-1990s telecom boom, companies laid more than 80 million miles of fiber optic cable across the US. The pitch was a number WorldCom kept repeating: internet traffic was doubling every 100 days. It was not. Real traffic was roughly doubling once a year, which is fast, but nowhere near the story the spending was built on.

So what happened to all that glass? Barely any of it got used. By some estimates less than 5% of the fiber laid in the boom was ever lit. Even four years after the bubble burst, 85% to 95% of it was still sitting dark. The industry literally invented a name for capacity built and never used: dark fiber.

The companies that laid it mostly did not survive to enjoy it. Global Crossing raised billions to wire the planet, then filed for bankruptcy in January 2002 with $12.4 billion of debt and thousands of miles of unused cable, after its chairman had quietly sold more than $700 million of his own stock. 360networks completed a $900 million IPO and was bankrupt about fourteen months later. WorldCom went down in July 2002 as the largest bankruptcy in US history at the time, $107 billion in assets, held up by roughly $11 billion of accounting fraud. Corning, which made the actual glass, fell from nearly $100 a share in 2000 to about $1 in 2002. All in, telecom stocks lost more than $2 trillion in value.

And the purest version of the madness was next door, in dot-coms. Pets.com went public in February 2000 having booked $619,000 of revenue while spending $11.8 million on advertising. It sold products for about a third of what it paid for them. It went from IPO to switching off the lights in 268 days, its stock from $11 to 19 cents, roughly $300 million burned. It is a punchline now. At the time it was a stock people lined up to buy.

Same shape as the railroads. Real technology. Genuine future. Ruinous timing.

AI: same script, much bigger budget

Now look at what is happening today.

The five biggest US cloud and AI players, Microsoft, Alphabet, Amazon, Meta and Oracle, have signalled combined capital spending of roughly $660 to $690 billion for 2026, close to double what they spent in 2025. Across the largest data center operators globally, the spend is heading toward $750 billion in a single year, and about three quarters of that, roughly $450 billion, is tied directly to AI: the chips, the servers, the buildings.

Each of these data centers costs billions. They are being built now, at full speed, for demand that is expected later. That is the tell. It is the railroad move and the fiber move, in concrete and silicon.

So here is the fair question. Is the money actually coming in to justify it?

Look at the two names everyone points to. OpenAI is running at roughly a $24 billion revenue rate as of early 2026, and still projecting a $14 billion loss for the year, with no profit expected before 2029 or 2030, while preparing to ask investors to value it above a trillion dollars. Anthropic has grown fast too, to around a $30 billion annual run rate.

Add up the frontier labs and you are somewhere in the range of $50 to $60 billion of annual revenue. Set that against roughly $450 billion of AI infrastructure spending in a single year. The revenue is real and growing quickly. It is also nowhere near what the build-out needs to break even.

And the demand everyone is counting on, enterprise adoption, is not showing up on schedule. An MIT report last year, looking at 300 deployments and 150-plus executive interviews, found that despite $30 to $40 billion of enterprise spending on generative AI, about 95% of organisations were seeing no measurable return. Only 5% were getting real value. Everyone is building for the enterprise wave. The enterprise wave, so far, is mostly stalled pilots.

This is why your laptop got more expensive

Here is the part that reaches people who never touch a data center.

Three companies, Samsung, SK Hynix and Micron, make over 95% of the world’s DRAM, the memory in your laptop and your phone. AI accelerators need a special, more expensive kind of memory called HBM, and it is far more profitable to make. So those three quietly shifted capacity toward HBM for the AI build-out and away from the ordinary memory the rest of us buy.

The result is a squeeze. Data centers now consume an estimated 70% of the memory chips produced worldwide. Prices have jumped hard. Samsung pushed a 32GB DDR5 module from about $149 to $239, a 60% jump. Contract prices for DDR5 have more than doubled. Samsung and SK Hynix are warning the shortage runs into 2027 and beyond, with customers reserving supply years ahead.

That is the bubble touching your wallet. You did not buy an AI product. But the AI build-out bid up the memory in the device in your pocket, and you are paying for it anyway.

So who actually won last time?

This is the twist worth sitting with.

When the fiber bubble burst, all that dark fiber did not vanish. It got sold off cheap. And the companies that scooped it up were not the ones that laid it. Google, Amazon and Facebook bought or leased that surplus capacity for a fraction of what it cost to build, and used it as the backbone for search, cloud and video. The infrastructure outlived the companies that funded it, and the winners were the ones who showed up after the crash with cash and a use for it.

That is the honest hopeful side. Demand for fiber did eventually catch up. It just took more than a decade, and it arrived long after the original investors were wiped out. The glut was real. The future was also real. Both things were true.

But this time the index is not made of Pets.coms

Here is the fairest objection to all of this, and it is a strong one.

The companies driving today’s build-out are nothing like Global Crossing or Pets.com. The fiber and dot-com bubbles floated on businesses with thin profits, sometimes almost no revenue, that needed a constant drip of fresh capital just to breathe. When the capital stopped, they were gone in months.

The names leading the AI spend are the opposite. Microsoft, Alphabet, Amazon, Meta, Nvidia and Apple are some of the most profitable companies that have ever existed. The so-called Magnificent Seven throw off something like 70% of the entire economic profit of the S&P 500. They are funding most of this build-out from the cash their existing businesses already generate, not from speculators who can vanish overnight. Cisco traded at 200 times earnings in 2000 on a far shakier story than any of these carry now.

So no, Google is not going to evaporate the way 360networks did. These giants can be wrong about AI for years and still be standing. That is a real difference, and it matters.

But look closely at what that argument actually says. It says the incumbents survive. It does not say the spending pays off. A profitable company can pour hundreds of billions into capacity that demand never fills, take the writedown, and walk away intact. The build is still an overbuild. The money is still gone. It just does not take the company down with it.

And the speculative layer has not disappeared. It has moved. Instead of Pets.com, this cycle has AI pure-plays and neocloud GPU landlords raising on the promise of the future, some of them buying chips with debt against contracts that only hold if the boom holds. If there is a Global Crossing hiding in this story, it is probably not Google. It is one of the names you only started hearing two years ago.

So, are we in a bubble?

Put it plainly. On the evidence, yes. Capacity is being built years ahead of the revenue and the adoption that would justify it, increasingly on debt, on a demand story that has not yet arrived. That is the exact pattern that broke railroads and telecom.

But a bubble does not mean everyone dies. The last two times, the technology was real and the future did arrive. It just arrived for different people than the ones who paid for it, on a slower clock than the spending assumed. The builders ate the loss. The infrastructure, and the payoff, went to whoever was still standing afterward with cash and a use for it.

The technology is not the bubble. The technology is probably the most important thing many of us will build on in our careers. The bubble is the belief that demand will arrive exactly when the spending needs it to. And “this time it is actually useful” is the line every bubble tells, right before it proves that being right and being early are two separate ways to go broke.

So no, I am not betting against AI. I am building on it. But when I sign off on that doubled memory quote, I remember how the last two of these ended.

Railroads were real. The people who overbuilt them went bankrupt, and someone else ran the trains. Fiber was real. Corning went from $100 to $1, and Google bought the cable for scrap. Both times, the future showed up exactly as promised. It just showed up for different people than the ones who paid for it.

That is the part nobody spending $450 billion this year wants to hear. The winners of the AI era may not be the names on today’s invoices. They may be whoever is standing there with cash and a use for it after the correction, buying the future at the price of scrap.

Twice is a coincidence. We are about to find out if three is a pattern.

Your Types Lie the Moment They Cross a Network Hop

arun rajkumar — Tue, 07 Jul 2026 12:47:04 +0000

A TypeScript type is a promise. And like any promise, it only holds while someone is around to keep it.

That someone is the compiler. It watches your code, checks every shape, and yells the second something does not line up. It is very good at its job. But it has one hard limit nobody tells you about on day one. It stops at the edge of your process.

The moment your data leaves the building, the compiler is gone. Over the network. Off a queue. Out of a webhook. Back from the database. Your type is still sitting there in the code, looking confident. But now it is just a sticky note that says "trust me." Nothing is checking it anymore.

I learned this the annoying way.

The bug that had no bug

We had two services talking to each other. One sent a payment amount. The other received it. Both sides shared the same type:

interface PaymentEvent {
  amount: number;
  currency: string;
}

Clean. Typed on both ends. The compiler was happy everywhere I looked.

Then one day the numbers in a report were slightly off. Not crashing-off. Just wrong enough to notice. No error. No stack trace. Nothing red anywhere.

Took us a while to find it. The sending service had started putting amount out as a string. "1000" instead of 1000. Somewhere upstream a value got serialized, went through a queue, and the quotes crept in.

On the receiving side, TypeScript still believed amount was a number. Because we told it so. It never looks at the wire. So "1000" * 100 did something silly, and the type sat there the whole time swearing everything was fine.

The type did not lie on purpose. It just was not there when it mattered.

Where your types are actually guessing

Here is the thing that clicked for me. Inside one process, types are real. The compiler saw the value get made and it saw it get used. It has the whole story.

But the second data crosses a boundary, your type is a guess. A hopeful one. These are the spots where it is guessing:

Anything you JSON.parse
A response from another service or a third-party API
A message off a queue or an event bus
A webhook you did not send
A row from the database
process.env (every one of those is a string, even the ones you treat as numbers)

In all of those, you hand TypeScript a blob of data and say "this is a PaymentEvent, take my word for it." And it does. That is the whole problem. It takes your word.

The fix is boring, and that is why it works

You stop trusting the label and you check the actual shape at the door. Every time data comes in from outside, you parse it before you use it.

We use Zod for this. You write the shape once as a schema. You get two things back from it: a real runtime check, and the type.

import { z } from "zod";

const PaymentEvent = z.object({
  amount: z.number(),
  currency: z.string(),
});

type PaymentEvent = z.infer<typeof PaymentEvent>;

Then at the boundary:

const event = PaymentEvent.parse(incoming);

If amount shows up as "1000", this throws right there, at the door, with a clear message. Not three services later in a wrong report. The bad data never gets in.

And look at the last line of the schema. The type comes from the check. You do not write the type by hand and hope the data matches it. The check is the source of truth, and the type just follows along. They cannot drift apart, because they are the same thing.

The rule we live by now

Trust types inside your own process. That is what they are for, and they are great at it.

The moment data comes from somewhere else, treat the type as a wish until you have checked it. Parse at every boundary. Build the type from the parser, not the other way round.

A type tells you what you meant. A parse tells you what you got. On a payments system, the gap between those two is real money.

So next time a value crosses a wire and lands in a nice typed variable, ask one question: who actually checked this? If the answer is "the compiler," the compiler already went home.