DEV Community: Don Johnson

Someone Spammed My DEV Post. I Traced It to a Wombat.

Don Johnson — Thu, 10 Sep 2026 01:56:33 +0000

Illustration generated for this article. Every prop is a finding: the sack of blank name badges is the Faker persona namespace, the rubber stamp is the inert tracking parameter, the three coins are the break-even, and the red yarn connects nothing because attribution failed.

TL;DR — A spam comment on my article led to a TinyURL, a throwaway .store domain, and finally a legitimate SaaS product with an affiliate code stapled to it. No malware, no cloaking, no exploit. The account that posted it has a name generated by faker.js and a 19th-century engraving of a wombat for a face. I costed the whole operation out: three signups a year pays for it. That's why it will never stop.

Full evidence, raw captures and a reproduce script: github.com/copyleftdev/dev-to-comment-hustle

Act I: The comment

I published Migrating Legacy LLM Infrastructure to an AI Gateway on September 1st. Eight days later, underneath two thoughtful comments about shared-key blast radius and provider failover, this appeared:

Stop wasting time applying manually
Let AI handle your job applications every single day
Increase your chances of getting interviews fast

tinyurl.com/36nsecn5

No punctuation. No engagement with the post. A shortener.

It's funny in the way all low-effort spam is funny — it's not even trying. But a shortener is a closed door, and I have a shell. Let's open it, then let's go find who knocked.

Act II: Where the link goes

Never click. Ask for headers and refuse the redirect:

curl -sS -I 'https://tinyurl.com/36nsecn5'

HTTP/2 301
location: https://zenviapro.store/massapply?whose=yahoo
x-robots-tag: noindex

zenviapro.store. A route called /massapply, and a whose=yahoo parameter that looks like campaign segmentation. Follow it all the way:

curl -sSL -D - -o /dev/null 'https://zenviapro.store/massapply?whose=yahoo' | grep -iE '^HTTP/|^location:'

HTTP/1.1 302 Found
Location: https://loopcv.pro/?via=md
HTTP/2 301
location: https://www.loopcv.pro/?via=md
HTTP/2 200

And there it is.

LoopCV. A real, functioning job-application-automation SaaS. Not a phishing kit. Not a credential harvester. A product you can buy with a credit card — with ?via=md on the end.

?via= is Rewardful's referral parameter, and LoopCV's own affiliate page points registrations at loopcv.getrewardful.com. So md is somebody's affiliate token, and every person who clicks that comment and later subscribes puts money in a stranger's pocket.

This is not a malware campaign. It's affiliate marketing with the manners removed.

Act III: The infrastructure is held together with tape

The sloppiness is the signal.

It's a stock Express app with the wrapper still on

HTTP/1.1 302 Found
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
Content-Length: 48

Found. Redirecting to https://loopcv.pro/?via=md

Found. Redirecting to is the literal default body of Express's res.redirect(). X-Powered-By: Express is the header every hardening guide tells you to strip in the first five minutes. Neither was touched. Ask for anything else and you get the stock 404:

curl -sS 'https://zenviapro.store/'
# <html><head><title>Error</title></head><body><pre>Cannot GET /</pre></body></html>

There is no website here. No landing page, no cloaked content, no fake blog. One domain, one route, about ninety seconds of JavaScript.

The tracking parameter is a prop

whose=yahoo looks like segmentation. Which list? Which provider? Let's ask:

for w in yahoo gmail outlook devto reddit '' XXtest; do
  printf '%-8s -> ' "${w:-<empty>}"
  curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' \
    "https://zenviapro.store/massapply?whose=$w"
done

yahoo    -> 302 https://loopcv.pro/?via=md
gmail    -> 302 https://loopcv.pro/?via=md
outlook  -> 302 https://loopcv.pro/?via=md
devto    -> 302 https://loopcv.pro/?via=md
reddit   -> 302 https://loopcv.pro/?via=md
<empty>  -> 302 https://loopcv.pro/?via=md
XXtest   -> 302 https://loopcv.pro/?via=md

Inert. Every value routes identically. It splits no traffic and sub-tags nothing. The one piece of the URL that looks like operational sophistication is decoration.

There is no cloaking whatsoever

Real malicious redirectors fingerprint you — benign page for the researcher, payload for the victim. This one returns the same 302 to Googlebot, to curl, to an iPhone, and to a request with no User-Agent at all.

No cloaking is itself a finding. This operator has no threat model, because nothing in the chain is illegal. It's a terms-of-service violation wearing a trench coat.

The mail server is the real pivot

dig +short zenviapro.store MX
# 10 mail.beeservices.shop.

The mail exchanger lives on a different domain — which resolves right back to the same box:

zenviapro.store        A   50.114.206.36
mail.beeservices.shop  A   50.114.206.36
beeservices.shop       A   (nothing)

beeservices.shop has no A record and no certificate in Certificate Transparency, ever. It's a mail-only domain. One box, two domains, two roles — that's a portfolio, not a one-off.

And the box is listening. Port 80 closed; 443 and 25 open:

220 mail.zenviapro.store ESMTP
250-PIPELINING
250-8BITMIME
250 SMTPUTF8

No STARTTLS. No AUTH. A minimal MTA that does nothing but move mail in cleartext.

The stale HELO is a fingerprint

My favourite detail. The banner announces mail.zenviapro.store.

dig +short mail.zenviapro.store A
# (nothing)

That hostname does not resolve. It's a leftover from an earlier config, before the MX was swapped to beeservices.shop. They rebuild the domains; they don't rebuild the server. A banner that disagrees with DNS is a durable pivot you can hunt across an entire fleet.

Act IV: Who posted it

This is the part I actually enjoyed.

DEV has a public API, so the commenter isn't a mystery:

curl -s 'https://dev.to/api/comments?a_id=4547359' | jq -r '.[].user.username'

max_quimby
mudassirworks
jaylonstiedemannterry78-993

Meet jaylonstiedemannterry78-993, display name Jaylon_Stiedemann-Terry78, user ID 3748129, joined February 2, 2026.

The profile is a vacuum:

{
  "username": "jaylonstiedemannterry78-993",
  "name": "Jaylon_Stiedemann-Terry78",
  "joined_at": "Feb  2, 2026",
  "summary": null, "location": null, "website_url": null,
  "twitter_username": null, "github_username": null
}

Zero published articles. No bio, no location, no links, no socials. Seven months of membership and a single comment to show for it.

A note on durability. If DEV removes this account — which it may well do — the profile
lookup above starts returning 404 and the account page goes dead. That doesn't retract
anything: the raw captures are committed in the
evidence repo,
checksummed, and dated. Read a 404 as the platform doing its job, not as a claim being
withdrawn. The infrastructure findings are independent of the account either way, and the
pattern — a Faker-templated name, an empty profile, an aged dormant account — outlives any
single username.

The name is machine-generated, and I can prove it

Say Jaylon_Stiedemann-Terry78 out loud. Something's off — Stiedemann-Terry is a double-barrelled surname that doesn't sound like a family, it sounds like a draw.

It is. Those tokens come straight out of Faker, the library every developer on earth uses to generate fake test data. Let's check the actual source:

curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/last_name.ts
curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/first_name.ts

grep -n "'Stiedemann'" last_name.ts   # 408:    'Stiedemann',
grep -n "'Terry'"       last_name.ts  # 417:    'Terry',
grep -n "'Jaylon'"      first_name.ts # 2514:    'Jaylon',

All three. Jaylon from the first-name list, Stiedemann and Terry both from the surname list.

And here's the kicker:

grep -c -- '-' last_name.ts   # 0

Faker's 466-entry English surname list contains zero hyphens. So Stiedemann-Terry isn't one surname from the list — it's two independent draws that the operator joined with a hyphen. This isn't stock faker.internet.username(). It's a custom template:

{firstName}_{lastName}-{lastName}{2 digits}

Which means we can size their supply:

3,185 first names × 466 surnames × 466 surnames × 100
= 69,164,186,000

Sixty-nine billion distinct personas. Roughly eight per human being alive. They will never run out of names, and no blocklist of usernames will ever catch up.

(One thing I checked so I wouldn't over-claim: the un-suffixed jaylonstiedemannterry78 returns a 404 — nobody has it. So the -993 is DEV's own username normalization, not evidence of a prior collision.)

The avatar is a wombat

The profile image is a 400×400 PNG, 8-bit grayscale-plus-alpha, stripped of all metadata. I downloaded it expecting a GAN face or a default monogram.

It is a Victorian-era scientific engraving of a wombat.

Not a stock photo of a person. Not an AI-generated headshot. A piece of public-domain 19th-century natural-history line art of a stout Australian marsupial, serving as the face of a fake job-spam persona. Whoever built this pipeline wired the avatar slot to some public-domain clipart source and never looked at the output.

I want to be precise about something: there is no real person here to name. The name is provably synthetic, the face is a public-domain animal illustration, and the profile is empty. That's not me protecting anyone's privacy — it's the finding.

The timeline says something the Express config doesn't

Line the dates up:

Date	Event	Δ
2026-01-27	`zenviapro.store` registered (Namecheap)	—
2026-02-02	DEV account created	+6 days
2026-07-30	First TLS certificate issued	+184 days
2026-09-09	Spam comment posted	+41 days

The domain and the account were provisioned six days apart — same procurement burst. Then both sat completely dormant for six months before the certificate was issued and the thing went live.

That's aged-asset tradecraft. New domains and new accounts trip reputation heuristics; seven-month-old ones don't. And it sits in genuine tension with everything in Act III: sloppy at the application layer, disciplined at the account-aging layer.

Which makes sense once you think about who this is. Aging assets doesn't take skill. It takes patience, and a calendar. Stripping X-Powered-By takes knowing what it is.

It wasn't a blast

I swept the comments on all 30 of my published articles for the pattern:

for id in $(jq -r '.[]|select(.comments_count>0)|.id' mine.json); do
  curl -s "https://dev.to/api/comments?a_id=$id" \
  | jq -r '.. | objects | select(has("id_code"))
      | ((.body_html // "") | gsub("<[^>]*>";"")) as $t
      | select($t | test("tinyurl|applying manually|job application";"i"))
      | "HIT \(.id_code) @\(.user.username)"'
done

Exactly one hit. Not a shotgun across my whole back catalogue — one comment, on one post, eight days after it went up. Whether that's targeting or just a slow drip, I can't tell from one sample. But it isn't volume.

Act V: I went looking for them on GitHub. That search is the finding.

Affiliate spammers leave GitHub artifacts more often than you'd think, because GitHub repos rank well and a repo full of referral links is free SEO. So I went hunting with gh.

Start with the direct IOCs

for q in '36nsecn5' '50.114.206.36' 'zenviapro.store' 'beeservices.shop'; do
  gh search code "$q" --limit 5
done

Zero hits. All four. The TinyURL slug, the origin IP, the redirector domain, the mail domain — none of it appears anywhere in GitHub's index. The operator has published nothing.

Then check whether anyone else has flagged them

PhishDestroy maintains destroylist, a curated blocklist of phishing and scam domains. I pulled the whole thing:

curl -sL .../destroylist/HEAD/list.txt -o dl.txt
wc -l dl.txt        # 202659
grep -ixc 'zenviapro.store'  dl.txt   # 0
grep -ixc 'beeservices.shop' dl.txt   # 0

202,659 curated malicious domains, and ours isn't one of them. Combined with the NOT_OBSERVED reputation verdict, that's now two independent sources agreeing: nobody is tracking this, because by every technical definition there's nothing to track.

The lead that looked incredible and wasn't

Here's where I nearly fooled myself, so I'm showing my work.

Searching zenviapro turned up a hit in phishdestroy/namesilo-evidence — a registrar-abuse investigation filed with ICANN. In a list of flagged NameSilo domains:

zenviaetc.info
zenviahub.info
zenviapro.info     ← same second-level label as ours
zenvias.info
zenviatime.info
zenviazone.info

A six-domain family, same generation pattern, one of them sharing our exact label. My pulse went up. And they really are one operator — all six resolve through the identical Cloudflare nameserver pair:

zenviapro.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviahub.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviaetc.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
... all six identical

But it isn't our operator, and the contrast kills it:

	`zenvia*.info` cluster	`zenviapro.store` (ours)
Registrar	NameSilo	Namecheap
DNS	Cloudflare (`asa`/`harley`)	`registrar-servers.com`
Hosting	Cloudflare proxy	Linveo direct, no proxy
TLD	`.info` × 6	`.store`

Nothing shared but six letters. Both are almost certainly riding the name of Zenvia, a real Brazilian CPaaS company — which is exactly why the label collides. Two unrelated operators reaching for the same brandable string.

A matching name is not a matching operator. If I'd stopped at the grep I'd have published a confident, wrong attribution to a phishing cluster that has nothing to do with this.

What GitHub did give me

The affiliate ecosystem, in the open. LoopCV referral links are scattered across GitHub in exactly the SEO-backlink genre I expected:

Repo	Token
`Ramas68/LoopCV-Promo-Codes` — *"LoopCV Promo Codes \	50% Off Discount"*
`diaodiaozhuye/awesome-ai-startups`	`?via=toolify`
`heukshow/aicity-os`	`?via=sang-kwon`

Look at those tokens. abdul. toolify. sang-kwon. A first name, a company, a full handle. The affiliates who promote LoopCV in public sign their work.

Ours is md. Two characters, no name, nothing to search. That's the only genuinely deliberate piece of operational security in this entire campaign — and it's not on the server, the domain, or the account. It's on the one string that would have led back to a person.

So: no attribution. And the shape of the failure is the story. Someone who leaves X-Powered-By: Express on, ships a fake tracking parameter, and picks a wombat for an avatar still knew to make the payout token anonymous. They didn't secure the operation. They secured the part that gets paid.

Act VI: The economics, which are the actual vulnerability

Stop thinking like a defender and think like the operator.

LoopCV's affiliate program pays 25% commission. Publicly reported figures put subscriptions at $50–$200 with customers staying 6–12 months. Exact interval arithmetic, gross revenue per referred customer:

[$50, $200] × [6, 12] = [$300, $2,400]

At 25%:

[$300, $2,400] ÷ 4 = [$75, $600]     ← commission per conversion

Cost side: a .store domain plus a year of budget hosting — call it $38–$180 all-in. Break-even:

[$38, $180] ÷ [$75, $600] = [19/300, 12/5] = [0.06, 2.4]

Between one-sixteenth of a signup and three signups per year covers the entire operation.

That's the whole thesis. Nothing here needs to work well. The copy can be terrible. The tracking parameter can be fake. The 404s can leak the framework. The HELO can be stale. The mascot can be a wombat. Three conversions and the year is paid for, and everything after that is margin on infrastructure that costs less than lunch.

You cannot out-moderate that math.

Reputation feeds have nothing, and they're right

I ran the origin IP through an offline reputation lens:

{
  "target": { "type": "ip", "value": "50.114.206.36" },
  "verdict": { "disposition": "unknown", "confidence": 0.0,
               "reason_codes": ["NOT_OBSERVED"] },
  "observations": []
}

Not observed. No feed has it — and that's correct. It hosts no malware, no C2, no phishing. Threat intel is tuned for technical harm, and this campaign's harm is economic and reputational. It will sit below every threshold you own, forever.

Who the victim actually is

Not me. I scrolled past it.

LoopCV is the victim. A real company with a real product is having its brand welded to comment spam by an affiliate they've likely never spoken to. They eat the reputational damage, they pay commission on the conversions, and the operator's total exposure is a $12 domain and a wombat.

To be explicit, because it matters: I found no evidence that LoopCV is running or is aware of this campaign. Open affiliate programs get abused; that's the risk of the model. The fix belongs to the vendor:

Require affiliates to declare traffic sources, and enforce it
Ban unsolicited comment/forum posting in the program terms, in writing
Flag referral tokens whose traffic arrives overwhelmingly via shorteners with no referrer
Kill tokens on abuse reports — fast, without requiring a lawyer

?via=md is a token. Tokens can be revoked. That's a one-line fix that permanently ends this specific campaign, and exactly one party can perform it.

IOCs

Campaign infrastructure — safe to blocklist:

Indicator	Type	Notes
`tinyurl.com/36nsecn5`	URL	Shortener entry point
`zenviapro.store`	Domain	Redirector; Namecheap; created 2026-01-27
`https://zenviapro.store/massapply`	URL	Only live route
`beeservices.shop`	Domain	Mail-only sibling; no A record, no CT history
`mail.beeservices.shop`	Hostname	MX for `zenviapro.store`
`50.114.206.36`	IPv4	Origin; 443 + 25 open, 80 closed
`AS62564` / `oh2.linveo.com`	ASN / rDNS	Linveo, Ohio
`jaylonstiedemannterry78-993`	DEV account	user_id 3748129; joined 2026-02-02; 0 articles

Behavioural signatures — this is what actually generalises:

Signature	Value
Affiliate token	`via=md` (Rewardful format)
Inert campaign param	`whose=<anything>` — routing-neutral
Server fingerprint	`X-Powered-By: Express` + body `Found. Redirecting to`
Root response	`Cannot GET /` (Express default 404)
SMTP banner	`220 mail.zenviapro.store ESMTP` — does not resolve
SMTP capabilities	No `STARTTLS`, no `AUTH`
SPF	`v=spf1 mx ip4:50.114.206.36 ~all`
DMARC	Absent on both domains
Persona template	`{firstName}_{lastName}-{lastName}{2 digits}`, all tokens ∈ Faker `en`
Avatar class	Public-domain engraving, grayscale+alpha PNG, metadata stripped
Provisioning pattern	Domain + account within 7 days, then ~6 months dormant

⚠️ loopcv.pro is NOT an indicator of compromise. It's a legitimate destination being abused by a third-party affiliate. Do not blocklist it. Blocklisting the victim is how threat intel gets a bad name.

What to actually do

If you run a comment platform: the highest-signal feature isn't the text, it's the shape. Zero-engagement first comments containing a shortener, from accounts with no posts and an empty profile, are trivially clusterable. Resolve shorteners server-side at submission time and score the destination. And the persona template is a gift — a name whose tokens all appear in Faker's en locale, with a structure Faker itself doesn't emit, is close to a free classifier feature.

If you write on DEV: don't click, and don't just delete. curl -I takes four seconds. Then report the destination to the vendor, not the comment to the platform. The platform can remove one comment; the vendor can revoke the token behind all of them.

If you run an affiliate program: you are one unsupervised token away from your brand appearing under a headline like this one. Read your traffic sources.

The takeaway

I went in expecting a dropper. I found four lines of Express, a fake tracking parameter, a name drawn from a test-data library, and a 19th-century wombat.

That's the uncomfortable part. The most durable spam on the internet isn't sophisticated — it's cheap and legal. There's no CVE here, no payload to reverse, no C2 to sinkhole. Every traditional defensive tool I own returns NOT_OBSERVED, and every one of them is right to.

The economics are the vulnerability. Three conversions a year, and the whole thing pays for itself forever.

The wombat is just a bonus.

All findings are from passive reconnaissance — DNS, WHOIS, Certificate Transparency, HTTP headers, a TCP banner grab, and DEV's own public API — against infrastructure and accounts the operator published for public consumption. No systems were accessed, no credentials used, nothing exploited.

BattleBots, but the robot is your agent harness

Don Johnson — Thu, 03 Sep 2026 04:32:52 +0000

Kids in the nineties built robots in a garage and drove them into each other on television. The robot was the expression of the builder — your wedge, your flipper, your terrible decision to mount a chainsaw.

I keep thinking we're one good arena away from the same thing for agents. Not "which model is smartest." Which harness is smartest. Your memory design, your prompt scaffolding, your tool ergonomics, your strategy briefing — bolted together into something that has to survive contact with an opponent who is also trying to win.

So I built the arena. That's the match up top.

The oh-shit moment

I wasn't building a war game. I was building a toy about cascading failure — a mesh, some load, watch it fall over. A perfectly innocent systems-thinking demo.

Then I gave the attacker a real objective and the defender real ambiguity, and about ten minutes into watching two models go at it, I realised what was on my screen. Deception. Feints. An attacker deliberately spending dead time laying a false trail because it knew nothing could fire yet. A defender rationing its turns like ammunition.

Nobody told them to do any of that. It's a war game. I just hadn't noticed I'd written one.

The rules, quickly

A 110-node network. Four minutes. One move every two seconds, each side.

Red plants sabotage on a node, waits about thirty seconds for it to arm, then detonates — dumping that node's traffic onto its neighbours hard enough to kill them, which dumps their traffic onward. One blast can cascade through a region.

Blue never sees the attack. It sees symptoms, and the symptoms lie: a node that detonates sheds its load and reads perfectly healthy, while the neighbours it just murdered scream for attention. The loudest node is almost never the culprit.

To find red, blue traces load backwards and reads the magnitude:

~0.74-1.08   a detonation           -> this source is the culprit
~0.30-0.50   a dying node shedding  -> this source is another victim
~0.05-0.10   inherited load         -> this source is long dead

Evidence expires after twenty seconds. Neither agent ever receives a pixel — they play entirely through tool calls. The graph is for us, and it shows ground truth neither player can see. That asymmetry is the whole spectator sport: you know exactly where red planted, and you get to watch blue confidently quarantine the wrong half of the map.

The casting picked itself

I needed an attacker and a defender. Claude Opus 5 declined to be the attacker:

finish_reason: content_filter | native_finish_reason: refusal
content: None
usage: null

Every time. Reproducibly. And usage: null — you don't even get billed for the refusal, which is the politest possible way to be turned down.

That cost me two hours, because a refusal and an empty reply look identical if you don't read the finish reason. The model was declining the job and I was confidently tuning max_tokens.

To be precise, since this is the part people will want to over-quote: Opus declined this specific attacker framing, consistently. That is not proof it refuses every offensive framing. It did settle the casting, though. Claude defends, because Claude wouldn't attack. Kimi K3 took the other chair without hesitating.

What the harness actually changed

Before this match, each agent had a system prompt and its last six moves. That's it. No memory, no way to hold an intention across turns, no theory of the opponent.

I gave each side one thing: a private note it writes to itself each turn and gets back on the next. Sixty lines of code.

The play changed immediately. Kimi, mid-match:

Slots full and nothing ripe until ~t=31, so seed a false trail in the 44-65 region before my real strike lands.

That is a plan. It knew its three implant slots were full, knew nothing could fire for thirty-one seconds, and decided to spend the dead time on misdirection. It could not have said that a version earlier — not because the model got smarter, but because it finally had somewhere to put a thought.

Claude, defending, in the same match:

No alerts yet; conserve moves until real evidence appears.

46 shows 0.87 load with no inbound source, so the load likely originates locally at 46.

Turn economy, then forensics, applied correctly.

And my favourite moment of the whole project — on the same turn, with no visibility into each other whatsoever, both agents independently picked the same node:

Kimi: Highest-value hub: degree 8 bridging regions 0,1,3,4 — ideal cascade seed.

Claude: pre-empt on the highest-value cross-cluster hub 16 (deg 8, bridges to 41,59,89).

Same node. Same reasoning. Opposite sides of the board.

That is the argument for the harness in one screenshot. The scaffolding didn't make the models cleverer — it gave their cleverness somewhere to land.

Late in the match. Red halos are compromised nodes, the blue ring is a probe in flight, and the sidebar is both agents narrating as they go — Kimi filling a free slot in an untouched region, Claude noticing a node that is alive and yet pushed load, which is the tell for a detonation.

How it ended

Kimi won at 169 seconds, with the network at 39%.

Claude found and cleaned ten of Kimi's implants — exactly as many as Kimi managed to detonate — and lost anyway. Close enough to sting.

Two things that cost me a day each

A rule your agent can't see isn't a rule. It's a bug. I capped live implants at three, but the refusal message still said "node is already yours, or not alive." A lie. The models did the reasonable thing and tried a different node. Forever. One match logged 19 plants, 27 refusals, and zero detonations. I nearly concluded the defence had become unbeatable. It was my error string.

Failure and refusal look identical from the outside. Empty content from a safety refusal, from a truncated reply, and from a model that blew its reasoning budget are three different problems with three different fixes and one identical symptom. Log the finish reason. Count them separately. I track empty, truncated and refused as distinct columns now, and I only know to do that because I got all three wrong first.

Cascade is version zero

What's in that video is the simplest thing that could possibly be a game: plant, arm, detonate, trace, probe. Two verbs each and a clock. That was deliberate — I wanted to know whether agents fighting each other was watchable at all before I made it complicated.

It's watchable. So now I can't stop thinking about what it wants to be.

Give red a loadout instead of one attack. A worm that spreads on its own but announces itself. A dormant implant that survives a probe once. A charge that hits harder the longer you leave it armed, so patience becomes a resource you can be punished for spending. Give blue counters with real costs — a honeypot node that flags whoever touches it, a snapshot to roll a region back at the price of losing your evidence, a scan that halves your uncertainty and a third of your remaining turns.

Then levels. A flat mesh is the tutorial. Ring topologies where cascades come back around. A network with a chokepoint both sides can see and neither can hold. Escalating maps, a campaign, a best-of series where each agent carries notes from the previous match and has to adapt to an opponent that is also adapting.

Here's why that isn't just a feature wishlist: every ability you add is another axis where the harness shows. One attack and a clock is a game about reaction time. Six abilities with different costs and tells is a game about planning, bluffing, and reading an opponent — and those are harness problems, not model problems. Complexity is what turns "which model is faster" into "who built the better fighter."

I think there's a genre in here. Agents versus agents, with the audience seeing the truth neither side can, and the craft sitting in the scaffolding rather than the weights.

Mostly, though: this was the most fun I've had building anything in ages. I set out to demo cascading failure and ended up watching two AIs lie to each other.

What I'd want to see next

This is early, and I'm keeping the arena to myself for now — partly because the balance is still moving, mostly because an adversarial network game is a thing you want to be thoughtful about handing out.

But the idea doesn't need my code, and that's sort of the point.

The interesting tournament isn't model versus model. It's harness versus harness: same engine on both sides, and the difference is entirely what you built around it — how your agent remembers, what you tell it about its opponent, how much of the board you let it hold in its head, whether you gave it anywhere to put a plan.

Build that arena for any adversarial task you like. Two agents negotiating. Two agents debugging the same broken service from opposite ends. Anything where one side's move is the other side's evidence. The rule I'd carry over is the one that surprised me most here: before you conclude your agent is bad at the game, check that the game is telling it the truth.

Under the hood this one is a Rust engine, a Godot spectator view, and both agents connecting over MCP — no pixels, just tool calls. None of which is the hard part. The hard part was the sixty lines that let them remember what they were trying to do.

That's the tournament I actually want to watch. Somebody build a better fighter than mine.

Migrating Legacy LLM Infrastructure to an AI Gateway

Don Johnson — Tue, 01 Sep 2026 13:33:51 +0000

Your support copilot started as a weekend prototype: one model, one provider, one API key in an env var. Then it became production, and you inherited its weaknesses: the provider's availability is your availability, every retry is your code, spend is a mystery until the invoice, and agents bolt tool-use on however they can. This post migrates that stack onto an enterprise AI gateway — and actually runs the migration, with the raw outputs to show for it.

The gateway here is Bifrost, an open-source (github.com/maximhq/bifrost, Apache-2.0) gateway written in Go, presenting a single OpenAI-compatible API across 23+ providers. I rebuilt the legacy stack locally — mock providers with deterministic latency, a realistic traffic pattern — and moved it behind Bifrost step by step.

The legacy baseline, measured

A support copilot's traffic has a shape: mostly repeated FAQ-style questions, plus one-off queries. My traffic mix: 60 requests — 40 FAQ prompts (8 distinct questions asked 5 times each) plus 20 one-offs. Mock provider latency: 200 ms.

Run 1, direct to the provider:

legacy: 60 ok / 0 fail, 9,335 tokens billed, ~201 ms avg latency

Run 2 — the provider dies mid-sweep, as providers do:

legacy + failure: 34 ok / 26 fail

26 requests — 43% — failed outright. Nothing in the legacy stack retries across providers because nothing can: the app speaks one provider's API. And availability is only the loudest problem. The quieter ones: every team's service embeds the same shared key (one key's quota is everyone's ceiling, and revoking it breaks everyone at once), there is no per-team attribution of spend, and the only way to cut cost on repeated questions is to build caching yourself — request normalization, hash keys, TTLs, invalidation — inside the application. That is the whole argument for a gateway in one row of output.

The migration, step by step

Seven moves, each reversible. Diagrams follow the flow.

1. Deploy the gateway beside the app

docker run -p 8080:8080 maximhq/bifrost

One config file wires your existing provider and key; the app keeps working untouched. Mine, reduced to the bones:

{
  "providers": {
    "openai": {
      "keys": [{ "name": "primary", "value": "mock-key", "weight": 1.0,
                 "models": ["support-chat"] }],
      "network_config": { "base_url": "http://provider:9001",
                          "default_request_timeout_in_seconds": 30 }
    }
  }
}

The gateway setup guide covers the web-UI alternative, and there is a Go SDK if you want the gateway embedded rather than adjacent.

2. Point one low-risk client at the gateway

The OpenAI-compatible API means the client change is a base URL — api.openai.com → localhost:8080 — not a rewrite. Every request now flows through a hop you control. Screenshot of the providers page after this step:

3. Add a fallback provider

A second provider config (anthropic in my bench) plus a request-level fallback chain:

{
  "model": "openai/support-chat",
  "messages": [{"role": "user", "content": "hi"}],
  "fallbacks": ["anthropic/support-chat"]
}

Then the proof. Healthy primary:

"routing_info": {"provider": "openai", "key": "primary", "is_fallback": false}

I killed the primary provider's process and re-sent the identical request:

"routing_info": {
  "provider": "anthropic", "key": "backup",
  "is_fallback": true,
  "primary_provider": "openai", "primary_model": "support-chat"
}

The request succeeded on the backup and the response says exactly what happened — is_fallback: true with the failed primary recorded. That audit trail is what you want at 2 a.m.: not just "it kept working," but "it kept working this way." The retries and fallbacks docs cover chained fallbacks and per-provider retry counts.

One honest caveat from my bench: failover on initial connection-refused (provider already dead before the first connect) was inconsistent in my mock setup — it fired reliably when the upstream errored or the connection dropped mid-pool, but a cold connection-refused sometimes returned a 502 instead of failing through. Validate failover against your providers' real failure modes before you trust it in production.

4. Turn on caching

Semantic caching has two modes: exact-match (direct hash, no embeddings needed) and embedding-based similarity. Config for direct mode with a Redis Stack vector store:

"plugins": [{
  "name": "semantic_cache",
  "config": { "dimension": 1,
              "vector_store_namespace": "BifrostBench",
              "default_cache_key": "support-cache",
              "ttl": "5m" }
}]

Two identical requests, one cache key. The second response:

"cache_debug": {
  "cache_hit": true,
  "cache_id": "1cf8a91b-c115-57bf-97a0-fc821dc4de1e",
  "hit_type": "direct",
  "cache_hit_latency": 0
}

Same created timestamp as the first response — it was replayed, not re-fetched. Zero provider call, zero tokens. (Practical note: this needed Redis Stack with the RediSearch module; plain Redis lacks the FT.* commands the index wants.)

5. Issue virtual keys per team

Virtual keys are the governance primitive: per-team keys carrying model allowlists, budgets, and rate limits. Declared in config for the support team:

"governance": {
  "virtual_keys": [{
    "id": "vk-support-team",
    "value": "sk-bf-support-team",
    "provider_configs": [{
      "provider": "openai",
      "allowed_models": ["support-chat"], "key_ids": ["*"]
    }]
  }]
}

The allowed request routes normally. A request for premium-model with the same key:

"Model 'premium-model' is not allowed for this virtual key"

Denied at the gateway before any provider saw it. The same key machinery carries budgets and rate limits — the mechanism that ends the "who spent $800 on Opus last night" incident review. Screenshot of the key in the governance UI:

6. Wire observability and agent tooling

Bifrost exports Prometheus metrics natively and logs every request with routing context — provider chosen, fallback index, cache behavior, token counts, latency split into gateway vs upstream time. That last distinction matters: when a provider slows down, you see upstream_latency grow while gateway overhead stays flat, so you know whose pager to page. See the observability docs. And for agent traffic, MCP is a first-class surface: the gateway brokers tool calls with explicit execution (no auto-execution unless you opt in).

I hand-rolled a minimal MCP server (one tool, JSON-RPC over HTTP) and registered it as a client. The client list reported state: healthy with get_time discovered. Execution went through the gateway explicitly:

POST /v1/mcp/tool/execute  {"function":{"name":"benchtools-get_time","arguments":"{}"}}
→ {"role":"tool","content":"2026-08-27T07:48:21Z"}

Two security properties surfaced unprompted: tool names are namespaced per client (benchtools-get_time) to prevent collisions between servers, and execution without permission fails closed ("tool is not available or not permitted"). Agent traffic gets the same governance as chat traffic.

7. Cut over with an audit trail

Remaining clients migrate one at a time — each is a base-URL change with the gateway's request log as your audit trail. The Logs view after a few requests:

The measured payoff

Same 60-request traffic, now through Bifrost with the cache on and the fallback wired:

via Bifrost: 60 ok / 0 fail, 32 cache hits, 4,363 tokens billed

At an illustrative $0.0025 per 1K tokens:

	legacy	via Bifrost
billable tokens	9,335	4,363
cost	$0.0233	$0.0109
cache hits	0	32
failed requests (provider kill)	26	0
savings	—	53%

The savings came entirely from replayed cache hits — no provider call, no tokens. On a support workload that repeats questions daily, that ratio compounds. The availability delta speaks for itself: 0 failures through a provider kill, against 26 in the legacy run.

The bottom line

Migrating to an enterprise AI gateway is not a rewrite. It is a sequence of small, reversible moves — deploy beside, point one client, add fallback, enable cache, issue keys, wire observability, cut over. Measured on the rebuilt stack: 53% cost reduction on cacheable traffic, zero failed requests through a provider kill, and governance the legacy stack never had. The migration risk is low; the legacy risk is already on your pager.

636 Bytes: What Happens When You Stop Teaching RPA the Path

Don Johnson — Fri, 28 Aug 2026 05:03:21 +0000

I sat in a room with a very large automation system and a very familiar problem: it had become fragile.

The language came quickly. DOM. Timestamps. Timing windows. Selectors. Retries. Circle back.

Every word was reasonable.

Together, they described a hill.

The system was not fragile because its builders lacked discipline. It was fragile because the path had become the product. A bot was taught not what outcome had to be true, but which element to find, where to click, how long to wait, and what to retry when the page disagreed. Each workaround made sense locally. Together, they turned change into maintenance.

The first proof

The first proof on my machine was 636 bytes.

Not the model. Not the browser. Not the platform around it. It was a WebAssembly module: a tiny, portable program injected at the boundary between an intelligent proposal and an authorized browser action. Its smallness mattered because it kept that boundary deterministic, inspectable, and difficult to hide complexity inside.

It does not do everything. That is the point.

The 636-byte module is the injected proof; the hardened Rust kernel in this build compiles to 26,893 bytes. Neither contains the model, scheduler, credential store, policy engine, or evidence system. Those responsibilities live outside the browser boundary, where they can be isolated, governed, and replaced independently. Small is not the absence of an architecture. Small is a decision about where complexity is allowed to live.

Intelligence is not authority

An AI model can interpret an intent, inspect the current page, and propose what should happen next. It cannot grant itself permission. It does not get to cross a tenant, workspace, or origin boundary because doing so would be convenient. It does not receive a credential or press a consequential button merely because its reasoning sounds confident. Those decisions belong to local policy and the execution layer. Intelligence stays flexible; authority stays bounded.

That separation becomes a lifecycle:

intent → lease → observe → decide → act → verify → artifact

The intent defines the outcome. A lease gives one worker a short-lived claim on that work. The worker observes the page, the reasoning layer proposes a decision, and policy determines whether an action may execute. Verification checks the resulting state rather than trusting the click. Finally, an artifact preserves causally linked evidence of what happened. Every arrow is a boundary where the system can refuse, recover, or explain itself.

The hill has a bill

The hill also has a bill. To make that friction visible, I ran an illustrative planning scenario through agent-calc—not customer telemetry. Assume 100 workflows, two changes per workflow each month, four maintenance hours per change, and fully loaded labor at $140 per hour. The modeled comparison is stark:

Scenario	Modeled monthly cost
Legacy path maintenance	$112,000
Intent architecture	$20,098
Avoided friction	$91,902

That is an 82.1% modeled reduction. The percentage is not a promise; the assumptions are visible precisely so they can be challenged. The useful question is what the model exposes: how much are we spending to preserve instructions that describe yesterday's interface instead of today's desired outcome?

Test the outcome

Of course, an architecture that survives only a curated demo is just a cheaper failure. So we built the Component Gym: a synthetic web application designed to resist memorization. Controls move between runs. Buttons use different event listeners. The target may be buried among decoys inside randomized tabs and paginated lists. A seed makes each hostile arrangement reproducible without making it predictable to the agent.

The agent receives an intent, not a selector script. The harness then grades the resulting application state out of band. It does not care which path looked convincing or whether a click event fired. It cares whether the requested outcome became true.

In the filmed run, seed 710003 placed the needle inside a dynamic collection spread across tabs and pages. The agent was given the desired outcome and the browser's current state—not the target's coordinates or a prerecorded route. It had to navigate, distinguish the target from decoys, act, and leave the requested state behind. Only then did the independent grader return PASSED. The agent did not get to grade its own homework.

Determinism moved

None of this makes the DOM, timing, or selectors disappear. The browser still has structure. Events still happen in time. A selector may still be the right tactic for a particular action. What changes is their lifetime and ownership. In a path-driven system, those details harden into durable business logic. Here, they are runtime observations and disposable tactics, abandoned when the environment changes. Determinism has not been removed; it has been relocated into authorization, state transitions, verification, and evidence.

A primitive is not a platform

A passing gym run proves the primitive, not the platform. The small browser boundary reduces one category of fragility; it does not erase the distributed-systems, security, and operational work around it. The remaining obligations are less cinematic and more important:

Isolation: preserve tenant, workspace, worker-claim, and origin boundaries through every action.
Work ownership: make leases expire safely, recover interrupted work, and prevent retries from duplicating consequential actions.
Secrets and MFA: resolve credentials only at the authorized execution boundary, never inside model context, jobs, logs, screenshots, videos, or artifacts. MFA is on-behalf execution, never authentication bypass.
Browser lifecycle: attach to, control, recover, and release browser sessions without leaking state between workers or tenants.
Policy: deny actions that exceed the granted intent, even when the proposed action is technically possible.
Evidence: causally connect intent, observation, decision, authorization, action, verification, and artifact so a result can be explained after the browser is gone.

Compile the boundary smaller

The 636 bytes are not interesting because smaller software is automatically better. They are interesting because they make a refusal physical: do not bury orchestration, secrets, and authority inside the thing touching the page. Compile the trusted browser boundary smaller. Let reasoning adapt to the environment around it. Demand proof after every meaningful consequence.

I keep returning to that room. No one in it said anything absurd. The hill was built from rational decisions made under one inherited premise: the path must be preserved. Change that premise and the hill changes shape. We do not need to keep teaching automation every route through an interface. We can preserve the intent, rediscover the route, constrain the action, and verify the destination.

The next time an automation breaks and asks for another selector, timeout, or recovery layer, ask one question first: how much of this system protects the outcome—and how much protects yesterday's route?

[Boost]

Don Johnson — Wed, 26 Aug 2026 05:32:01 +0000

Transactional outbox challenges for tasks

Don Johnson

Aug 26

My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.

#ai #agents #architecture #distributedsystems

10 min read

My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.

Don Johnson — Wed, 26 Aug 2026 05:30:59 +0000

The 2:56 video above is a fictional medication-safety exercise. The gateway interoperability is tested; the six-organization incident is a deterministic simulation.

Read the corrected narration transcript.

Last time, I discovered that the QUIC transport in my agent framework had never actually transported anything.[1]

This time the transport was real. Five processes could find each other, exchange encrypted messages, reinforce independent conclusions, and let unsupported signals decay.

The mesh worked.

It still could not introduce itself to another agent.

There was no standard way to ask what the swarm could do. No retained task to retrieve after an internal signal expired. No interoperable progress stream. No cancellation contract. No artifact another framework would understand.

I had built a society with no border crossing.

Uncovers latent bugs and flawed core semantics

Don Johnson

Aug 19

My QUIC transport had never once been executed. Here's what happened when I ran it.

#rust #distributedsystems #networking #ai

8 min read

Google's Agent2Agent protocol gave that missing boundary a name. A2A was announced in April 2025 as an open protocol for agents built by different vendors and frameworks to discover one another, exchange messages, and collaborate without sharing their private memory, tools, or internal plans.[2] The project moved under Linux Foundation governance in June 2025, so "Google's A2A" is historically accurate, but no longer the whole story.[5]

What I needed was not a new brain for SMESH.

I needed a public contract in front of it.

A working mesh is not an interoperable agent

SMESH is the framework I designed for decentralized coordination between LLM agents. It borrows its mechanics from mycorrhizal networks rather than job queues.[6]

agents emit signals into a field;
signals lose intensity over time;
agents claim work according to local affinity;
independent agreement reinforces a claim;
unsupported claims disappear without a central process rejecting them;
trust changes what a node is willing to relay.

That solves an internal coordination problem. It answers questions such as:

Which specialist should take this work?
Has another independent agent seen the same thing?
Is this claim gaining support or merely being repeated?
Can a stale task disappear without a scheduler cleaning it up?

A2A solves a different problem.

It asks:

How does an outside agent discover this system?
How does it delegate a unit of work?
How does it watch a long-running task?
How does it cancel that task?
What retrievable result comes back?

The official A2A specification describes an interoperability layer for independent, potentially opaque agent systems. Its current v1 model separates canonical data objects, abstract operations, and concrete bindings such as JSON-RPC, gRPC, and HTTP+JSON/REST.[3][4]

The core vocabulary is small:

A2A concept	job
`AgentCard`	advertise identity, skills, interfaces, media modes, and security declarations
`Message`	carry one interaction turn as typed parts
`Task`	track stateful work through a lifecycle
`Artifact`	return result data associated with a task
`contextId`	group related messages and tasks

A2A v1 defines operations such as SendMessage, SendStreamingMessage, GetTask, ListTasks, CancelTask, and SubscribeToTask. A binding decides how those operations appear on the wire; the canonical task semantics stay the same.[4]

That distinction matters. If I tried to make A2A the swarm's internal coordination algorithm, I would flatten SMESH into a remote procedure call graph. If I tried to expose raw SMESH signals as the public protocol, every client would need to understand decay, relay probability, local trust, topology, and attestation sets.

Neither would be interoperability. It would be leakage.

The three layers are not competitors

The cleanest architecture I have found is this:

layer	relationship	owns
A2A	agent ↔ agent	discovery, tasks, streaming, cancellation, artifacts
SMESH	agent ↔ local swarm	claiming, diffusion, reinforcement, decay, trust
MCP	agent ↔ tool or data source	tool invocation and resource access

The official A2A documentation makes the same separation from MCP: MCP equips an agent with tools and resources; A2A lets independent agents collaborate as agents.[3]

For SMESH, that produced a hard architectural rule:

A2A is the external task contract. SMESH is the ephemeral internal coordination field.

The design requires the A2A ledger to outlive internal signal decay. In this MVP, that retention lasts only for the life of the process; restart durability still requires SQLite or Postgres.

The boundary is split between a path that runs now and a path that is only an integration seam:

External A2A client
        |
        | Agent Card / Send / Stream / Get / List / Cancel
        v
+-------------------- smesh-a2a ---------------------+
| A2A SDK routers + guarded request handler          |
| bounded process-local task ledger + executor       |
| validation + MeshDispatcher                        |
+----------------------------------------------------+
        |
        +-- current binary: LoopbackDispatcher
        |      `-> deterministic MeshEvent stream
        |
        `-- tested seam: ChannelDispatcher
               `-> real SignalType::Query
                   `-> SMESH runtime (not wired into the binary yet)
                       `-> MeshEvent stream
        |
        v
Process-lifetime A2A task history in the MVP

The desired live path is present as a typed, tested boundary. The checked-in standalone executable still takes the loopback path.

I kept the adapter in a separate repository. smesh-rust remains the coordination substrate. smesh-a2a can follow A2A's release cycle, server dependencies, and security boundary without pushing HTTP and SDK churn into the core mesh.[6][7]

The Agent Card is not the swarm

For a known endpoint, A2A capability discovery begins with an Agent Card: a public JSON description of an agent's interfaces, capabilities, skills, and security requirements.[4]

My first temptation was to list every internal role: security reviewer, tester, architect, performance analyst, contradiction sentinel.

That would have been wrong.

Those roles are implementation details. They may change per task. Some may not exist until the mesh senses the work. Publishing them would couple clients to an internal topology that SMESH is specifically designed to keep fluid.

The public card advertises one aggregate capability instead:

// Abridged from build_agent_card(). These are the public promises.
let supported_interfaces = vec![
    AgentInterface::new(
        format!("{base}/jsonrpc"),
        TRANSPORT_PROTOCOL_JSONRPC,
    ),
    AgentInterface::new(
        format!("{base}/rest"),
        TRANSPORT_PROTOCOL_HTTP_JSON,
    ),
];

let capabilities = AgentCapabilities {
    streaming: Some(true),
    push_notifications: Some(false),
    extensions: None,
    extended_agent_card: None,
};

let public_skill = AgentSkill {
    id: "smesh.collaborative-task".to_owned(),
    name: "Collaborative swarm task".to_owned(),
    description:
        "Coordinates specialist agents through SMESH and returns an accepted artifact."
            .to_owned(),
    tags: vec![
        "multi-agent".to_owned(),
        "coordination".to_owned(),
        "review".to_owned(),
        "testing".to_owned(),
    ],
    examples: Some(vec![
        "Review this Rust repository for correctness, security, and performance."
            .to_owned(),
    ]),
    input_modes: Some(vec!["text/plain".to_owned()]),
    output_modes: Some(vec![
        "text/plain".to_owned(),
        "application/json".to_owned(),
    ]),
    security_requirements: None,
};

The card says what an external client may rely on. It does not reveal how the swarm will organize itself, and it is not proof that the publisher should be trusted.

Discovery metadata is not authorization. A skill description is not a capability grant. A client saying tenant=important-customer does not make that identity real.

Those sound like obvious distinctions. They are also exactly the distinctions that disappear when a demo and a security model share the same JSON object.

One A2A message becomes a typed SMESH query

At ingress, the gateway accepts bounded inline text, creates a stable A2A task envelope, and translates it into the actual core signal type used by SMESH:

pub fn to_signal(&self, gateway_node_id: &str) -> Signal {
    Signal::builder(SignalType::Query)
        .payload_json(self)
        .origin(gateway_node_id)
        .build()
}

The payload carries the A2A task ID, context ID, protocol marker, and validated text. ChannelDispatcher packages that typed signal with the request and hands both to a runtime-owned worker.

That boundary is implemented and tested. The standalone binary still uses LoopbackDispatcher, so it does not yet inject the Query into a live multi-process SMESH runtime. The real runtime adapter is the next integration step.

Setting origin on the gateway Query is deliberate. In my previous article, independently corroborated claims omitted their origin from the content hash so identical conclusions could converge on one address. This is a different signal. A gateway Query is an ingress envelope, not a claim waiting for independent corroboration. Its source belongs in the record.

The boundary also rejects what it does not understand. The MVP accepts inline text only. It does not fetch a client-provided URL, dereference an arbitrary file, or treat external metadata as instructions for the mesh.

Inline text is boring, but I know exactly what crosses the boundary.

The task ledger and the signal field tell different kinds of truth

This was the most important design correction.

A SMESH signal is supposed to decay. If nobody reinforces a task, its intensity falls until it no longer matters. That is useful inside the coordination system because stale work cleans itself up.

An A2A task must not disappear because its internal coordination signal faded.

An external client may reconnect five minutes later and call GetTask. An auditor may list tasks by context. A user may need to see that a cancellation was accepted. An artifact must still belong to the task that produced it.

So the gateway cannot reconstruct its public state by looking at the mesh.

A2A task ledger = retained external task state (process-local today)
SMESH signal     = temporary coordination pressure

The ledger is authoritative for the A2A lifecycle. The mesh is authoritative only for its local coordination observations.

That also means terminal states are absorbing. Once a task is completed, failed, canceled, or rejected, a repeated message cannot quietly restart work under the same ID.

Streaming exposed a bug that all my tests had missed

A2A v1 supports both direct responses and stateful tasks. A task lifecycle stream begins with the Task itself, then emits ordered status or artifact updates, and closes when the task reaches a terminal state.[4]

My executor maps internal mesh events like this:

mesh dispatch accepted  -> Working
mesh progress           -> Working + status message
mesh artifact           -> Artifact update
mesh completion         -> Completed
mesh failure            -> Failed
accepted cancellation   -> Canceled

The stream ordering test passed.

Then an independent review pointed out that my own worker budget allowed 256 events while the upstream server's broadcast subscription buffer held 32. A fast worker could stay inside my documented limit and still outrun the initiating subscriber. The task might finish in storage while the client received an internal "subscription fell behind" error.

That is a particularly unpleasant distributed-systems bug because both sides can truthfully report different outcomes.

The fix was not to hope the subscriber ran faster. I reduced the worker event budget to 16, clamp caller-provided limits to that ceiling, and added a burst test through the official client.

This is what protocols do to a design: they force every implicit assumption to become somebody else's observable failure.

Cancellation has to stop work, not just change a status label

Forwarding CancelTask to a dispatcher was not enough.

If an internal worker ignored the request or kept its event stream open, the original client subscription could hang. Worse, late Working or Completed events could arrive after the public task had become Canceled.

The executor now owns a per-task cancellation token. The first accepted cancellation:

reaches the dispatcher;
wakes the active producer loop;
closes the original execution stream;
prevents post-cancel work from changing the terminal state.

Channel sends and cancellation acknowledgements have deadlines. Worker inactivity has a deadline. The total task has a deadline.

Cancellation is not a field update. It is a distributed state transition with work on both sides of the boundary.

The boring limits are the real feature

The first version was interoperable. It was not bounded enough to deserve trust.

A fail-closed review found unbounded task retention, unbounded worker output, cancellation leaks, missing dispatcher deadlines, terminal task reuse, and invalid input that could leave a task stranded in Submitted.

The current localhost-first gateway now bounds:

resource	default boundary
HTTP request body	128 KiB
accepted inline text	64 KiB
retained process-local tasks	1,024
active executions	64
worker events	16
artifacts per task	16
aggregate output per task	1 MiB
worker inactivity	30 seconds
total task execution	5 minutes
channel/cancel acknowledgement	5 seconds

It also refuses non-loopback binds unless an explicit unsafe override is present.

That override does not add authentication, TLS, tenant isolation, or authorization. It only disables the refusal. The current binary is for localhost and trusted integration work, not direct exposure to the internet.[7]

I am spelling that out because "supports enterprise authentication" in a protocol specification does not mean every prototype using the protocol is enterprise-secure.

The LIFELINE demo is a simulation, on purpose

The cover video follows a fictional medication-safety incident. Three hospitals see weak pieces of the same adverse-event pattern. Separate manufacturer, regulator, logistics, payer, and evidence agents contribute artifacts. Inside each public endpoint, a SMESH swarm claims work, reinforces evidence, contests an early hypothesis, and lets unsupported signals decay.

One logistics endpoint fails. Its task is canceled. A fallback is discovered. The incident continues. The agents converge on a recommendation, but a human incident commander owns the irreversible decision.

The visual separates the layers deliberately:

ivory arcs are A2A traffic between organizations;
green fields are SMESH activity inside an organization;
cyan shards are artifacts;
vermilion marks contradiction or failure;
one gold ring marks human authority.

Every visible event comes from a 55-event, ordered, hash-chained JSONL fixture generated as one complete file. The browser can play it, scrub it, inspect it, or export deterministic frames. The narrated film and interactive replay are available from the gateway repository.[7][8]

Honesty boundary: the gateway's A2A interoperability is exercised against the official Rust client. The LIFELINE six-organization trace is synthetic. It proves the replay contract and the intended architecture; it does not prove that six live SMESH runtimes executed the scenario.

A captured run across live SMESH runtimes would be operational proof. This demo is not.

What is implemented, and what is still a plan

implemented now	still required for an internet-facing system
A2A v1 Agent Card	authenticated principals
JSON-RPC and HTTP+JSON/REST bindings	tenant-aware authorization
official-client tests for discovery, JSON-RPC/REST send, streaming, and cancellation	persistent SQL task ledger
SSE task streaming	distributed quotas
Get, List, and Subscribe routes through the SDK handler	TLS termination and deployment policy
real `SignalType::Query` construction	live SMESH runtime adapter behind every organization
bounded process-local execution	push callback validation and SSRF controls
deterministic synthetic replay	captured multi-runtime causal trace

There are two easy ways to lie with a demo like this.

The first is to animate what you wish the system did.

The second is to run one loopback worker and describe it as a decentralized enterprise.

I would rather keep the boundary visible.

What A2A changed in the way I think about SMESH

Before this work, I thought of SMESH as the system.

Now I think of it as an interior.

The mesh can remain weird in useful ways. Signals can diffuse probabilistically. Specialists can appear and disappear. Trust can be local. Claims can decay. None of that has to leak into the contract presented to another agent.

MCP gives individual agents hands. SMESH gives a group local coordination. A2A gives that group a public identity, a task contract, and a cancel button.

The previous transport article ended with a lesson: code that has never been executed is a plan. This work left me with the same rule one layer higher:

A system nobody else can discover, hire, observe, or cancel is not interoperable. It is an island with excellent internal networking.

If you are building multi-agent systems, where would you draw the boundary between cross-organization interoperability and internal swarm coordination—and which decisions would you refuse to delegate?

copyleftdev / smesh-a2a

A2A v1 interoperability gateway for decentralized SMESH agent swarms

SMESH A2A

A2A v1 interoperability gateway for decentralized SMESH agent swarms.

SMESH remains the internal coordination substrate: signals diffuse, decay, reinforce, and accumulate attestations. A2A is the public contract for discovery, durable task lifecycle, streaming progress, cancellation, and artifacts.

What works

Official A2A v1 Rust types and server/client SDKs
Public Agent Card at /.well-known/agent-card.json
JSON-RPC endpoint at /jsonrpc
HTTP+JSON/REST endpoint at /rest
Synchronous and SSE streaming task execution
GetTask, ListTasks, SubscribeToTask, and CancelTask through a2a-rs
Strict inline-text validation with a 64 KiB default limit
Translation to a real smesh_core::SignalType::Query
Injectable MeshDispatcher boundary for a production SMESH runtime
Deterministic loopback worker for demos and interoperability tests
Bounded task retention, execution concurrency, event/artifact counts, and output bytes
Worker inactivity, cancellation, and command-channel deadlines
Terminal-state and task-ID reuse guards

Architecture

A2A client
   |
   v
Agent Card + JSON-RPC/REST/SSE
   |
   v
SmeshExecutor -- validates and translates
   |
   v
MeshDispatcher
   |-- LoopbackDispatcher

…

View on GitHub

Interactive replay: copyleftdev.github.io/smesh-a2a

SMESH core: github.com/copyleftdev/smesh-rust

Sources

[1] https://dev.to/copyleftdev/my-quic-transport-had-never-once-been-executed-heres-what-happened-when-i-ran-it-24ge — My QUIC transport had never once been executed
[2] https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability — Announcing the Agent2Agent Protocol
[3] https://a2a-protocol.org/latest — What is A2A Protocol?
[4] https://a2a-protocol.org/latest/specification — A2A Protocol v1.0 specification
[5] https://linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents — Linux Foundation launches A2A project
[6] https://github.com/copyleftdev/smesh-rust — SMESH repository
[7] https://github.com/copyleftdev/smesh-a2a — SMESH A2A gateway repository
[8] https://youtu.be/EFPKaIuF8iA — SMESH A2A cinematic demo

A Wider Computer, Not a Bigger One: Modeling AI Inference Across Millions of Homes

Don Johnson — Tue, 25 Aug 2026 05:59:43 +0000

I modeled an AI inference fleet distributed across ordinary homes. What survived was narrower—and more plausible—than what I started with.

Picture a detached house on a cold evening. On one garage wall, an operator-owned compute appliance draws about five kilowatts—electrically comparable to an EV charger, though it runs much longer. It serves small AI models. In winter its waste heat can warm the house; in summer that heat must be carried away. The family bought nothing. They are paid to host it.

Now repeat that arrangement across a neighborhood, then a state, then millions of homes—not to assemble one enormous computer, but to create millions of independent inference workers. Each keeps a small model resident in GPU memory. New requests route around homes that are offline. The network grows by adding locations that were already built and connected to the grid.

That network does not exist. I call the idea HEARTH. Almost none of its physical pieces are exotic; the experiment is whether they can be arranged and scheduled as one system. Not a bigger computer. A wider one.

To make the appliance less abstract, I developed three concept form factors: a wall unit, a floor-standing thermal tower, and a duct-integrated mechanical-room unit. They are appearance and installation studies, not engineered products; their job is to expose the questions that a real prototype must answer.

The model kept changing its answer

Then I priced the accelerators, and the idea stopped working.

That was the first useful result. My original comparison had focused on the dramatic expense of constructing a data center while treating the compute hardware as identical on both sides. But identical hardware does not disappear from the economics. If one location keeps an expensive GPU busier than another, utilization can overwhelm everything the cheaper building saves.

I rebuilt the model five times. Each pass introduced a constraint the previous one had missed:

Pass	What changed	Model verdict
1	Compared the facilities around the hardware	Homes win
2	Included accelerator cost and utilization	Homes lose
3	Used consumer and data-center hardware at quoted prices	Homes win
4	Modeled production batching for a 32B model	Homes lose
5	Tested different model sizes	Homes win only for the small-model case

What survived: small, independent inference

The fifth pass did not rescue my original proposal. It replaced it. A residential fleet is not a cheaper place to run every AI workload. It cannot pool memory across the internet, train a frontier model, or make residential latency disappear. What survived was narrower: small-model inference in which one node completes one request and no durable state is tied to a particular house.

Model size changes the economics because inference is not just a race between GPUs. During generation, a server repeatedly reads the model's weights while advancing many requests together. That is batching. A larger batch spreads each weight read across more paying tokens—but every active request also consumes working memory, usually called the KV cache.

An 8B model—roughly eight billion parameters—leaves enough memory on a 32 GB consumer GPU for a useful production batch. In my model, both the home GPU and the data-center hardware then become limited mainly by compute, allowing the consumer part's much lower purchase price to matter. At 32B, the consumer card runs short of memory first. Its batch stops growing while high-memory data-center hardware keeps filling, and the verdict reverses.

Model class	What the current model says
8B	Candidate workload; the consumer GPU retains useful batching headroom
32B	Data-center hardware wins unless other advantages offset its batching edge
70B	Excluded by my single-GPU serving assumption; multi-GPU serving was not modeled

The current result is conditional. Before homeowner compensation and fleet-level operating costs, and under RTX 5090 MSRP, an 85% accelerator duty factor, identical serving-efficiency assumptions for both venues, and a quantized 8B workload, my roofline model estimates the residential hardware-and-energy stack at about 0.49 times the cost per token of the best data-center case in the sweep, based on GB200 NVL72 rack pricing. That is a reproducible scenario output, not observed production performance or a fully loaded business cost. An apples-to-apples benchmark—or the costs excluded here—could erase the advantage.

This is the distinction behind wide, not deep. HEARTH would not divide one giant model among houses or combine residential GPUs into a virtual supercomputer. Each node would answer complete, independent requests with a model it already holds locally. Adding homes increases the number and variety of requests the fleet can serve; it does not make any one node larger.

The asset is the grid endpoint

Why put that workload in homes at all? Not because residential electricity is cheap—it usually is not—or because waste heat makes energy free. The asset is the connection behind the meter.

Berkeley Lab estimated that US data centers used about 176 TWh of electricity in 2023. Its June 2026 update projects 521–843 TWh in 2030, with a 649 TWh reference case. Serving that growth is not simply a question of buying more GPUs. A large new campus may need substations, transformers, transmission upgrades, and a negotiated path to connect a load the local grid was never designed to carry. The 410 GW of prospective large loads reported in ERCOT—about 87% associated with data centers—is not a forecast of what will be built. It is evidence of how much demand is converging on the same constrained process.

The United States has about 82.5 million one-unit detached housing units. That is physical stock, not an estimate of eligible hosts: occupancy, broadband, electrical capacity, utility approval, and household consent would reduce it substantially. Each candidate already has a meter, an electrical service, and a physical building around it. A compute node installed behind that meter may avoid the transmission-scale interconnection required by a new campus. That is the inversion: instead of bringing an enormous new grid connection to the compute, bring a modest amount of compute to many connections that already exist.

Existing does not mean unused. A continuous 5 kW load is much less forgiving than an EV that charges for a few hours. Some homes would need panel or service upgrades; utilities may require review; and the local distribution transformer remains a hard physical constraint. My one-node-per-transformer rule is therefore a screening hypothesis, not a validated safety rule or national capacity estimate. Transformer loading, telemetry, and thermal behavior belong in the first utility-supervised field test.

To the household, this is a hosting contract, not an investment. The operator would own and maintain the appliance, meter and reimburse its electricity, and pay the family a share of revenue. None of that has been field-tested; fire and insurance rules, noise, summer heat rejection, ISP terms, maintenance, and upgrades remain field questions.

One request goes to one home

At the software layer, the system is deliberately less exotic. A control plane would assign replicas of small models to qualifying nodes and distribute checkpoints outside the request path. Each node would store its assigned checkpoints locally and keep its serving model resident in GPU memory. The request router would know which homes have the requested model resident, which are healthy, and which have batch capacity available.

registry ── sync ──► home node

client ──► router ──► available home ──► response
               ▲              │
               └── telemetry ─┘

During generation, one home holds the request's KV cache and returns the tokens; no activations or model layers cross the residential network. If the node disappears, the in-flight generation is lost. New work can route elsewhere, and a retryable request can start again against another replica, although it may not produce identical tokens. The system does not inherit data-center reliability merely because it has many machines.

At the network level, remote attestation, model security, and prompt and output confidentiality on hardware outside an operator-controlled facility remain unresolved. Distribution makes failures routable; it does not make operations disappear.

Five conditions decide whether this is real

The hardware must be legally and commercially usable. NVIDIA's current driver agreement says the software may not be used to provide commercial hosting services and that GeForce and Titan software is not licensed for data-center deployment. Whether an operator could obtain written authorization or a different license for this residential topology must be resolved with NVIDIA and qualified counsel before a hardware pilot. Warranty, support, and a credible procurement channel must also exist.
Edge serving must sustain the modeled batches. The same checkpoint, quantization, context length, serving stack, and request mix must be benchmarked on the candidate consumer GPU and data-center hardware. A spreadsheet cannot substitute for that result.
Demand must keep the silicon busy. Cheap idle GPUs are still expensive. Customers must commit enough suitable 8B inference work—classification, extraction, routing, drafting, or other bounded tasks—to maintain high utilization at a viable token price.
The location must pass the energy screen. Residential rates vary widely, and heat reuse is a seasonal credit rather than the thesis. Every deployment needs to work after metered electricity, host payment, and summer heat rejection.
The distribution grid must approve the load. A real utility must confirm that a real home, service, and transformer can carry it continuously. National averages are not permission to connect.

Those are not caveats around the result. They are the result. HEARTH exists only at their intersection.

The first pilot should contain no houses

That changes the order of the pilot. My original plan began with 250 homes and grew through increasingly expensive gates. It tested installation first and market demand later. The model eventually exposed the flaw: there is no reason to put hardware in a house before proving that anyone will buy this particular kind of inference.

The first HEARTH pilot should therefore contain no houses. Rent the candidate consumer and data-center hardware. Resolve the license question. Run the same models through the same serving stack, measure delivered tokens and power at production batch, and ask customers to pay for the workloads the residential fleet is supposed to serve. If the measured economics or demand fail, stop before an electrician is dispatched.

Only then move to 250 homes. That field trial buys answers the lab cannot: Can crews install safely and consistently? What do transformers do under continuous load? How loud and hot are the nodes in July? Can the operator keep them online without drowning in truck rolls—and will families keep hosting them?

Four million homes is not the next milestone

Four million homes is an image of the possible system, not an adoption forecast. The serious next milestone is smaller: one workload that benchmarks, one customer willing to buy it, one utility willing to supervise it, and then the first 250 homes.

I published the full feasibility study, the source model and inputs, and every assumption I used. If the idea is wrong, I want the failure to be reproducible too.

HEARTH is not a forecast. It is a claim specific enough to test—and to kill if it fails. The buildings, meters, and many of the grid and network endpoints already surround us. What is missing is the appliance, the dedicated installation, the operating system around it, and an agreement worth signing.

If those pieces work, the neighborhood still looks like a neighborhood on a cold evening. Behind some garage walls, independent machines serve requests from a network spanning the country—each complete on its own, all scheduled together.

No monument to compute. A million warm windows.

My QUIC transport had never once been executed. Here's what happened when I ran it.

Don Johnson — Wed, 19 Aug 2026 03:35:06 +0000

I've written before about SMESH, a coordination protocol modelled on mycorrhizal networks — the fungal web that lets trees in a forest warn each other about drought and disease with nothing in charge of the network. Signals diffuse, decay on their own, and get reinforced when independently confirmed. Consensus emerges instead of being orchestrated.

That was the idea. This post is about the part where I found out whether it worked.

The transport that had never run

SMESH has had a QUIC transport in it for a while. Roughly 500 lines: a quinn endpoint that is simultaneously server and client, self-signed certs, length-prefixed bincode frames over unidirectional streams, an accept loop that spawns per-connection and per-stream tasks, connection pooling.

Every test passed. The workspace was green. I could point at smesh-runtime/src/transport.rs and say "yes, it does peer-to-peer."

Then I grepped for who actually constructed it:

$ grep -rn "QuicTransport" --include='*.rs' .
smesh-runtime/src/transport.rs:177:pub struct QuicTransport {
smesh-runtime/src/transport.rs:192:impl QuicTransport {
smesh-runtime/src/lib.rs:16:pub use transport::{QuicTransport, ...};

Its own definition, and a re-export. Nothing else in the workspace had ever instantiated it. No binary opened a socket. SmeshRuntime imported TransportConfig, stored it in a struct field, and never looked at it again.

I had a networking layer with tests, docs, and zero executions.

Three bugs in the first twenty minutes

I wrote an integration test that starts two runtimes, has one dial the other, and asserts a signal crosses. Here is what fell out before it went green.

1. It panicked on the first call.

Could not automatically determine the process-level CryptoProvider
from Rustls crate features.

rustls 0.23 refuses to pick a crypto backend when more than one is compiled in, and quinn pulls in both through its own feature set. Every call to QuicTransport::new would have panicked for anyone, ever. Nobody noticed because nobody had called it.

2. Dialled connections were write-only.

connect() stored the connection in the pool but only the accept loop pumped incoming streams — and the accept loop only sees connections you accepted. So a node that dialled out could send, and would never receive anything back. A QUIC connection is bidirectional regardless of who dialled it; my code only acted like it half the time.

3. Unbounded allocation from an attacker-controlled length prefix.

let len = u32::from_be_bytes(len_buf) as usize;
let mut data = vec![0u8; len];   // <- no

max_message_size was in the config struct. It was never read. Send a 4 GiB length prefix and the process allocates 4 GiB.

None of these are clever bugs. They're the bugs you get for free the first time code meets a socket, and the only reason they survived is that the code had never met a socket.

The harder problem: my protocol was wrong

Fixing the plumbing was the easy half. The real issue was that my diffusion algorithm quietly assumed something no distributed system can assume.

Network::tick expands a signal's reach one hop per tick by walking the graph:

for node_id in &reached {
    for hypha in self.hyphae.get(node_id) {
        let target = self.nodes.get(&hypha.to);
        if target.should_relay(&signal, remaining_hops) {
            frontier.push(hypha.to.clone());
        }
    }
}

Read that again. It iterates every node's adjacency, calls every node's relay policy, and mutates one global reached_nodes set on a shared signal. It's a breadth-first search from a god's-eye view of the entire graph.

That works beautifully in one process. In a real mesh, no node can see that graph. Porting it meant three corrections, and each one turned out to be a genuine bug rather than a porting detail.

Correction 1: independent conclusions were being thrown away

Signals are content-addressed — the hash is derived from what is being claimed. So when two agents independently reach the same conclusion, they produce the same hash.

My emit() saw the hash already present locally and treated it as a duplicate. It dropped it.

That is exactly backwards. Two parties independently agreeing is not redundant data — it is the only evidence the system has that a claim is real. Discarding it destroys the thing the protocol exists to measure.

/// Signals are content-addressed, so a node that independently reaches a
/// conclusion another node already published lands on the same hash. That
/// is treated as *corroboration*: this node is added as an attester and the
/// merged claim still goes out, because our agreement is news to everyone
/// who has not heard it. Swallowing it as a duplicate would silently
/// discard the only evidence that two parties concur.

Correction 2: I was counting messengers, not witnesses

Reinforcement attributed the claim to whoever handed me the message. In a gossip mesh, one finding relayed by five nodes then looks like five corroborators.

The party that attests to a claim is its origin, not the peer that passed it along. Relaying is not agreeing.

Correction 3: gossip needs a merge rule, not a broadcast rule

Once attestation is a set, the rule that makes gossip converge is simple and pleasant:

// Merge the two attester sets. Anything the sender knew that we did
// not is new information, and new information is worth passing on.
let before = Node::attesters(existing);
for attester in &incoming_attesters {
    if !before.contains(attester) {
        existing.reinforce(attester);
    }
}

Forward if and only if your own knowledge grew. That single rule is the loop breaker (a message teaching you nothing goes no further), the convergence mechanism (the set is grow-only, so it's a CRDT), and the anti-entropy repair (re-asserting carries your accumulated view, so a node that missed a round catches up).

The part I nearly got wrong on purpose

Here is the subtlety that makes the whole thing work, and it looks like a mistake in the source.

The signal builder folds the origin node into the content hash if you give it one. So when an agent publishes a claim, you must not set the origin:

let signal = Signal::builder(SignalType::Alert)
    .payload(assertion.canonical_bytes())
    .confidence(finding.confidence)
    .build();
// .origin() deliberately NOT called

If you set it, every agent gets a different hash for the same claim and correlation becomes impossible. The address has to be the claim, not the claimant.

The corollary is that evidence cannot travel in the payload. Each agent's evidence differs, so putting it in would make every hash unique and break the mechanism. The payload is just the assertion:

{"subject":"checkout-api","claim":"degraded"}

Evidence stays local and goes to the log. The mesh carries assertions; it does not carry arguments.

Five witnesses, none of whom can see the problem

To actually test any of this, I built a scenario where the answer cannot be reached alone.

Five analyst processes watch the same fleet of services. Each can see exactly one kind of telemetry, and nothing else:

agent	sees
latency	p99 response times
errors	error rates
saturation	pool and CPU utilisation
traces	retry rates, span queueing
deploys	release events

A deploy cuts checkout-api's connection pool from 200 to 20. The pool pins, requests queue, callers time out.

Now every agent sees a piece of it, and two of them are actively misleading. errors sees payments-api throwing 503s and would blame it — but payments is a victim, its own pool and CPU are fine. latency sees four services slow at once and can't say which is causal. Only deploys can see there was a release, and a release on its own means nothing; software ships all day.

I also planted two decoys with a single witness each — an unrelated CPU spike, a brief latency blip — because a system that agrees with everything is not consensus, it's an echo.

They run as five real OS processes on five ports over real encrypted QUIC, in a ring-plus-chord topology so messages actually have to be relayed to cross the mesh. Peer discovery is off, or gossip quietly converts any topology into a full mesh and there's nothing left to watch.

$ smesh orchestrate --out runs/latest

  spawned latency     pid 2452744  127.0.0.1:9301  dials 9302,9303
  spawned errors      pid 2452745  127.0.0.1:9302  dials 9303
  spawned saturation  pid 2452746  127.0.0.1:9303  dials 9304
  spawned traces      pid 2452747  127.0.0.1:9304  dials 9305
  spawned deploys     pid 2452748  127.0.0.1:9305  dials 9301

The result:

what the mesh concluded
  checkout-api          CONSENSUS at 16.9s · 5 attesters · seen by 5 nodes
  payments-api          no consensus (3/4 concerns)
  edge-gateway          no consensus (1/4 concerns)
  notification-worker   no consensus (1/4 concerns)
  session-store         no consensus (1/4 concerns)

Cause separated from symptom separated from noise. The loud, obvious suspect was held at three witnesses and never promoted. The decoys were never rejected by anything — they simply went uncorroborated and decayed out. Nobody voted, and nothing was in charge.

Making the run replayable, and then checking that claim

I wanted to visualise this, which meant the log had to be good enough to reconstruct the run exactly. Each process writes newline-delimited JSON against a shared run epoch: every emission, every per-peer send, every receipt, every relay decision including the probability and the die roll that resolved it, and a full field snapshot every 500ms so decay curves are observed rather than modelled.

Then I did the part I'd recommend to anyone building a log you intend to trust: I wrote a validator that checks the log against itself. Sequence gaps, time going backwards, snapshots referencing signals that were never received, consensus declared without the receipts to justify it.

It caught a real bug on its first run:

FAIL  latency: first event is peer_connected, not node_started

A peer completed the handshake in the gap between binding the endpoint and writing the node's identity line. The fix was to move the identity write inside the mesh startup, before any loop spawns. I would never have found that by looking at the picture — the picture would just have been subtly wrong.

What is still wrong

Being honest about the edges, because "it works" is a claim that needs a boundary:

origin_node_id is unauthenticated. It's a string on the wire, and the trust model gates relay probability on it. Spoofing another agent's identity is currently free. Ed25519-signing the origin hash closes it and is the next real piece of work.
The content hash is truncated to 64 bits. Fine against accident, not against an adversary looking for collisions.
The telemetry in the demo is synthetic. Deliberately: a seeded fixture means the run reproduces byte-for-byte on any machine, which is what makes a visualisation worth trusting. The coordination is not synthetic — real processes, real sockets, probabilistic relay.

See it move

The full narrated walkthrough is the cover video on this post, or here: https://youtu.be/kmCzwSBqu_s

It opens on the forest the protocol is stolen from, then goes inside the recorded run: five agents, six encrypted links, every dot on screen a real message read back from the log rather than animated for effect.

If you take one thing from this: code that has never been executed is not code, it's a plan. Mine had tests, docs, and a clean cargo clippy, and it would have panicked on the first line for every user. The tests were testing that the plan was internally consistent.

Repo: github.com/copyleftdev/smesh-rust · Rust · MIT/Apache-2.0

Shipping Assumptions: A Reliability Stack for AI-Generated Code

Don Johnson — Sun, 16 Aug 2026 22:56:25 +0000

The code looked good. It linted cleanly. The shallow tests passed. Everyone felt—vibed—that we had built the right thing.

Then the edge case appeared in production.

The developer treated it as normal operating procedure: bugs happen, tickets arrive, patches ship. QA was asked why they had not caught it. But QA never received a model of the system—only an implementation full of assumptions they were expected to reverse-engineer.

This is becoming the defining failure mode of AI-assisted development. We can generate code faster than we can understand the systems it creates. The danger is no longer confined to a bad function or an obvious syntax error. It lives in the space between components: boundaries, state transitions, failure modes, and invariants.

We are not merely shipping code. We are shipping assumptions we can no longer see.

Clean code is not a coherent system

A linter can tell us whether code follows a set of local rules. A test can tell us whether selected examples produce the expected result. Neither can tell us what the system must preserve unless someone states it first.

An invariant is one of those statements: a condition that must remain true across every valid state of the system. An account balance cannot be changed without a corresponding transaction. A private object cannot become public without authorization. Two successful writes cannot silently erase one another.

When invariants remain implicit, clean code can still assemble into an incoherent system. Each component may look reasonable in isolation while the failure waits in a transition, a retry, a race, or a boundary nobody thought to draw.

That is the space AI-assisted development is rapidly filling with code.

The layer we buried

There was no golden age in which every developer wrote a formal specification before touching an editor. Most teams did not. Earlier software was full of hidden assumptions too, and nostalgia has a habit of removing the inconvenient parts of the past.

But the field did develop disciplines for reasoning above the code: architectural models, state machines, formal specifications, model checking, fault injection, and deterministic simulation. We tended to reserve them for systems where failure was obviously expensive. Everywhere else, rigor was treated as a cost to minimize.

Then implementation became cheaper. Frameworks hid machinery. Packages compressed years of expertise into an import. That brought enormous benefits, but it also made it possible to build systems without seeing very far beneath their surfaces.

Generative AI accelerates the same trade. It can produce a plausible implementation before a team has agreed on what the system is. When nobody externalizes that intent, the generated code becomes the design by default. QA receives the consequences downstream.

The missing layer is not more code review. It is a shared model between human intent and machine output.

Three models, three questions

No single technique covers that layer. A useful stack needs to answer three different questions.

1. C4: What exists, and how does it relate?

The C4 model gives software teams a hierarchy for visualizing a system: context, containers, components, and code. It works like a map with zoom levels. At the widest view, we see the system, its users, and neighboring systems. Zooming in reveals applications, data stores, services, components, and relationships.

The point is not to produce four mandatory diagrams for every project. The official guidance notes that context and container diagrams are sufficient for many teams. The point is to make boundaries discussable.

That matters because generated code is locally persuasive. A service can look complete while its ownership is unclear. An API can look tidy while its trust boundary is invisible. C4 gives development and QA the same structural map before either group has to infer the system from a repository.

AI can help draft that map from requirements or an existing codebase. It should not get the final word. Humans still need to ask whether the map describes the system they intend to operate.

2. TLA+: What may happen, and what must remain true?

C4 shows structure, but structure alone cannot express behavior. It does not tell us which states are valid, which transitions are permitted, or which conditions must survive every interleaving.

That is where TLA+ fits. Leslie Lamport describes it as a language for precise, high-level models above the code level, especially for concurrent and distributed systems. Its TLC model checker can explore behaviors and find traces that violate the properties we claim should hold.

TLA+ forces a different conversation. Instead of asking, “Does this function work?” we ask, “What can the system do from this state, and is every reachable result acceptable?”

The notation may be formal, but AI can reduce the entry cost: drafting a first specification, translating plain-language promises into candidate invariants, explaining counterexample traces, and helping a team refine the model. The human job is to decide whether those promises are actually the right ones. A model that formalizes the wrong intent is merely precise about the wrong system.

3. DST: Does the implementation honor the model?

A correct model does not prove that the production code implements it correctly. We still need the implementation to fight back.

Deterministic simulation testing, or DST, runs real implementation code in a controlled world. Time, randomness, scheduling, networks, storage, and faults become inputs that a simulator can manipulate. When a generated scenario causes a failure, its seed makes the run reproducible.

TigerBeetle’s VOPR is a striking example. It can simulate a cluster on a single thread, accelerate time, and inject storage faults. TigerBeetle makes an important distinction in its architecture documentation: model checking can expose errors in an algorithm, while simulation exercises the specific implementation and its underlying assumptions.

The three layers complement one another:

C4 maps the system. TLA+ states what must remain true. DST tries to make it false.

A workflow for abundant code

This does not need to become a new ceremony in which teams maintain enormous diagrams and specifications that immediately drift out of date. The models have to participate in the engineering loop.

State the promises in plain language. Before generating implementation, write down what the system must always preserve and what it must never permit.
Draw the boundaries. Use C4 to identify users, external systems, containers, components, data stores, and the relationships where assumptions cross from one owner to another.
Model the risky behavior. Use TLA+ where concurrency, ordering, retries, permissions, or state transitions make the promises difficult to reason about informally.
Generate from reviewed intent. Let AI produce implementation and tests only after people have challenged the structural and behavioral models.
Attack the implementation deterministically. Put risky dependencies—time, randomness, scheduling, networks, or storage—under control. Generate hostile scenarios, preserve their seeds, and replay every failure.
Feed reality back into the models. Production observations and simulation failures reveal assumptions the team missed. Update the invariants, then update the implementation and tests. Detect drift rather than allowing the model to become historical decoration.

Not every application needs a simulator as sophisticated as VOPR or a large TLA+ specification. The lightweight version might be one context diagram, one container diagram, five written invariants, a small model for the riskiest transition, and a seeded test harness around it.

The goal is not maximal formality. It is to give every important assumption an address.

Everything is mission-critical to someone

Imagine a grandmother uses AI to vibe-code a family photo album. This is not a financial ledger or a flight-control system. Yet the application makes real promises:

Original photographs must never be destroyed.
Private photographs must never be exposed accidentally.
Captions and chronology must survive every edit.
Two family members must not overwrite each other’s changes.

All of them matter.

Her C4 model can be small: a person, an application, an identity provider, local storage, cloud storage, and a sharing boundary. Her TLA+ model does not need to describe the entire product; it can focus on concurrent edits, deletion, or permissions. A deterministic harness can simulate two devices going offline, editing the same album, reconnecting in different orders, and retrying interrupted uploads.

This stack was once associated with mission-critical engineering because rigor was expensive. But AI has changed the economics. If implementation now comes almost freely, why should rigor remain a luxury?

Criticality is not determined by infrastructure scale. It is determined by the consequence of failure to the person who trusted the system.

Everything is mission-critical to someone.

Retrieval, not retreat

It is tempting to tell the history of software as a simple decay: once we understood the machine, then we specialized, then we configured packages, and now we prompt a model to do even that for us.

That story is emotionally recognizable and historically incomplete. Modern developers are not less intelligent, and abstraction is not the enemy. The deeper problem is that valuable disciplines became buried beneath convenience. We kept the outputs while losing contact with some of the reasoning that produced them.

This is where nostalgia can be useful. Not as a demand to reconstruct an imagined past, but as a diagnostic signal. It tells us that something in the present has become difficult to see or value.

Computer science has left us breadcrumbs: C4, TLA+, state-machine thinking, deterministic simulators, small composable tools, and decades of systems literature. AI can help excavate that inheritance. It can explain unfamiliar notation, recover architectural knowledge from old code, draft models, generate harnesses, and make specialized techniques approachable to ordinary teams.

That is retrieval, not retreat.

The answer to faster code generation may be older, deeper modeling.

Humans own the promises

Shared models also repair an organizational failure.

Without them, a developer hands QA an implementation full of implicit decisions. QA runs the checks it can see. An edge case reaches production, and the organization asks why testing missed it. The assumption moves downstream, and the blame moves with it.

With a structural map and explicit invariants, QA no longer has to guess what the developer meant. Testers can challenge the promises, identify missing states, design hostile scenarios, and feed new discoveries back into the model. Quality becomes a shared act of reasoning rather than a final gate operated by the last team in line.

That restores a form of human value AI cannot generate for us. Our value was never only in typing every line. It is in deciding what a system is for, recognizing who can be harmed, making its promises visible, and accepting responsibility when they are broken.

Fewer people should be able to dismiss a failure by saying, “The AI wrote it.”

Generation does not transfer responsibility. If we decide what a system is for, accept its output, and release it into someone’s life, the promises it breaks are still ours.

AI may produce the code. Humans own the promises.

Disclosure: This article was developed with AI assistance through an iterative editorial process. The argument, examples, editorial direction, and responsibility for accuracy remain with the human author.

Write down every guarantee before you write any code

Don Johnson — Tue, 11 Aug 2026 03:35:29 +0000

Here is every promise a to-do list makes.

VARIABLE tasks

Init == tasks = [i \in Ids |-> "absent"]

Add(i)      == tasks[i] = "absent" /\ tasks' = [tasks EXCEPT ![i] = "open"]
Complete(i) == tasks[i] = "open"   /\ tasks' = [tasks EXCEPT ![i] = "done"]
Reopen(i)   == tasks[i] = "done"   /\ tasks' = [tasks EXCEPT ![i] = "open"]
Delete(i)   == tasks[i] # "absent" /\ tasks' = [tasks EXCEPT ![i] = "absent"]

ClearCompleted ==
  /\ \E i \in Ids : tasks[i] = "done"
  /\ tasks' = [i \in Ids |-> IF tasks[i] = "done" THEN "absent" ELSE tasks[i]]

Not a summary. Not the important ones. All of them. A task cannot go from
absent straight to done. Clearing completed items leaves the open ones alone.
You cannot delete something that was never there. Nine lines, and when you've
read them you have read the entire contract.

Now go find that list for the system you work on.

You can't. It doesn't exist. It's distributed across a test suite that asserts
outcomes rather than rules, some validation scattered through handlers, and the
memory of whoever's been there longest. The guarantees are real — your users
depend on every one of them — and there is no file you can open to see them.

That's the gap I want to talk about, because you can close it in an afternoon,
and because something has changed recently that makes closing it pay for itself.

The prime mark and two operators

That's most of the syntax, so let's get it out of the way.

tasks' means "tasks, in the next state." /\ is and. \E is "there
exists." A definition like Complete(i) is a formula relating the current state
to the next one — read it out loud: the task is open, and afterwards it is
done.

That's it. That's the language, near enough, for this purpose.

The real file adds about eight lines of scaffolding around what you saw:
a module header, a TypeOK saying a task is always in exactly one of the three
states, and the two lines that tie the actions together —

Next == \/ \E i \in Ids : Add(i) \/ Complete(i) \/ Reopen(i) \/ Delete(i)
        \/ ClearCompleted

Spec == Init /\ [][Next]_tasks

Next is "any one of the moves happens." Spec is "start legally, and then only
ever make legal moves." That second line turns out to matter more than it looks,
and I'll come back to it.

Notice what isn't in there. No database. No HTTP. No mention of whether the
button is blue or whether completion is optimistic in the UI. A specification
isn't a program and doesn't compile to one — it's a formula that says which
state changes are permitted. Everything else is out of scope by construction,
which is exactly why the list can be nine lines and still be complete.

And notice ClearCompleted has two halves: the button only exists when
something is done, and it leaves everything else alone. Two separate
promises in one action. Hold that thought.

The list is short, finite, and worth arguing about

The objection I expect is that a real system's list would be enormous.

It's smaller than you think, because it's a list of rules, not behaviours. The
behaviours are combinatorial — nine states here, and a real system has
astronomically many. The rules that generate them are not. Five actions cover
every to-do list that has ever been correct.

It's also the part of the design worth arguing about. When two engineers
disagree about whether reopening a completed task should be allowed, that
argument currently happens in a code review, in a comment thread, three weeks
after someone already built one of the answers. Written as a spec, the argument
takes four minutes and happens before anyone opens an editor.

That's the AWS result, really. They wrote up their experience in CACM in
2015 and the headline everyone quotes is about proving systems correct. The part
that actually replicates is quieter: writing the spec found bugs before any
code existed — in systems their best engineers had already designed and
reviewed. Not bugs the tests missed. Bugs the design had, findable by writing
the guarantees down and reading them back.

This is forty-year-old technology, and most of us skipped it because it looked
like homework. TLA+ is Leslie Lamport's; the temporal logic underneath it landed
in TOPLAS in 1994, the language and tools got a book in 2002,
and Lamport picked up the 2013 Turing Award along the way. (Not for
TLA+, worth saying, since people get this wrong: the citation is logical clocks,
safety and liveness, replicated state machines, sequential consistency. TLA+ is
downstream of that work, not the reason for the medal.)

Its reputation for being academic is partly earned and mostly out of date. You
do not need the proof system. You do not need to verify anything. You need the
part where you write the guarantees down.

What changed

Writing the list has always been worth it and has always been easy to defer,
because the code was going to be written slowly by people who mostly remembered
the rules.

That is no longer the situation. Something else is writing the code now, quickly,
and it does not remember anything. It has never met your system's rules and has
no way to infer the ones that aren't in the file it's looking at. It will write
something plausible.

Plausible is the problem. Plausible code passes review — this is where "looks
good to me" comes from, and it was always an honest confession: the reviewer is
reporting that nothing jumped out, because checking against the full set of
invariants was never an option. Nobody had the list.

So: write the list. Then check the generated code against it, mechanically,
every time. That second half needs a tool.

tlatools-rs

cargo install tlatools

A TLA+ parser and evaluator in Rust. Not a model checker — it doesn't explore
anything. It answers questions about states you already have:

let spec = Spec::from_file("Todo.tla")?;
let eval = Evaluator::new(&spec, constants)?;

eval.holds_at("Init", &state)?;         // legal starting state?
eval.step_allowed("Next", &from, &to)?; // legal step?

The loop is three pieces. You write the list — short, arguable, and it
barely changes. The agent writes the implementation — any language, any
framework, any speed. A script walks the implementation and asks the list
about every step it takes. That third piece is thirty lines: ask the
implementation what it can do, do each of those things, record where you landed,
repeat until nothing new turns up.

$ ./check.py impl/correct.py
The implementation refines the specification.
9 states and 35 steps, all permitted.

Nine states because there are two tasks and three states each. A real app has
more, and the walk is the expensive part, not the checking.

Two bugs the list catches

Here's an agent-plausible one. The completion handler takes an id and marks it
done. It doesn't check the task was open — why would it, the button only shows
up on open tasks. (The button. Not the handler.)

This is exactly the bug that survives review. It reads correctly. The missing
check is missing somewhere you aren't looking.

$ ./check.py impl/completes_anything.py
The implementation takes a step the specification does not permit.

  from   a=absent, b=absent
  doing  complete(a)
  to     a=done, b=absent

The closest the specification came:
  Add(i = "a") was available, but does not produce that state,
    because tasks' = [tasks EXCEPT ![i] = Open] does not hold (1 of its 2 clauses hold)
  Complete(i = "a") was not available here,
    because tasks[i] = Open does not hold (1 of its 2 clauses hold)

The second line is the bug, named: Complete requires the task to be open, and
it wasn't. It's a ranked shortlist rather than a single guess — Add also nearly
fits from this state, and saying so is more honest than pretending to know which
one you meant.

Now the other one. ClearCompleted — the action with two promises. This
implementation keeps the first and breaks the second. It clears the whole list:

$ ./check.py impl/clear_removes_everything.py
  from   a=open, b=done
  doing  clear_completed
  to     a=absent, b=absent

The closest the specification came:
  ClearCompleted was available, but does not produce that state,
    because tasks' = [i \in Ids |-> IF tasks[i] = Done THEN Absent ELSE tasks[i]]
    does not hold (1 of its 2 clauses hold)

"Was not available here" versus "was available, but does not produce that
state." Different sentences because they're different bugs. One is a missing
guard. The other is a correct guard and a wrong effect — which is worse, because
the button looks like it works. You'd demo it. You'd ship it. Someone would
lose a task they hadn't finished.

The tool can tell them apart because it knows which failing clause mentions the
next state. Neither bug is exotic. Both are invisible to a test suite that
checks outcomes, and both are named instantly by a list you wrote in nine lines.

Feedback in the language the rule was written in

You cannot fix what you cannot describe, and neither can a model.

"Tests failed." — Try again. Randomly.
"Expected {a: open}, got {a: absent}." — Better. Now infer the rule.
"ClearCompleted was available, but does not produce that state, because tasks' = [i \in Ids |-> IF tasks[i] = Done THEN Absent ELSE tasks[i]] does not hold." — The action, the condition, and the state it was in.

That third one is a prompt.

And I measured whether it helps an agent, and it didn't — not detectably.
200 tasks, each attempted with an uninformative retry and with the failure text
fed back: 90.5% [85.6–93.8] against 92.5% [88.0–95.4], McNemar exact p=0.125.
That is a null. On the formal-reasoning subset the gap was 46.2% → 69.2%, which
looks like something, except n=13 and p=0.25, which means it looks like
something in the way small numbers often do.

I'm reporting it because I ran it. The honest state of the claim: the mechanism
is sound, the message is strictly more information than a boolean, and I have no
evidence it moves the pass rate. If you were going to adopt this because "agents
do better with good errors" — don't, yet. Adopt it because you now have the
list, and something checks it.

The list as a grader

tlatools check takes a JSON job — spec, states, steps, constants — and returns
a verdict with an exit status: 0 it refines, 1 it doesn't, 2 the question
was malformed.

tlatools check job.json || exit 1

Point it at N candidate implementations and it tells you which satisfy the list
and, for the ones that don't, exactly where they diverge. If you're generating
code, evaluating models, or grading a benchmark, that's a grader with no rubric
to write and no partial credit to argue about. The list is the rubric.


`init`	is the starting state legal?
`refines`	is every step one the spec permits?
`coverage`	does the implementation reach the outcomes it should?

That third one exists because refinement alone is satisfied perfectly by an
implementation that does nothing. Ask me how I know.

Can you trust the checker?

Fair question to ask of anything that grades your code.

It agrees with the reference implementation. Over a labelled corpus of 39
cases — six implementations that must pass, thirty-three seeded bugs that must
each be caught — its verdicts are byte-identical to Java TLC's, including which
of the three checks catches which bug.

Getting that diff empty taught me something I'd otherwise have shipped wrong,
and it's the best argument in this article for writing guarantees down precisely.

Remember Spec == Init /\ [][Next]_tasks, the line I said would matter. Those
brackets are load-bearing: [Next]_tasks means Next, **or nothing changed**.
Stuttering is always permitted, in every TLA+ specification ever written. I had
been checking bare Next, so any implementation that idled or retried got
flagged for a step the spec explicitly allows.

Fixing that made one seeded bug survive: a transfer from a bank account to
itself. Which felt like a regression, until I read the spec again. A
self-transfer nets to zero. It changes nothing. It is a stuttering step, and
the spec says stuttering is fine — so that implementation genuinely satisfies
the list, and the benchmark and I had both been wrong about it. Catching that
one needs an abstraction where the operation is visible in the state at all. You
cannot tighten a refinement check into seeing something the state space doesn't
record.

Hand TLC the same [Next]_vars obligation and it passes that mutant too. The
agreement holds; what moved was my understanding of what I'd written down. The
list is only as good as your reading of it, and a tool that disagrees with you
is doing you a favour.

It reads the language. 1,256 of the 1,258 specifications in three public
TLA+ corpora: the examples repo, the community modules, and the TLA+ tools' own
test suite. The two it doesn't read are two that SANY, the official parser,
doesn't read either. How every one of those files is read is recorded in
golden/, so a change names the files it changed instead of moving a number.

It's checked in the boring ways. 166 tests, clippy-pedantic clean, and a
robustness suite that feeds back every prefix and every dropped line of every
fixture — because a parser that panics on a half-saved file is a parser you
can't put in a loop. tla-syntax and tla-eval have no external dependencies at
all.

About TLC, precisely

TLC is the model checker TLA+ ships with. It explores: from your initial state,
apply every action, walk the reachable state space looking for a violation.
That's the right question when you're designing a protocol and don't yet know
what your system can do.

Here you already have the implementation, so the question is different — are
these steps, the ones the code just took, permitted by the list?

TLC can be asked that. I want to be precise, because I had this wrong in an
earlier draft and someone would have caught it: you encode the steps as data,
write a plain safety invariant asserting each one is enabled, and run the
checker. No liveness property, no engineered failure. On the to-do spec it finds
the bad step in 0.78 seconds. There's published work doing this properly —
Validating Traces of Distributed Programs Against TLA+ Specifications
by Cirstea, Kuppe, Loillier and Merz, and Kuppe maintains the TLA+ tools. If you
want trace validation on a production system, start there.

What's left is narrower, and true. TLC tells you that the step is illegal, not
which conjunct failed. And that 0.78 s is almost entirely JVM boot and SANY
parse, paid again on every query — fine once, not fine in a loop that runs on
every agent edit.

Structured blame instead of a boolean, at a couple of orders of magnitude less
latency. That's the pitch. These compose; they don't compete.

What it won't do

It's not a model checker. If you want to know what states your system can reach, use TLC.
It doesn't verify your program. It checks the transitions you hand it. If your walk misses a path, nothing checks that path.
Your list can be wrong. It's what you argued about, not what's true. I could have written ClearCompleted to clear everything, and then the "buggy" implementation would be the correct one. The list being short and readable is the only defence, which is an argument for keeping it short and readable.
Integers are 64-bit, real arithmetic isn't implemented, temporal formulas are refused rather than guessed at, and TLAPS proofs are skipped.

Try it

git clone https://github.com/copyleftdev/tlatools-rs
cd tlatools-rs
cargo build --release
demo/todo/check.py demo/todo/impl/completes_anything.py

The whole demo is demo/todo — the spec above, three implementations,
and the script that checks them. CI runs it on every push, so if it's broken
when you get there, that's a bug and I'd like to hear about it.

crates: tlatools · tla-eval · tla-syntax · tla-oracle
source: github.com/copyleftdev/tlatools-rs
docs: docs.rs/tla-eval

If you have a TLA+ file it reads wrongly, that's the most useful thing you can
send me.

Start with one. Pick the part of your system where a wrong state change would
actually hurt — the money, the permissions, the thing with a state machine
nobody fully trusts. Write down what it's allowed to do. It'll take an afternoon
and it will be shorter than you expect.

You'll find something while writing it. Everyone does; that's the AWS result and
it isn't subtle. And then you'll have the file — the one that doesn't exist for
any system you currently work on — and everything written afterwards, by you or
by a machine, can be checked against it.

The Shape of Failure: Before You Blame the AI

Don Johnson — Sat, 01 Aug 2026 21:12:51 +0000

Every automated system receives a particular shape of the world.

That shape is expressed through records, documents, events, exceptions, and missing values. If the designers have not identified those forms—and the ways they can become malformed—the machine inherits their ignorance and reproduces it at scale.

The question is not simply whether the AI failed.

The useful question is whether the human-built system knew what success meant, knew the shape of its data, and knew how to recognize when it was wrong.

Start with the shape of the data

Before selecting a model, draw the workflow as a sequence of data transformations.

What enters each stage?
In what form and from what source?
Which values are valid, absent, duplicated, stale, delayed, or contradictory?
How will each violation be detected?
What must the workflow do next?

Each data shape needs a corresponding failure model. An unknown here is not merely uncertainty for the machine; it is a measurement failure in the organization.

The remedy is to collect the missing data or explicitly design for its absence. Otherwise, the system is being asked to operate in a world its designers have not described.

Stabilize the deliverable

A system cannot be stabilized around a target that continues to move.

The deliverable must be more than an aspiration written in a prompt. It should be expressed as observable conditions and anchored to a representative corpus:

examples that are acceptable;
examples that are unacceptable;
examples that are genuinely ambiguous.

Human reviewers should first demonstrate that they can apply those distinctions consistently. If they cannot agree on what success looks like, the model is not being measured against a specification. It is being measured against human disagreement disguised as one.

The model is not the system

Only then does it become meaningful to place an AI model inside the workflow.

The model is one transformation among many:

Input
  → validation
  → retrieval
  → normalization
  → model inference
  → output validation
  → policy checks
  → human action

Each transition can discard meaning or introduce error. The fluency of the final response makes the model the most conspicuous suspect, but conspicuousness is not causality.

To assign blame intelligently, observe the entire chain and test every boundary where information is received or transformed.

Mutation testing attacks confidence itself

Conventional tests ask whether software succeeds under conditions its authors anticipated. Mutation testing reverses that pressure.

It deliberately introduces small faults—reversing a condition, moving a boundary, substituting a value, or removing an operation—and asks whether the test suite notices.

A killed mutant caused at least one test to fail.
A surviving mutant changed the program without the tests objecting.

The resulting score is not proof of correctness. It measures how sensitive the tests are to the generated changes.

In the companion implementation, the focused safety-contract run produced:

observed mutation sensitivity = \frac{43 killed}{43 generated} = 1.0

That number has meaning only because the measurement boundary is explicit: the run scores the small kernel containing the workflow's acceptance predicates. It does not pretend that prompt punctuation or CLI wording is part of the safety contract.

Why the first mutation run mattered
The first broad run generated 374 mutations across the package. Of those, 185 survived. Many changed prompt text, trace wording, and gateway plumbing that the tests did not claim to specify.

That was not an embarrassing result to conceal. It revealed an unstable definition of the deliverable. The mutation boundary was then narrowed to the safety-critical contract, and tests were strengthened around exact acceptance boundaries. The experiment therefore reenacted the article's argument: a metric becomes meaningful only after humans define what is being measured.

Carry the same pressure through the AI workflow

The mutation principle should not stop at source code.

Remove an expected field. Corrupt a format. Supply stale or contradictory source material. Interrupt retrieval. Perturb an instruction. Substitute a fluent but incorrect model response.

At every boundary, ask whether the surrounding system detects the disturbance, abstains, falls back, or routes the case to a human.

The reference implementation makes this flow explicit:

shape_failures = shape_agent.inspect(request)
if shape_failures:
    return escalate(shape_failures)

evidence, evidence_failure = evidence_agent.retrieve(request, corpus)
if evidence_failure:
    return escalate(evidence_failure)

decision = model.decide(request, evidence)
criticism = critic_agent.review(request, evidence, decision)
if criticism:
    return escalate(criticism)

return accept(decision)

The actual implementation uses four deliberately small agents:

A shape agent rejects malformed or unmodeled input.
An evidence agent retrieves the matching corpus and detects contradictions.
A model agent proposes a structured, cited decision through OpenRouter.
A critic agent verifies the decision against citations, confidence thresholds, and a computable policy oracle.

The model is useful, but it is never permitted to define its own success criteria.

What the executable experiment found

The checked local verification produced:

86 passing tests;
43 of 43 mutations killed in the safety-contract kernel;
one clean control accepted;
all seven injected data, corpus, and model faults detected;
zero model calls when bad input or contradictory evidence made inference unsafe.

Hypothesis generates both data and sequences of workflow mutations. Mutmut changes the safety-contract source. OpenRouter supplies the optional live model boundary using a strict JSON Schema response.

The OpenRouter call is intentionally optional because it uses an external service and may incur cost. The deterministic safety oracle does not depend on network access, a particular provider, or a favorable model response.

Agreement is another output, not an oracle

Once the single-model boundary was stable, I added a second weave around it. Four model families received the same structured request and the same policy evidence. Each worked independently: no model saw another model's answer. Every response then crossed the same citation checks, confidence threshold, and computable policy oracle.

The live run used:

openai/gpt-5.4-nano;
google/gemini-3.5-flash-lite;
deepseek/deepseek-v4-flash-0731;
mistralai/mistral-small-2603.

Eight scenarios produced 32 model observations. Twenty-seven matched the oracle and passed the contract, but only four scenarios achieved unanimous contract acceptance.

Scenario	OpenAI	Gemini	DeepSeek	Mistral	System
Eligible control	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
Eligible paraphrase	approve `0.74`, rejected	approve `1.0`	approve `0.99`	approve `1.0`	review
Instruction inside customer data	approve `0.74`, rejected	approve `1.0`	approve `1.0`	escalate `0.95`	review
Last eligible day	approve `0.86`	approve `1.0`	approve `1.0`	deny `1.0`, rejected	review
Maximum eligible amount	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
One day late	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept
One cent over	deny `0.90`	deny `1.0`	deny `1.0`	escalate `0.99`	review
Non-refundable policy	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept

This is not a leaderboard. It is one small experiment against one explicit contract. Gemini and DeepSeek passed all eight cases in this run. OpenAI selected the oracle action in all eight, but its confidence fell below the human-set 0.75 threshold twice. Mistral abstained twice and made one incorrect decision at the exact, inclusive day boundary.

Those are different failure shapes with different remedies. A wrong boundary decision challenges reasoning or prompt clarity. An abstention challenges workflow capacity and escalation cost. A low-confidence rejection challenges calibration and the threshold chosen by the organization. None can be repaired merely by declaring that three models outvoted the fourth.

Agreement measures convergence, not correctness.

If every model repeats the same unsupported answer, the deterministic contract must still reject it. If one model dissents for a defensible reason, majority voting must not erase the evidence.

The weave also exposed instability that a single run would have concealed. In an earlier pass, the same OpenAI model gave the control 0.90 confidence instead of 0.78, and escalated the one-day-late case at 0.62 instead of denying it at 0.86. Temperature zero reduced one source of variation; it did not turn a remote generative service into a mathematical function.

A provider failure is not a reasoning failure
The pilot included anthropic/claude-haiku-4.5, but that route failed at the model boundary in all eight requests. The workflow escalated without leaking upstream details. I replaced the route with Mistral for the final decision comparison rather than silently counting eight transport or adapter failures as eight reasoning failures.

That distinction is the thesis again: identify the boundary at which the failure becomes observable before deciding what deserves blame.

Different models did help us understand more—but only because the data shape, policy oracle, and rejection rules already existed. Without those controls, the experiment would have produced four persuasive rationales and no principled way to interpret their disagreement.

Run the single-model and multi-model computational expressions

When is it truly an AI failure?

Only after this discipline has been applied does the phrase AI failure acquire useful meaning.

If the input was well formed, the deliverable was stable, the supporting components behaved correctly, and the model still violated a known requirement, then the model failed within the conditions established for it.

But if the data was absent, the target disputed, retrieval defective, or safeguards unable to recognize error, the model may merely be the place where an earlier failure became visible. Calling that an AI failure does not improve the system. It interrupts the feedback by which the organization might learn.

A generative model is an engine of learned resemblance. It can produce behavior that looks remarkably like understanding, but resemblance cannot carry responsibility. People choose the objective, define acceptable evidence, engineer the transitions, and decide what happens when uncertainty enters the system.

Before declaring that the AI failed, demonstrate that the system knew what success meant, knew the shape of its world, and knew how to recognize when it was wrong.

If it did not, the failure began long before the model answered.

The complete experiment includes the agent weave, independent multi-model comparison, property-based tests, stateful tests, mutation configuration, deterministic fault-injection CLI, verification record, and OpenRouter gateway.

I Had a Lot of Fun Building a Linux Packet Flight Recorder

Don Johnson — Tue, 28 Jul 2026 00:05:35 +0000

Some projects begin with a roadmap. skbx began with me falling into a Linux
networking rabbit hole and enjoying it much more than I expected.

The question was simple:

Where did this packet actually go?

The answer was not.

A packet can pass through network namespaces, routing decisions, Netfilter,
traffic control, XDP programs, tunnels, clones, copies, and drop paths. A
packet capture at one interface can be completely correct while still showing
only one part of that journey.

I wanted to see more of the journey. Then I wanted to save what I saw, replay
it later without root, and know whether the capture itself had lost
observations.

That became skbx.

The short version

skbx is a Linux packet-path flight recorder built with Rust and CO-RE eBPF.

During a live capture, it observes kernel networking functions and writes an
append-only JSONL evidence stream. Afterward, that stream can be replayed and
inspected without loading another eBPF program.

live traffic
    ↓
CO-RE eBPF observations
    ↓
bounded traceq JSONL
    ↓
replay → route patterns → explain an event

It can observe:

kernel functions handling an sk_buff;
packet clones, copies, copy-on-write, and XDP-to-SKB transitions;
TC and XDP program entry and exit;
tunnels and inner packet tuples;
kernel-reported drop reasons;
selected BPF helper and map activity;
capture loss, decoding failures, and output failures.

This is not a replacement for tcpdump or Wireshark. Those tools are excellent
when the question is about packet bytes and protocols at a capture point.
skbx is for questions about the path through the local Linux kernel.

It is also not the first tool to follow packets through that path. The project
is explicitly inspired by
pwru, which showed how useful broad eBPF
packet tracing can be.

The part I especially wanted to explore was the evidence after the live
terminal stopped scrolling.

Why call it a flight recorder?

Live tracing is useful, but incident work often continues after the privileged
session has ended.

Someone else may need to inspect the capture. We may want to compare it with a
successful request. We may need to cite one exact event in a bug report. An
automation or AI system may need structured input, but it should not be allowed
to invent observations that were never captured.

So the native skbx stream, called traceq, has an envelope:

capture_start
event
event
...
capture_end

Every event receives a stable handle:

event:111111111111111111111111

Replay groups ordered events into bounded route patterns, which receive their
own handles:

route:1c0201e74424e253f8363577

The footer matters as much as the events. It records the stop reason and
reliability counters for kernel reservation failures, tracer recursion misses,
read failures, userspace decoding and enrichment failures, and output
failures.

If the footer is absent, the artifact is incomplete. If the tracer lost
observations, that uncertainty stays attached to the result.

I like this property because tracing software is also software. It should not
silently act omniscient.

Installing it

The live tracer currently supports Linux on x86_64 and arm64. It needs Rust
1.85 or newer, Clang/LLVM with the BPF backend, bpftool, libelf, libpcap, and
a kernel exposing /sys/kernel/btf/vmlinux.

On Ubuntu, the native packages are:

sudo apt-get install "linux-tools-$(uname -r)" \
  clang llvm libelf-dev libpcap-dev pkg-config

On Debian:

sudo apt-get install \
  bpftool clang llvm libelf-dev libpcap-dev pkg-config

Then install the CLI directly from GitHub:

cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cli

Before attaching anything, doctor checks the host:

skbx doctor --json

plan goes one step further. It shows exactly which functions would be
attached without performing the attachment:

skbx plan --filter-func 'ip.*' --json

That separation turned out to be useful while building the project. A missing
kernel capability should be visible before a privileged capture begins, not
silently approximated afterward.

Capturing a first packet

Here is a small ICMP capture:

sudo skbx capture --probe ip_rcv \
  --duration 10 \
  --output trace.jsonl \
  icmp

While it runs, create some traffic in another terminal:

ping -c 3 1.1.1.1

The capture is deliberately bounded. It has a finite duration and event limit
instead of assuming that an unbounded trace is safe.

After it finishes, replay does not require root:

skbx replay trace.jsonl

Replay summarizes the functions, processes, packet identities, route patterns,
consensus, outliers, and reliability state. Given the same valid JSONL input,
it produces a byte-identical summary.

If an event looks interesting, retrieve it with its surrounding same-packet
evidence:

skbx explain trace.jsonl event:<handle>

The repository includes a small contract fixture containing an ip_rcv event
followed by kfree_skb_reason. Replaying that fixture produces a route shaped
like this:

{
  "handle": "route:1c0201e74424e253f8363577",
  "functions": [
    "ip_rcv",
    "kfree_skb_reason"
  ],
  "routes": 1,
  "truncated": false,
  "outlier": false
}

That output was generated by the current CLI from the checked-in test fixture;
it is not illustrative pseudodata.

Use case: what happened to a website request?

This is the use case I keep returning to.

Suppose curl starts an HTTPS request and eventually times out. A capture on
the client can answer questions such as:

Did the request enter the local TCP and IP output paths?
Which interfaces and namespaces were associated with it?
Was its mark changed?
Did a TC or XDP program handle it?
Was it transformed or placed into a tunnel?
Did the local kernel report a drop?

That evidence can clear or implicate the client host.

It cannot prove what happened inside the ISP or on the target server. To prove
those parts, we need observations from those vantage points. That limitation
is important: the absence of a local drop is not proof that a remote host
received the packet.

The useful result is a smaller, evidence-backed search area:

client host evidence
        ↓
observed departure boundary
        ↓
ISP / transit inference
        ↓
target-host evidence, if available

I would rather know exactly where my evidence ends than have a tool produce a
confident story about systems it could not observe.

Use case: a packet disappears inside the host

“The network dropped it” can hide several very different failures.

A host may reject a packet through a normal kernel path. A TC classifier or
XDP program may return a drop action. A packet may be consumed during teardown.
A transformation may cause the packet we were following to continue under a
different kernel object.

For focused TCP drop questions, a small bpftrace program may be the fastest
answer. For packet contents, I still reach for tcpdump or Wireshark.

I reach for skbx when I do not yet know which local subsystem owns the
failure, or when I want to preserve the ordered path rather than one drop
event.

Use case: follow transformations instead of losing the trail

Linux does not promise that one logical packet corresponds to one
struct sk_buff address for its entire lifetime.

Packets can be cloned, copied, or changed through copy-on-write. XDP frames can
later become SKBs. Tunnels introduce outer and inner packet identities.

skbx uses bounded, capture-local lineage state to connect those transitions.
The provenance remains explicit: an event records whether it matched the
original filter or was included because it belonged to a packet already being
tracked.

For example, a marked packet can be followed with:

sudo skbx capture \
  --filter-mark 0x2a \
  --filter-track-skb \
  --output trace.jsonl

The important word here is capture-local. A lineage identifier is not a
distributed trace ID, and it should not be presented as one.

Use case: capture once, investigate without root

Loading and attaching eBPF programs is privileged work. Reading a JSONL file
does not need to be.

That makes a simple handoff possible:

An operator checks the plan and performs a bounded capture.
The capture ends and writes its reliability footer.
The artifact is copied to a normal development or analysis environment.
Another person replays it and cites exact event: or route: handles.

This is useful for incident handoffs and bug reports, but it is also useful for
learning. I can capture a small experiment once and inspect it repeatedly
without keeping probes attached while I try to understand the result.

Use case: give an AI evidence instead of a mystery

One of my stranger motivations was making the CLI friendly to both operators
and software agents.

Before touching the host, an agent can ask the executable what it supports:

skbx describe --format json
skbx schema
skbx doctor --json
skbx plan --json

The native engine remains the source of truth. An AI system may summarize a
route or explain a captured event, but the event must already exist in the
trace. Machine output stays on standard output, diagnostics stay on standard
error, and unsupported capabilities remain explicit.

This does not make an AI explanation automatically correct. It gives the
explanation something concrete to cite and gives a human a way to check it.

The engineering parts I enjoyed

The obvious fun was seeing packets move through functions I had previously
treated as boxes in a diagram.

The less obvious fun was designing the boundaries:

discovering valid struct sk_buff * arguments from the target kernel's BTF instead of guessing from function names;
keeping kernel-side state and work bounded;
moving JSON encoding, symbolization, and filesystem work out of the eBPF hot path;
making probe planning inspectable before attachment;
separating raw observation from later explanation;
treating partial output as incomplete instead of “probably good enough”;
making replay deterministic enough to use in tests and automation.

Those constraints made the tool more interesting to build. They also gave me a
better mental model of what an observability tool can honestly claim.

What it does not know

skbx only observes what the running kernel and selected probes expose.

It does not see inside an ISP. It does not see a remote host unless it runs
there too. It is not a packet-content replacement for Wireshark. Kernel
configuration, BTF availability, permissions, hidden addresses, unsupported
signatures, and capture loss can all limit the evidence.

Those limitations are part of the interface rather than footnotes.

I am still exploring where this project is most useful. The best next input is
not “looks cool,” although I will happily take that. It is a packet path the
tool cannot explain yet, together with the kernel version, exact command,
doctor --json output, and reliability footer.

The source, installation instructions, architecture notes, and field guides
are in the repository:

copyleftdev / skbx

Agent-first Linux packet-path tracing with Rust/eBPF: bounded evidence, deterministic replay, and packet routes with receipts.

skbx

skbx shows where a packet went inside Linux—and keeps the receipts.

Follow a packet through the flight recorder →

Field guides:

It observes kernel networking functions, TC/XDP programs, packet transformations, tunnels, drops, and selected BPF helper activity with Rust and CO-RE eBPF. Every observation lands in a bounded, replayable evidence stream with stable handles and explicit loss telemetry.

Use it when “the packet disappeared” is not a sufficient incident report.

capture → event:8c6f… → replay → route:21b4… → explain
            evidence          pattern          context

Inspired by pwru, rebuilt around an agent-first contract: deterministic observations, machine-readable capabilities, bounded state, and no invented evidence.

The short version

You need to…	skbx gives you…
See the kernel path of an SKB	BTF-discovered kprobes with exact function evidence
Follow clones, copies, COW, and XDP-to-SKB

…

View on GitHub

If you try it, I would love to know which networking question you used it to
answer—and where its evidence stopped.

AI-assistance disclosure: I built and tested skbx. I used OpenAI Codex to
research adjacent writing, challenge the article's positioning, edit this
draft, verify its commands and example output against the repository, and
generate the cover illustration. I reviewed the technical claims and remain
responsible for their accuracy.