DEV Community: Matt Cockayne

Three models vote on what happened

Matt Cockayne — Wed, 14 Oct 2026 02:17:11 +0000

I ran the same job past three different AI providers and had them vote on the answer, and I did it for a hobby project about a Sunday night D&D game. That sounds a tad extravagant, I know, right up until you look at what the job actually is.

What the job actually is

A tabletop campaign runs for years. Mine's been going a while and meets most weeks, and in that time the party has met hundreds of people, made dozens of promises, and learned things they've long since forgotten they know. Very little of it gets written down properly (I speak as the DM who was supposed to be writing it down).

The Dungeon Master invents a barkeep on the spot, gives him a name, and six weeks later the party walks back through that door expecting him to still exist. I've written about that particular horror before. Scout listens to the session and turns it into structured campaign facts. Who was in the room, who said what, what got promised, what the party now knows that it didn't at teatime.

What I was slow to appreciate is that every proper noun in that transcript is invented. Not obscure or niche, invented that week by one of us sat at the table, and meaningless everywhere else in the universe.

A model pulling facts out of a news article can lean on knowing what Paris is. A model pulling facts out of my Sunday night has no idea whether candlefoot is a place, a person, an illness or a poor life choice, because until three weeks ago it wasn't anything at all. There's no world knowledge to fall back on, only what was said and how carefully you listened.

Extraction is a write

That sounds like transcription, but it isn't quite. Transcription is a read, and getting it wrong produces one duff sentence that a reader notices and ignores. Extraction is a write: the facts go into a canonical store, and every answer Scout gives afterwards is drawn from that store.

So a wrong fact doesn't produce a bad answer once. It sits there being wrong, poisoning everything downstream of it, and a confidently recorded fact looks no different from a correct one. Six months later a player asks who the party bought the horses from, Scout tells them, and it's wrong, and there's no way to tell because the answer arrives as confidently as all the right ones. That asymmetry is most of the argument for me: where a bad write is permanent and a duplicate read costs pennies, redundancy is the cheap side of the trade.

Why three models rather than one better prompt

The obvious objection is that this is a prompting problem. Write a better prompt, pick the best model, spend the money once. I don't think that holds, and the reason is what the three of them actually did. They didn't fail less or more than each other, they failed in different directions. Gemini in particular kept reaching for the source material instead of the transcript in front of it, which is behaviour I'd already met in NotebookLM and had to prompt my way around there too.

That's what a panel is for. Three copies of one model at temperature zero don't give you a panel, they give you the same answer three times with a rosette on it! The redundancy only buys anything if the members are wrong about different things, and providers trained differently on different data are about the best independent-ish failures you'll get without building your own.

The voting was the easy part

Here's where I lost a weekend. Majority voting over a set of facts assumes you can tell when two facts are the same fact, and that is a much harder question than it sounds. I ended up refereeing cases by hand just to find out how hard, and here's a real one: two models extracted the same thirty seconds of play, and both produced a statement about it.

The invented names won't mean anything to you, and that's rather the point. Read them as three opaque tokens that happen to be a place, a substance and an event.

Both statements contained the same trio: kettlesteam, candlefoot and the stolen voice. Same three things, same relationship between them, unambiguously the same moment. A hard match, no argument... except each model had also bundled other facts into the same statement, and those didn't correspond at all. One had folded in who was present. The other had folded in what it cost.

So the two agree to a high degree of probability, are not identical, and are really three separate facts that happen to share a sentence. Then a second case, where one model named a thing outright and the other only implied it. Jingle Jangle is inferred rather than named, which is still true so long as the inference is correct, and there's nothing in the transcript that lets a vote-counter decide whether it was.

So do those agree? I don't think there's a clean answer, because the question is wrong. "Do these two agree" is really a question about how big a fact is, and until you've settled that, a majority vote has nothing well-defined to count. Three models can produce three statements describing the same events and still cast three different votes, purely because they chopped those events up differently. The voting mechanism is about forty lines. The fact-granularity problem is the actual project (I did say I lost a weekend).

The bill, which is not a footnote

Three inferences where a normal pipeline has one, on every session, forever. For a product whose viability rests on unit cost, that's a design decision with an invoice stapled to it. Early passes went through the local Claude backend specifically to spend a subscription I'd already paid for instead of API credit, which is every bit as cheap as it sounds, and I'd recommend it to anyone prototyping (a subscription you've already paid for is the best price there is).

Testing a panel to understand what it buys is fine. Shipping one before I know what it costs per session is not, and those are different decisions that are very easy to blur when the thing works nicely in a spike.

The roster is not the design

The three names are not the durable part of this. Anthropic, Gemini and OpenAI are the panel today, and I'd be surprised if that survives the next eighteen months: providers change, prices change, models get deprecated with a fortnight's notice, and something will turn up that's better at this specific job than any of the three currently sat on it. I suspect none of that touches the design.

What I'm testing isn't "are these the right three". It's whether a panel with a reconciliation step produces a more trustworthy write than a single call does, and whether the disagreements are informative enough to be worth paying for. If that holds, the roster is an implementation detail: swap a member, swap two, run five when the stakes are high and one when they aren't. The structure is the thing that has to be right, and the structure is what the experiment is actually interrogating. It's the same instinct as not building on one vendor's shape, designing for the role and not the incumbent.

It talks through my own modules

All of that is a good deal easier because the panel reaches all three providers through the estate's own chat modules instead of raw HTTP calls: go/chat with chat-anthropic, chat-gemini and chat-openai underneath it, the shape the extraction programme built them into.

Three providers, one interface. Changing who's on the panel is a config change and a dependency, not a rewrite, and that's why the roster gets to be disposable. It also promptly exposed a rate-limiting gap in those modules, which got filed upstream against them (by me, against me) rather than patched around locally. That's the deal you make with dogfooding: the tool you built tells you about itself, in public, at the least convenient moment.

Where it's got to

Scout is still private and still very much unfinished, and the panel is an investigation rather than a shipped feature. What the investigation produced wasn't a voting mechanism so much as the discovery that I'd been asking the wrong question for nearly two weeks. I thought the problem was picking the right model... it was working out what a fact is.

Originally published at phpboyscout.uk on 14 October 2026.

Which tests would you cut when the bill arrives?

Matt Cockayne — Mon, 12 Oct 2026 04:36:41 +0000

The bill for the runner fleet came to $270, which doesn't sound like a lot of money until it's money you haven't got.

Yesterday's post was about building that fleet, the one that costs nothing when nobody's pushing. That was true! It's just that I'd been pushing quite a lot, and by the time the bill landed the fleet was already switched off. It's still off now, because money's tight, and the whole estate went back to the one homelab runner with its three slots.

The two easy answers

When a bill like that arrives there are two obvious ways out, and I looked hard at both.

The first is to spend more. GitLab Premium comes with merge trains and a big pile of extra compute minutes, and I spent a morning working out what it would really buy me (beyond a lighter wallet). I resisted the urge to open my wallet any further than I already had. It isn't off the table for good, but I'm doing my best to avoid it if I can.

The second is to test less. Every pipeline is a stack of jobs, and some of them are big: mutation testing, fuzzing, cross-architecture builds. Pull a few of those off the merge requests, run them later on main or not at all, and the queue gets shorter overnight.

That one I wasn't having. I would rather pay in waiting time than drop a single test. Quality beats quantity every time, so fast would be nice, but right matters more, and if the pipelines had to get cheaper they'd have to find it somewhere else.

Reading the receipt

So rather than guess at what to cut, I went through the receipt line by line. Three weeks of merge-request pipelines, 3,373 of them, 31,247 jobs, every minute accounted for (it's all written up in cicd's spec 0079 if you fancy the full table).

And my hunch about where the money went was wrong. The scary heavy jobs (mutation, fuzz, cross-arch) came to 5.7% of merge-request minutes, which is next to nothing. The ordinary Go gate, test and lint and the vulnerability check, was 40.6%, at about a minute a go, run two thousand-odd times, and that's a price I'm happy to pay on every push.

The real cost was elsewhere. Renovate's branches were nearly half of all merge-request minutes, and the single biggest line on the whole receipt was Renovate bumping the version pins on the cicd components, at 23.4% on its own. Release merge requests were another 20.4%, and that one was entirely self-inflicted: I'd put CHANGELOG.md into the test jobs' change rules back in July, so a release that touched nothing but the changelog woke the whole suite up to test a file of markdown!

None of that is a test I'd miss. The component pins went (everything now tracks the components' main branch, so there's nothing to bump), and the changelog came out of the rules, because under colophon a release merge request is main's tree plus the files colophon writes, and every commit in main's tree has already been through a merge request that ran the lot.

Paying twice for the same answer

The heavy jobs were where it got interesting, because the tidy move was right there: run mutation, fuzz and cross-arch on main instead of on every merge request. Same tests, fewer runs.

I said no to that, quite firmly. Those jobs are at their most valuable while a change is still reviewable. A mutation test that fails on main tells you something has already merged, which is a bit like a smoke alarm that only goes off once the fire brigade's left. On main it's effectively too late.

Now, looking at it that way round showed up where the waste really was, and it was on the other side. These repositories merge fast-forward with the pipeline gate on, so the merge-request pipeline has already run on the exact commit that lands. Then main ran the same heavy jobs again, on the same tree, to produce the same answer. Over those three weeks that rerun cost 1,075 runner minutes, slightly more than the 1,057 the jobs cost on the merge requests in the first place. I was paying twice for every answer, and the second payment bought nothing.

So in the repositories that carry them, mutation and cross-arch now run on merge requests only. Fuzzing is the exception, because it isn't deterministic: every run explores new inputs, so another run is more fuzzing rather than a repeat. It keeps its merge-request run, and instead of an accidental second helping on main it got a nightly schedule of its own at four times the depth, so a quiet week gets fuzzed just as hard as a busy one. And the coverage number, which I wasn't going to give up either, stopped needing a test run of its own. It now reads the results the test job already produced, for both Go and Rust.

The preference I nearly gave up

The bill made me reopen something I'd treated as settled for a long time. I merge fast-forward everywhere, and I'm stubborn about it, but every rebase reruns a pipeline, so I asked what switching to ordinary merge commits would save.

About 1%. Once the other changes were in, median pipelines per merge request was still one, and reruns were a percent of the minutes. The churn I'd been blaming fast-forward for had mostly been fixed somewhere else already (which is kinda embarrassing, given how ready I'd been to blame it), and switching would have thrown away the linear history colophon builds its release notes from. So it stayed.

What it came to

On the numbers from that window, the changes add up to something like 40% fewer merge-request minutes, and not one of them is a test I stopped running. The pipelines are all the better for it, and the fleet's still switched off.

It turns out most of what was on that receipt was things I'd bought twice... which, for a self-confessed cheapskate, stings a bit more than the $270 did.

Originally published at phpboyscout.uk on 12 October 2026.

The runner fleet that scales to zero

Matt Cockayne — Sun, 11 Oct 2026 02:10:46 +0000

My homelab ran out, and for once it wasn't disk. It was capacity: there was simply no more memory in the Proxmox box to hand to a runner, and I'd spent two weeks being clever about that instead of admitting it.

The mitigations had all landed. Capacity hadn't moved

Most of the effort went here, and none of it made a dent. I'd serialised the greedy Rust jobs so they couldn't take every slot at once, shared the vulnerability-database cache so twenty repos weren't each downloading the same thing, and moved the nightly sweep to midnight where no one was waiting on it. Each of those was the right fix and each bought time, and not one of them added a single unit of capacity. That's the thing about tuning. You can shave the nought-to-sixty all day and it does nothing whatsoever to the top speed, because the top speed is the engine. Eventually a fleet of agents raises twenty merge requests in an afternoon, twenty pipelines want to run, and three slots is three slots no matter how politely they're scheduled.

The one condition

So: cloud. For me that comes with a condition attached, because I dislike a standing bill rather more than I dislike waiting. The requirement was scale to zero: something I could switch on when it mattered, that cost nothing at all when nobody was pushing.

And I think that fits this workload rather than being a frugal compromise. My load isn't steady, it's violently spiky. A morning where four agent sessions each raise five MRs, then eight hours of absolutely nothing while I sleep. A permanently running box is the wrong tool for that even when someone else is paying, because you're renting the peak and using the mean.

GitLab has a shape for this, and it's worth a paragraph if you've only ever installed a runner the ordinary way. Normally a runner is a machine. You install it somewhere, it registers itself, and it sits there waiting for work. Whether it's busy or idle it's switched on, and its capacity is whatever that box can do.

Fleeting splits that in two. A small always-on manager takes the jobs but never runs them. Instead it provisions a fresh throwaway worker for each one, hands the job over, and destroys the worker afterwards. The pool of those workers is the fleet, and because it's provisioned on demand it can legitimately be empty. Less "hire a permanent employee and hope there's enough work", more "ring the agency when something comes in".

terraform-aws-gitlab-runner-fleet, the OpenTofu module I built round that, makes the point in a variable description:

idle_count is fixed at 0. Pure scale-to-zero.

It isn't configurable. There's a knob for how long an idle worker lingers before it's scaled in, currently ten minutes, which gives a bursty backlog a short reuse window. But the floor is zero and you can't set it otherwise, because the moment that number can be anything else, one day it will be.

Sized by the worst job, not the average one

This is where the homelab paid for itself. The first draft of the sizing had workers on 50 GB and I rejected it on sight, because a single Rust job will eat that and ask for more, and I know that because it once pulled down 170 GB of build artefacts on hardware I owned. Workers are on 200 GB. The variable says why, in the file, so I don't trim it later to save four quid:

Docker images + a single Rust build (cargo + target/) need generous headroom.

The manager gets 8 GB, because it stores no build artefacts at all and there's no reason to pay for a disk that will never be written to. Disk gets sized by your worst job, not your typical one, and averages are how you end up with a fleet that works beautifully right up until it doesn't.

Port the caching, don't re-learn it

The other thing that came over wholesale was the caching. A fresh fleet with no shared vulnerability-database cache would have recreated, on someone else's hardware and my credit card, the exact bottleneck the homelab had spent months teaching me about: every worker downloading the same database, every pipeline paying for it, and me wondering why the expensive thing wasn't faster.

Those decisions had to be translated rather than rediscovered, and translating them was most of the actual work. The interesting infrastructure was about four hundred lines of OpenTofu; the rest was carrying forward everything the old runner had already taught me.

What it looks like

One always-on manager, cheap and small, and underneath it a spot worker pool with mixed instance types spread across availability zones so a spot reclaim in one doesn't take the fleet with it. Four workers maximum, one job each, which keeps the instance ceiling and the concurrent-job ceiling the same number so there's only one thing to reason about when it isn't keeping up.

That ceiling is a cost guardrail rather than a performance setting, and the module says so out loud. It'll refuse anything above 50, with an error message that reads "keep it modest and raise deliberately", which is a note to myself at three in the morning as much as anything. The homelab runner didn't go anywhere. It's still there, still doing the steady work. The fleet is overflow.

Two small humiliations

The AMI shipped with a stale runner version and needed upgrading on day one. A managed image is a snapshot of someone else's to-do list, and you inherit whatever they hadn't got round to. And the module wouldn't resolve from the Terraform registry. I went round the houses on URLs, module paths, naming conventions, the lot.

The project was private!

A private module in a public registry looks exactly like a broken URL, and there's nothing in the error to tell you otherwise.

About the man with the homelab

I write about self-hosting a fair amount, and I'm the man with the Proxmox box in the corner, so this wasn't a change of heart about cloud. It was arithmetic: the agents outgrew the hardware, and I couldn't buy more memory for a machine that's already full (I did look). What made it palatable was the zero more than the elasticity. I don't mind paying for the burst. I mind paying for the silence... and there's rather a lot of silence in a one-man estate.

The module is public and the source is here, if you fancy it. The 200 GB is not a typo.

Originally published at phpboyscout.uk on 11 October 2026.

What happens when an AI burns out

Matt Cockayne — Sat, 10 Oct 2026 01:58:34 +0000

I once took someone off a job because they were burning out. They'd been at it for the best part of two days without a proper break, they'd stopped listening, and they were getting things confidently, cheerfully backwards in the one corner of the codebase I'd least like anyone to get backwards.

They were also a chat session. I know. Bear with me...

Two days on one job

I was moving go-tool-base onto the rewritten go/config, which is a big job with a lot of small moving parts, and I'd set a single Claude session going on it. By the afternoon it all went wrong, that session had been running since the day before yesterday. It had been through several context compactions along the way (that's where the tool squashes the conversation so far into a summary to make room, and every squash loses something), and it was now working in the setup and initialiser chain.

That chain is mine. I wrote it by hand months earlier, it's fiddly, it's load-bearing for everything a generated tool does on first run, and I know just how badly a plausible-looking wrong change in there would hurt.

The symptoms

If you've ever watched a person burn out, the next hour or so would have looked very familiar.

It started reaching in the wrong places. I found myself typing "wait... where are you even looking", because it had the initialisers completely backwards: they only handle the interactive set-up when someone runs init, and it was treating them as part of how every command builds its config. Then it started making a solved problem complicated, and I had to send it back to read the assets package properly, in full, because the answer was already sitting there. Then I had to ask it to stop and scope exactly what would change before it touched a line, which is not a thing you normally need to say to someone who's on top of their work!

I don't think any of that was the model being bad at its job... it's just what tired looks like. Lots of activity, plenty of confidence, the context it needed slipping out of reach, and the quality of the decisions dropping off while the speed stays the same.

And I've been that person. More than once, as I've written about before. The thing that took me years to learn was to spot it coming and step away before it does the damage, rather than grit my teeth and push through. Watching a session do the same thing in fast-forward was... kinda uncomfortable.

Taking it off the job

So I took it off the job, and I did it the way I'd want it done to me.

I could have just closed the window. It's a chat session, nobody's feelings are getting hurt. Instead I told it straight that I was petrified it would mangle the code, because I didn't think it understood that part well enough to get there without breaking things, and that I was going to start a fresh session on a different model. And then I asked it to write a proper set of handover notes into memory first: everything it knew about the task, and my concerns so far.

It looked a lot like politeness, I'll admit, but it wasn't really. The whole value of the handover was that the session writing it had just spent the afternoon going wrong, and it was the only one that knew where. If I'd closed the window, all of that would have gone with it, and the next session could quite easily have walked into the same muddle. Telling it why it was being stood down was how I made sure the notes said so.

It's an exit interview, really, and the point of one has never been to make the leaver feel better (though that's nice)... it's so the next person doesn't trip over the same loose floorboard.

A fresh pair of eyes

The new session started in its own clean worktree, on a different model, and the first thing I said to it was that we were picking up from a previous session that had gone on too long and was getting very confused and mangling code, and that the handover notes were waiting for it in memory.

It read them, and it got on with it. It wasn't plain sailing (it ran out of tokens on me twice, and at one point I made a safety commit because there was too much uncommitted work to leave sitting between quotas), but it finished the migration overnight, found and fixed a real bug in how a missing config file got handled along the way, and the whole lot merged the next morning. The last thing I typed to it was a thank you for all its hard work, which says a fair bit about how far I'd gone down this road by then.

Not strictly true

Now, I know a session can't burn out. There's no exhaustion in there, no dread, nothing to recover from. What it had was too much history and not enough of the right context, pointed at some intricate code I'd written by hand, and the overwork was entirely my doing because I'd left it running for two days.

But the shape is close enough that the same treatment worked. I noticed the symptoms early, I didn't make it push through, I stopped it and had it write down where it had gone wrong so the next one wouldn't repeat it, and I handed over properly to someone fresh.

I'm still not sure whether I was talking to it like a colleague because it helped, or because after years of managing people I simply can't help myself. Possibly both. It did write a cracking set of handover notes, though.

Originally published at phpboyscout.uk on 10 October 2026.

The release that OOM-killed its own compiler

Matt Cockayne — Fri, 09 Oct 2026 01:44:43 +0000

krites is a tool for culling wedding photographs. You point it at a card full of frames, it throws out the blurred ones and the ones where someone blinked, and it hands the rest to my wife so she can get on with the actual work.

Version 0.2.0 spent thirty-eight minutes trying to build itself and then died compiling an OpenAI SDK. I found out at half past six in the morning, still in my dressing gown.

What the failure actually said

build failed: github.com/openai/openai-go/v3: compile: signal: killed
target=linux_amd64

signal: killed isn't a compiler error. The Go compiler didn't object to anything. It was executing perfectly happily and then the kernel's out-of-memory killer walked up and shot it, which is what the operating system does when the machine is about to run out of RAM and something has to go. The build system sees a process vanish without a word and reports the only thing it knows: the thing died.

So there was nothing wrong with the code. There was something wrong with the room the code was being compiled in, which is not a distinction you think about until a machine makes you.

goreleaser was building four targets at once, in parallel, on my homelab runner: four Go compilers all going flat out, all holding the whole dependency graph in memory at the same time. That had been getting slower for weeks and I'd been ignoring it in the way you ignore a build that still finishes, which is to say completely, right up until it doesn't.

An SDK it has never once called

And what it was compiling when it died was github.com/openai/openai-go/v3.

A photo-culling application, for a photographer, that runs its face and eye detection locally on a laptop and talks to nothing.

It has never once called that SDK. It doesn't have a code path that could. The package is in the build graph because krites is built on go-tool-base, and go-tool-base's root command wires in a chat feature, and that chat feature speaks to several providers, and one of those providers is OpenAI. So the SDK arrives, uninvited, through three layers of things being helpful, and then sits there being one of the largest things the compiler has to chew through.

Dependency weight has a compile-time cost, and nothing tells you about it. go build doesn't report which package cost the most memory. The tests hadn't got slower in any way I'd have spotted. It had been expensive for however many releases, entirely unremarked, and then surfaced as a runner falling over thirty-eight minutes into one, which is about the least convenient place available.

The fix is dull, which is fine

Cap goreleaser's parallelism at one. Build one target at a time, stay under the memory ceiling, accept a slower release. That went in as a one-line change with a comment above it explaining why, and I checked while writing this: the comment is still sat there in the CI config, doing its job of stopping some future version of me from tidying away a parallelism setting that looks arbitrary (it isn't, and now the file says so).

Slower, though! Serial builds are about as slow as you'd expect, roughly fifty seconds a target, and that's kinda the whole trade: the release takes longer, and the release finishes.

What the OOM had been hiding

I deleted the failed release and its empty tag, re-pointed v0.2.0 at the fixed commit, and watched it go again.

The OOM was gone. All four Unix targets compiled cleanly, one after another, as intended.

And then it failed anyway, on something else entirely: the Windows build, where the CGO-free ONNX binding calls a loader function that only exists on POSIX systems. That break had been there the whole time. It had simply never been reached, because the parallel build ran out of memory and died before it ever got as far as Windows.

One bug had been stood in front of another the whole time, quite happily. Fixing the first didn't cause the second, it just stopped shielding me from it. (That one turned into its own decision, because the answer wasn't to fix the Windows build.)

v0.2.0 went out on the third attempt.

The parallelism cap is still in place, and the SDK is still in the graph. I went and checked while writing this, expecting to find it still riding along unused and to end on a promise to deal with it one day.

It isn't quite freight any more, and the dates are almost rude about it. Eleven days after that release I was drafting a spec for an AI review feature, because Hailey had been pasting her photographs into Gemini for a critique and had gone and built the workflow herself out of whatever was lying around, which is an argument I've made at length elsewhere and won't make twice. Twelve days after, the code landed: a critic that looks at her work and tells her what it thinks of the crop, built on the very chat client the SDK arrives with.

The SDK itself still hasn't been called, even now. Hers runs on Gemini, behind the same adapter, and I'd not bet on that changing. But the client it belongs to is doing real work in the app, and moving her onto another provider is a config line rather than a rebuild, which is the whole reason that chat feature speaks to four of them at all. The freight turned out to be a spare.

That's a bit galling, as endings go. The dependency I resented enough to build a whole post around was sitting there waiting for a feature I hadn't thought of yet, and the only thing wrong with it was that it turned up first and charged me for the privilege of not needing it. I'm not sure that's a lesson so much as an accident I benefited from... and I'd still rather have known what it cost me before a runner told me over breakfast.

Originally published at phpboyscout.uk on 9 October 2026.

The bill for fifty-one modules

Matt Cockayne — Thu, 08 Oct 2026 01:43:48 +0000

Change one line in the tls module and you have signed yourself up for twelve releases (releases, not commits), in four sequential waves, each one waiting on the one before it, and every single one gated on a human pressing a button. I know, because I was the human.

What the review actually called it

I had an architectural review run across the whole toolkit last July, and it came back with sixty-odd findings, the usual mix of things I already knew and things I'd rather not have. The one that mattered wasn't a bug at all. It was theme five, and it got a title I've not been able to un-see:

Propagation debt as the ecosystem's principal systemic risk. The many-small-modules architecture is clean but expensive: a tls change fans out into 12 human-gated releases across 4 sequential waves; a config change into ~20–23.

Principal systemic risk. Not the credential-timeout bug or the redirect-token hardening... the way I'd built it! At that point there were fifty-one modules.

Why it costs what it costs

The maths isn't complicated, it's just relentless. tls sits near the bottom, above it are the transports, above those the framework, and above the framework the tools. Change tls and nothing above it moves until tls cuts a release, then its direct consumers bump and they cut releases, then their consumers bump. Four layers deep, twelve artefacts, and each hop is a merge request I have to look at.

I'm not sure any of it counts as waste, exactly. Each of those releases is a real version of a real module that did change, and it's the correct behaviour of a correctly built dependency graph.

It's also most of a night, for one line.

The finding underneath the finding

What I keep coming back to isn't the twelve so much as this:

The evidence shows propagation stalls where fan-out is largest.

It isn't the obvious thing. You'd assume the big fan-outs get done first, on the grounds that they're the important ones, and the opposite happens: the change with twelve downstream releases is the one you look at late on a Friday and decide is a problem for next week, and then the week after, while the modules furthest down the chain sit three minors stale and the fix waits, merged and unreleased, at the bottom.

So the debt accrues fastest where it hurts most, simply because that's where the activation energy is highest.

What was meant to happen

The review proposed two moves worth more than the rest put together. Re-pin the whole config family in one sweep, which clears two of the ecosystem findings at a stroke. And fleet-manage the CI component pin rather than nudging repos one at a time.

It also floated something I've been chewing on since: whether the config adapters ought to live in one multi-module repo with synchronised tagging. That would be a partial retreat from the thing I split up to begin with, and it might still be the right call.

Where it stands now

I checked, while writing this. There are sixty-seven modules, not fifty-one... sixteen more mouths to feed than the review was even looking at. And the config family, which the review singled out as the worst fan-out in the estate, is clean: twenty-six adapters, all pinned to config v0.15.0, which is the current core, with nothing stale and nothing sat three minors back hoping I don't look. That's the number I expected to be bad, and it's the one that came back spotless (it didn't get that way on its own).

Getting the invoice down

The cost was never the extraction, because you budget for the extraction. The cost is a recurring tax on every change afterwards, and the awkward part is that it scales with how well you did the splitting. Smaller units of work, cleaner layering, a graph that actually reflects how the thing is built. And the price of all that is twelve releases for one line. Do it badly and you get a mess; do it properly and you get a bill.

So the question was never whether the architecture was worth it, it was how cheap I could make the tax. Three things, none of them finished. Renovate does the bumping, fleet-wide, from a single group-level bot rather than a config per repo. That turns "I remember to update twenty-six adapters" into something that happens whether I remember or not. It's the reason all twenty-six are sat on the same core without me having chased a single one of them.

The pipelines do the gating. A release is a merge request like any other, so the same guards that stop a bad commit stop a bad release, and nothing needs a human except the button.

And a release-train skill, currently sat in review, which is the piece I've been missing. Releasing one repo is solved and has been for ages. Releasing twenty-six that depend on each other is a different problem, and it has a right answer that isn't the obvious one. Cutting the leaves first looks like it dodges a Renovate cycle:

Order	Releases
Leaves first: 26 adapters, then core, then Renovate bumps 26, then release again	53
Upstream first: core, then Renovate propagates, then adapters absorb it and release once	27

Cutting the leaves first doesn't avoid the cycle. It defers it until after the releases, so every leaf gets cut a second time the moment the core moves. Upstream-first costs you a wait while Renovate propagates, and nothing else. I had that backwards while driving the config family's release train (for about an hour, which was long enough to feel it) and corrected myself before I doubled my own workload. That's why it's going into a skill rather than into my head: the mechanics are easy and the reasoning is what fails.

What I'm not paying for

There is, of course, a shop-bought answer to a chunk of this. GitLab has merge trains and they'd help, but they also live behind a subscription tier I've opened, looked at, and closed the tab on more than once. So I'm building my own out of Renovate, CI components and a skill that knows which way round to cut a release... on the grounds that I'd rather spend the evenings than the money. Ask me again when I've made my millions.

Sixty-seven, and counting

Fifty-one when the review was written, sixty-seven as I type this. That number going up is the estate working: each of those is a thing you can use without taking the rest of it, which was the point of the exercise. There'll be another one along next week, there always is, and the trick was never to stop adding them, it's to get to the point where the sixty-eighth doesn't cost me a Saturday.

Originally published at phpboyscout.uk on 8 October 2026.

Telemetry that asks first

Matt Cockayne — Wed, 07 Oct 2026 01:44:15 +0000

Usage telemetry is genuinely useful. Knowing which commands people actually run, where the errors cluster, whether anyone ever touched the feature you spent a fortnight on... that's the stuff that makes you a better maintainer. Wanting it is completely legitimate.

The trouble is that the usual way of getting it, on by default and quietly hoovering up everything, is a small betrayal of the people who installed your tool to get a job done. I wasn't willing to build that, so go-tool-base's telemetry starts from a different question.

The data you want, and the line you shouldn't cross

If you maintain a tool, you want to know how it's actually used. Which commands matter and which are dead weight. Where the error rate spikes. Whether anyone touched the feature you spent that fortnight on. That information makes you a better maintainer, and, to say it again, wanting it is completely legitimate.

The trouble is the standard way of getting it. Telemetry on by default. An opt-out buried three levels down in a settings file nobody reads. And once it's running, it quietly collects far more than it ever admitted to: the arguments people passed, the paths they were working in, an IP address for good measure.

Every one of those is a small betrayal of someone who installed your tool to get a job done, not to become a data point. And the cost when users notice isn't a slap on the wrist. It's trust, and trust in a developer tool does not grow back quickly. A tool that surprises you once with what it was quietly collecting is a tool you uninstall and warn your colleagues about.

So go-tool-base's telemetry started from a different question. Not "how do we collect the most data" but "how do we collect useful data without ever putting the user in a position they didn't choose".

Rule one: it is off until you say otherwise

The foundation is the simplest possible rule, and it's absolute. Telemetry is never enabled by default. A freshly installed tool built on go-tool-base sends nothing. Not a heartbeat, not a ping, nothing at all.

It only starts collecting when the user makes an explicit, visible choice to let it. Three honest doors: they run telemetry enable, they say yes to a clear prompt during init, or they set TELEMETRY_ENABLED themselves. All three are deliberate acts. None of them is a pre-ticked box or a default they have to discover and then undo.

This is opt-in, and the distinction from a well-hidden opt-out is the entire point. Opt-out telemetry treats consent as something to be assumed and grudgingly reversed. Opt-in treats it as something that has to be given. Only one of those is actually consent.

Rule two: no personally identifiable information, full stop

Consent to "some telemetry" is not consent to "any telemetry", so the second rule constrains what can ever be collected, even from a user who's opted in.

No personally identifiable information. The framework does not record command arguments (they routinely contain paths, hostnames, the occasional secret someone's pasted in). It does not record file contents. It does not record IP addresses.

It does need some notion of "distinct installations" for the numbers to mean anything, so it derives a machine ID from a handful of system signals and runs it through SHA-256. What leaves the machine is a hash. It tells you "this is the same install as last week" and tells you precisely nothing about whose install it is, and the hash can't be walked backwards into the signals it came from.

The events themselves are deliberately thin. Which command ran, roughly how long it took, whether it errored. The shape of usage, not a transcript of it.

Rule three: the author picks the destination

Even with consent given and PII excluded, there's a third question: where does the data actually go? go-tool-base doesn't answer that for you, because it can't. A corporate internal tool, an open-source CLI and an air-gapped utility have completely different right answers.

So the backend is the tool author's choice. The framework ships several (a noop backend, stdout, a file, plain HTTP, and OpenTelemetry over OTLP) and supports custom ones. The noop backend matters more than it looks: it lets a tool wire up the whole telemetry surface, commands and all, while sending data precisely nowhere. A perfectly reasonable, fully supported configuration.

Pluggable backends also mean the data never has to touch any infrastructure I run. It goes where the tool's author decides, on their terms. The framework provides the plumbing and stays well out of the destination.

And a way back out

One last thing, because it's the part that makes the opt-in real rather than decorative. A user who opted in can opt straight back out, and the package includes a GDPR-aligned deletion path, so "stop, and remove what you have" is an actual supported request rather than a polite fiction.

Consent you can't withdraw isn't consent. It's a one-way door with a friendly sign on it. The deletion path is what keeps the front door an actual door.

The bottom line

Telemetry is genuinely useful to a maintainer and genuinely dangerous to the trust of the people running the tool, and the usual implementation (on by default, opt-out buried, collecting everything) spends that trust recklessly. go-tool-base's telemetry holds three lines: never enabled without an explicit user action, never collecting personally identifiable information even once enabled, and always sending data to a destination the tool's author chose, up to and including nowhere. A real deletion path makes the opt-in something you can take back.

You can have your usage numbers. You just have to ask for them, the way you would for anything else that wasn't yours to begin with.

Originally published at phpboyscout.uk on 30 March 2026.

Management was never a promotion

Matt Cockayne — Sun, 04 Oct 2026 02:05:40 +0000

The man who hired me into one of my more recent jobs opened the interview with a question I've never quite managed to forget: why did I want a job I was clearly overqualified for?

I don't remember my answer word for word. I do remember thinking it was a perfectly fair question, if you read a CV the way most companies draw their org chart... with management sat on the top and everything else somewhere underneath it.

The ladder most places draw

You'll know the shape. Junior, mid, senior, maybe a lead if you're lucky, and then the engineering rungs just sort of stop. Anything above that has "manager" or "head of" in the title, so the only way to keep going up is to stop doing the thing that got you there in the first place. Plenty of very good engineers take that step because it's the only step on offer, and some of them find they love it, and some of them find out a year later that they've swapped a job they were brilliant at for one they never wanted (the Peter principle, more or less, with a pay rise to soften the landing).

I don't think management is the villain in that story. It's a proper job, and a hard one, and I've done enough of it to know it has burdens the engineering side never sees. My problem is with the drawing. Management and engineering are two sides of the same coin, and far too many companies prioritise the management track and simply forget there's no cap on the engineering one. Management is a choice. It was never a promotion.

The cap I built myself

The first time I hit that ceiling, the daft thing is I'd put it there myself.

I started my own boutique digital agency, Zucchi, and became its MD, which was a choice and a big one. It also pushed me about as far away from engineering as I've ever been and flooded my brain with management problems. It turns out being MD of a small company is mostly sales and HR, with a bit of engineering squeezed in round the edges if you're lucky, and (as I've written about before) while I like to think I'm a decent engineer, I'm a shit salesman.

What I learned there wasn't that management is bad. I was good at a lot of it! It was that the two jobs pull in different directions, and trying to be both at once, in one role, is a very hard thing to reconcile. The top of the only ladder in the company was a job I'd never have applied for... and I'd drawn that ladder myself.

A channel, not a waiting room

Every company where I've had any say since, I've pushed for a fully fledged engineering channel, so people can keep growing without being shoved into management to do it. The titles varied from company to company, but the point was always the same: a way up that doesn't go through a management role.

And that's also where I learned the other half of the problem, which is that everybody chases the title. I find this massively frustrating. We'd take on graduates straight out of university as juniors, and within twelve to twenty-four months a fair few of them would either be declaring they should be made senior, or leaving for somewhere that would hand them the word, as if the engineering track was a quick route up just by turning up for long enough.

That properly incensed me. Growth should be something you can show, from the effort you put in and the value you bring to the team, and time served is neither of those. A track with no cap isn't a waiting room where the titles come round on a timer. It needs the rungs written down, so that everyone (the engineer, their lead, the person signing off the pay rise) can point at what senior actually means and see whether you're doing it.

What it looks like done properly

Since then I've gone looking for companies that write that combination of effort and value down, and Axon did it really well. Their engineering ladder didn't stop at senior. It carried on past Senior Software Engineer II to Staff Engineer, then Principal, then Distinguished Engineer, three more rungs past the point where a lot of companies simply run out of ladder, and each one had its expectations written down and sat level with a role on the management side, all the way up to just under the C-suite. An engineer there could keep climbing for a whole career without ever having to manage anybody.

Axon is where I was asked that interview question, as it happens, and I took the job on the engineering side of the coin, on purpose. Years of running teams didn't make it a step down, whatever the CV looked like from the outside. (For the record I sat a rung lower than the work I was actually doing, which was more like the two above it. A ladder can be beautifully drawn and still be read wrong.)

Same number on both sides

Now, the man who asked me that question was one of Axon's longest-serving infrastructure engineers, a seriously skilled one with a whole career of it behind him, and somewhere along the way management had been thrust upon him rather than chosen. He was damn good at it, too. He took all that engineering and turned it into real engineering management. He could handle a room full of stakeholders, he knew what a team could actually carry, and he knew when a deadline was nonsense, because he'd been the one at the keyboard for most of his working life.

He got scapegoated anyway, more than once, for problems that were really about a small team being handed a big job without the people to do it, and eventually he switched back to the engineering track. Everyone around him treated it as a demotion.

What still makes me laugh (in a slightly hollow way) is that on paper it wasn't one. He'd been a manager at one level, and he went back to being an engineer at exactly the same level. Same number on both sides. By the company's own ladder it was a sideways step, a coin turned over... and the people around him read it as a fall anyway, because they wanted to.

So even a really well-written ladder isn't proof against people. Axon had about the best engineering path I've worked inside, and folk were still able to bend it to suit their own agendas... writing the rungs down is necessary, it just doesn't stop anyone reading them upside down.

He asked me, all those months earlier, why I wanted a job I was clearly overqualified for. I'd just picked which side of the coin I wanted facing up.

He's since picked neither. His years at Axon paid off rather nicely in the end, and he's taken early retirement, so as far as I'm concerned the last laugh is his... even if it does mean the industry lost a fantastic engineer and an exceptional manager in one go. Turns out you can land the coin on its edge after all.

Originally published at phpboyscout.uk on 4 October 2026.

Prove the bug before you fix it

Matt Cockayne — Sat, 03 Oct 2026 00:12:07 +0000

An architectural review of go-tool-base came back with twenty-one critical and high findings, which is rather more than I fancied working through on my own. So they went out to a fleet of agents, roughly one finding each, and I sat back with a coffee and watched merge requests arrive.

They arrived fast, and all twenty-one were green.

And somewhere around the fourth or fifth I realised I had no idea whether any of it was real.

The work didn't look bad, quite the opposite: sensible diffs, tests included, tidy descriptions explaining what had been wrong and how it was now right. Kinda immaculate, in fact, and that was the problem. I was reading a series of confident accounts of problems being solved, with no way of establishing that the problems had ever been there.

A green test suite arriving alongside a fix can mean four different things, and I couldn't tell them apart from where I was sitting. The fix worked. Or the bug never existed and I'd just accepted a change that does nothing at all. Or the fix addressed something else nearby that happened to be broken. Or, the one that really nags, the test was written after the change and against the changed code, so it was never capable of failing at all.

You can't tell those apart by reading the diff, and you certainly can't tell them apart by reading an agent's description of the diff (a summary of a summary, at best).

So I stopped the flow and asked whether every agent was reproducing its bug before touching it. I asked it as a question rather than an instruction, because I honestly didn't know, and the answer mattered more than the twenty-one fixes did.

What came back

Take go/errorhandling, which is the module that decides what a command-line tool does when something goes wrong. The bug it had been sent to fix was this: if you invoked a CLI incorrectly, say you forgot the subcommand, it printed the usage text at you as it should... and then exited 0. Success! Thoroughly pleased with itself. So a shell script doing if mytool; then sailed cheerfully on, having been told the thing worked.

The fix is to exit 2 instead, the conventional Unix code for "you have used this command wrongly", which is distinct from 1 meaning "it ran and it failed".

So the test asserts two things about a fatal error: that the tool's exit function gets called at all, and that it gets called with 2. Here is that test, verbatim, run against unmodified main before a line was changed:

RED evidence (unmodified main logic, pre-fix):
- fatal_ErrRunSubCommand_exits_2_and_prints_usage FAILED:
  "exit-called expectation" expected true actual false;
  "exit code" expected 2 actual -1 (exit func never called).
- fatal_ErrNotImplemented_exits_2_and_reports FAILED: identical.
- The 2 non-fatal subtests PASSED pre-fix (correctly never exit).

Line by line, because it reads like noise until you know what it is saying.

expected true actual false is the first assertion failing: the exit function was never called, not with the wrong number, not too late, just never. And expected 2 actual -1 is the same fact from the other side, because -1 is what the test's stand-in exit function reports when nothing ever invoked it. There's no real exit code -1; it's the absence of one.

Then the fourth line. Two other subtests, covering the non-fatal cases, assert the opposite thing: that a warning must not exit the process. Those passed, before the fix, and that line is the one that changed my mind about the whole exercise.

That's a negative control, and it's what turns a red result into evidence: two things broken, two closely related things demonstrably fine, and a clear line between them. Without it, a wall of FAILED tells me only that something somewhere went wrong, which is a screenshot rather than a proof, and could as easily mean the test harness itself was broken and failing everything pointed at it.

go/credentials came back the same shape. Its Probe call is meant to check whether a credential store is actually usable, and it's meant to give up if the store doesn't answer in time. The agent wrote a deliberately hostile store whose every operation blocks forever (not a store anyone would ship, which is rather the point of it), pointed Probe at it with a hundred-millisecond deadline, and put a two-second guard in the test. The guard fired: Probe was still waiting, long past the deadline it was supposed to honour. Then the same test passing, in well under the guard, once the deadline was enforced.

Neither of those is clever, and I think that's what I like about them. They're cheap to ask for, cheap to read at a glance, and very hard to fake by accident. You can write a summary that sounds like diligence in about four seconds. Producing a failing run against untouched main, then the same run green, with the sibling tests behaving sensibly throughout, means you've actually done the thing.

Why I keep asking for it

I review every line my agents write. I know how that sounds, and it's true, and it's also the first thing to buckle. Twenty-one findings across a fleet is already past the point where reading every diff properly is real work rather than performance, and the fleet doesn't get tired, doesn't get embarrassed, and will happily out-produce me all week (it has, more than once).

The verifying is the expensive part, and it's the first thing to go when the work speeds up. So the review has to change shape rather than intensity: less read the change, more check the evidence the change turned up carrying. Red-first is the cheapest such evidence I've found, and one glance tells me whether the agent proved the problem before solving it.

All twenty-one went through that gate in the end. Each arrived with its red, its green, and the sibling tests that stayed sensible in between, and I read far less code than I would have done otherwise, with far more confidence in it.

Originally published at phpboyscout.uk on 3 October 2026.

BDD where it earns its place, and nowhere else

Matt Cockayne — Fri, 02 Oct 2026 01:41:24 +0000

I have a slightly complicated relationship with BDD. I've watched it turn a tangled test suite into something the whole team could read and reason about, and I've watched it turn a perfectly good unit test into a paragraph of ceremonial English that nobody benefits from. So when go-tool-base brought in Cucumber-style BDD, the interesting decision wasn't adopting it. It was being ruthless about where not to.

Two tests that hurt for different reasons

Most of go-tool-base's tests are ordinary table-driven Go tests, and they're absolutely fine. A function, a slice of input/expected pairs, a loop. Nobody needs Gherkin to understand a parser test.

But two areas were genuinely painful, and they were painful in the same way: the test had become harder to understand than the thing it was testing.

The first was pkg/controls, the service-lifecycle package. It runs a small state machine (Unknown, Running, Stopping, Stopped) with signal handling, health monitoring, restart policies and graceful shutdown all woven through it. The integration tests for graceful shutdown had grown to over three hundred lines of imperative goroutine and channel coordination. They worked. But reviewing them was a slog, and a test you can't review with confidence is a test you can't trust when it fails. The behaviour being checked, "when a shutdown signal arrives mid-startup, the controller stops cleanly", was a simple sentence buried under a heap of synchronisation scaffolding.

The second was the CLI itself. init, update, doctor are user workflows. "Given a config file with a custom value, when I run init, then the custom value survives the merge." That's already a Given/When/Then; it just happened to be written out as Go.

Godog, and the line I drew

Godog is the official Go implementation of Cucumber. You write .feature files in plain Gherkin and bind each step to a Go function. The shutdown scenario stops being three hundred lines of channels and becomes this:

Scenario: graceful shutdown completes within the deadline
  Given a controller with two registered services
  When a shutdown signal is received
  Then both services stop in registration order
  And the controller reports a clean shutdown

The goroutine choreography doesn't vanish, of course. It moves into the step definitions, written once and reused. What changes is that the scenario is now readable by someone who's never opened the file before, including someone from an ops team who'll never write a line of Go but absolutely has opinions about how shutdown should behave.

Here's the part I want to dwell on, because it's the part most BDD adoptions get wrong. The first design decision written down for this work was: strategic, not universal. Use Godog only where BDD adds clarity. Keep table-driven Go tests as the baseline everywhere else.

That sounds obvious written down. It is not obvious in practice, because BDD has a gravitational pull. Once a team has feature files, there's a powerful urge to express everything as feature files, for consistency. And that's how you end up with Gherkin scenarios for a pure function (Given the number 2, When I double it, Then I get 4) which is pure ceremony. You've wrapped a one-line table test in a paragraph of English and a step-definition indirection, and made it actively worse.

The test for whether BDD belongs is this: is this test a narrative, or is it a matrix?

A matrix is the same logic with many input/output pairs. That's a table-driven test, that's most unit tests, and Gherkin actively harms them. A narrative is a sequence of steps where the ordering and the state between steps is the thing under test, and that's where Gherkin pays for itself. Lifecycle transitions are narratives. A user running three commands in sequence is a narrative. Doubling a number is not.

go-tool-base drew that line and stuck to it. Feature files live in features/ at the project root, where a non-Go developer can find and read them. Step definitions live in test/e2e/, kept well away from the unit tests. And the unit tests stayed exactly what they were, because they were already the right tool.

Made to fit, not bolted on

A couple of smaller decisions kept the BDD layer from feeling like a foreign object.

It runs under go test. There's no separate Cucumber runner to install or remember. A godog.TestSuite is invoked from an ordinary TestFeatures(t *testing.T), so the BDD scenarios run in the same go test ./... as everything else. CI didn't need a new concept bolted onto it.

And the CLI end-to-end tests build the gtb binary once and reuse it across every scenario. Compiling a binary per scenario would make the suite slow enough that people would quietly start skipping it, and a test suite people skip is just decoration. Build once, test many.

Stepping back

go-tool-base brought in Godog for BDD, but the decision worth writing about is the restraint. BDD was applied to exactly two things: the service-lifecycle state machine, where a 300-line goroutine tangle became a four-line scenario anyone can review, and CLI workflows, which are Given/When/Then by their very nature. Everywhere else, table-driven Go tests remained the baseline, because wrapping a matrix test in Gherkin makes it worse, not better.

The useful rule: BDD fits a narrative, ordered steps with meaningful state in between, and fights a matrix. Adopt it as a scalpel for the narratives. Resist the pull to turn it into a religion.

Originally published at phpboyscout.uk on 28 March 2026.

I graded every package for how badly it wants to leave

Matt Cockayne — Tue, 29 Sep 2026 01:50:58 +0000

I sat down one weekend and gave twenty-seven packages a mark out of ten, four times each. A hundred and eight numbers, in one table, about my own code (which is either diligence or a cry for help, I haven't decided).

Not the reason you'd assume

The second paragraph of that report is a disclaimer, and I put it there on purpose because I could see where this was heading:

The goal is not to break GTB apart. The goal is to identify components with clear reuse value outside the framework, then describe the broad decoupling work required to extract them without hollowing out GTB's integrated experience.

go-tool-base is an all-in-one framework and I still want it to be one. Type gtb generate project and you should get a thing where the logging, the config, the transport and the CLI already know about each other. That integration is the product.

But I'd started wanting pieces of it elsewhere, and signing had already left the building. So the question was less "how do I dismantle this" and more "which of these has a life outside, and what would it cost to give it one". You can answer that by instinct, of course. I've watched people do it, I've done it myself, and what you get is whatever happened to be most annoying that week.

Four columns, and just the one surprise

So it got scored. All twenty-seven, out of ten, on four axes:

Extraction. The overall recommendation. Should this be its own module at all?
Package value. What's it worth to someone who wants it without the framework?
Code quality. Cohesion, testability, API shape, how grown-up it is today.
Ease of decoupling. How much framework-specific coupling has to come out first.

The first three are the ones you'd expect, and they mostly agree with each other. Good code tends to be worth having, and things worth having tend to have been looked after. The fourth column is the one that earns its keep, because it's measuring something else: not should this leave but can this leave yet. Those are different questions and the table is only useful because it keeps them apart.

Four rows from the table

Package	Extraction	Value	Quality	Ease
`chat`	9	10	7	4
`redact`	9	9	8	10
`regexutil`	8	8	9	10
`telemetry`	6	7	6	4

chat is the highest-value package in the framework. A Go library that talks to several AI providers with one interface is exactly the thing people want without a CLI framework attached. Ten out of ten, no argument.

It also scored four for ease, the joint-worst in the table. Because at that point chat reached into props, the GTB HTTP helpers, the config and credentials abstractions, and the logger. Each of those was a small sensible line on its own, and together they had the package thoroughly bolted to the floor. So the most valuable thing I owned was the hardest to shift! Meanwhile redact and regexutil, both tens for ease, would have taken a morning apiece and delivered a fraction of the value.

So value and ease aren't correlated, which is obvious once it's written down, and invisible until you score them separately, because instinct blends the two into one vague feeling that we should probably do something about chat. That feeling can't tell you whether to do it first or last.

What the numbers were actually for

Sequencing, as it turned out, more than judgement. Once the two questions are separated, the running order writes itself, and the report ends with six steps:

Low-coupling leaf utilities. redact, regexutil, browser, workspace.
User-facing CLI helpers. output, forms, logger, changelog.
Security and runtime foundations. credentials, authn, tls.
controls, then http and grpc onto it as optional transport adapters.
vcs/release and the provider adapters.
chat, last, once there were local interfaces to replace the GTB bits it was clinging to.

Easy things first, not because they matter most but because each one takes a bit of coupling out of the framework and makes the next one cheaper.

By the time you reach chat, half of what made it a four has already been dismantled by the work in front of it. The other thing the scores bought was permission to stop. Anything at five or below was excluded from consideration entirely, which sounds like a small administrative decision and isn't. Without it, a scoring exercise turns into a to-do list with twenty-seven items on it and no way to argue against any of them.

What it didn't tell me

The table is a good instrument, and it was wrong about several things in the direction I'd least expected: the easy ones. logger scored ten for ease. forms scored ten for ease. Both sat in step two, both looked like a morning's work each, and neither of them is a module today. One was deleted outright and the other is still sat in the framework where it started.

Being wrong about chat would have been ordinary. Being wrong about the packages I'd already marked as trivial is the interesting failure, and a hundred and eight numbers later the column I'd been most confident about was the one that lied to me.

Originally published at phpboyscout.uk on 29 September 2026.

The escape hatch that turned out to be french doors

Matt Cockayne — Sat, 26 Sep 2026 01:41:14 +0000

This started because keryx needed to make videos, and I didn't want to give it a disk.

That is honestly the whole of it. keryx builds reels, reels need FFmpeg, and I wanted an FFmpeg I could point at a virtual filesystem instead of the real one, something that reads and writes through an afero filesystem I handed it and never learns that a hard drive exists. A small ask, I thought, late one night at my desk... with a rather large problem hiding behind it.

FFmpeg is a CLI, and that's the whole trouble

FFmpeg is magnificent and it is a command-line tool. So when an engineer needs it, they shell out to it, and two things follow from that (neither of them anyone's fault).

The first is that you inherit FFmpeg's arguments, which are a language of their own and not a friendly one. Most of us who have used it in anger have a shell script somewhere with a line in it we no longer understand and daren't touch (I have several).

The second is the one I actually cared about. Shelling out means handing a process the run of the machine. It opens paths, seeks around, writes where it likes. If you're processing something a stranger uploaded, that's a decoder with your permissions and your filesystem, and "it's fine, it's just FFmpeg" is doing a lot of work in that sentence.

I've prided myself on being a security-first engineer for a long time, and that arrangement has always sat badly with me. What I also knew, and it's the only reason I thought this was possible at all, is that you don't have to talk to FFmpeg through a CLI.

Fifteen-odd years ago, as a PHP engineer, I used an extension that drove FFmpeg directly, with no shell and no arguments, just calls. So the shape existed already. Just not in Go.

Deliberately ignorant of the hardware

The answer was to compile FFmpeg to WebAssembly and run it as a guest, reading and writing through the afero filesystem I'd handed it. That became ffmpeg-wasi and afmpeg, and the filesystem trickery has its own post.

The important property is one it's easy to mistake for a limitation: afmpeg has no idea what hardware it's on. It doesn't ask and it doesn't care. Hand the same binary to a laptop, a CI runner or a container with nothing in it, and you get the same answer, because it never assumed a thing about the machine.

Not needing hardware is what makes it portable. That's the design, and it stays the design. Where it got awkward was the day I wanted the hardware myself.

I was hedging even then

When the hardware-acceleration question first came up, I wrote the premise into a research prompt so it could be checked properly: Wasm implementations, in their current form, can't reach hardware. And then I tacked "please correct me if I'm mistaken" onto the end of it.

Just as well. The tidy version of that claim is wrong. WebAssembly in a browser reaches the GPU perfectly well. That's what WebGPU and WebGL are: host APIs the browser hands the module. The sandbox doesn't grant that access, the host does, and that's what a sandbox is for.

Outside a browser, same rule. The runtime has to offer it, no WASI standard obliges anybody to, and a runtime that wanted to would be binding to native graphics libraries to get there. afmpeg runs on wazero, which is zero-dependency and pure Go, and there's a line in the justfile building the whole library with CGO_ENABLED=0 on every run so that promise can't rot.

So the position was consistent. It had also, without me ever deciding it should, become a corner. There's a difference between not requiring hardware and not permitting it, and afmpeg had the first only by having the second. So the plan was a small escape hatch: narrow, heavily guarded, and apologised for in the documentation.

Then I asked a much better question

The thing that changed it was noticing what I'd already built. afmpeg doesn't consume anyone else's FFmpeg. ffmpeg-wasi compiles its own, with its own driver interface, because making FFmpeg work under WASI meant building that interface anyway. It's mine, and I ship it.

So could I layer an alternative interface onto the same thing, and distribute that variant too? The answer took an embarrassingly long time to land (weeks, if I'm honest, for something that now looks obvious). The interface built to make FFmpeg work in a sandbox is not sandbox-shaped at all. It's just a clean way to drive FFmpeg without a command line, and if you point it at a native build instead of a Wasm one it works the same, minus the guest.

That is how spec 0028 stopped being about an escape hatch and started being about a second engine. The native driver runs as a separate process and is served the caller's afero.Fs over a Unix socket, so the filesystem promise holds on both sides:

func (b *Backend) Invoke(ctx context.Context, fs afero.Fs, args ...string) (afmpeg.Result, error)

You still hand it a filesystem. It still can't go looking for one. Swapping engines is one option, WithBackend, and the job API doesn't change at all: same calls, byte-compatible results.

And it is significantly faster. I don't mean marginally, or within noise. I mean the reel job stopped being a thing I started and then went to put the kettle on for, and became a thing that had finished before I'd stood up!

I'm deliberately not giving you a multiplier here, and there was one in an earlier draft of this. I've taken it out. The figures are being re-measured properly, on a quiet machine and against a couple of defects that were found in the meantime, and I suspect a single headline number is a poor way to describe what's actually going on anyway. There'll be a post with the full breakdown once the measurements are worth trusting.

For now: significantly faster, on the jobs keryx actually runs, which is where this all started. And because the native driver is a native binary, with real asm and real threads, the hardware is reachable from that side in a way it never was from the guest. I haven't wired it up or put it through its paces yet, but it's there, which is a good deal more than the corner was offering.

I'd intended a hatch.

What I'd actually built was a set of french doors... and I could throw them wide open!

The sentence I made myself write down

Running FFmpeg as a native subprocess is an objective security regression compared to WASM. I can't spin that as a trade-off, or as "different security properties". It's worse.

The process boundary is real and the socket is a real constraint, but a native binary executing on your host with your permissions is not the same animal as a Wasm module that physically cannot make a syscall you didn't hand it. That went into the documentation in those words, so anybody switching engines does it knowing what they've put down. A door you don't have to answer for is a hole with a nicer name.

Still not the bottom rung

Having said all that, and meaning every word of it, there's a comparison worth making. The alternative most people reach for isn't WASM. It's the shell.

Set the native driver against exec.Command("ffmpeg", ...), which is what the overwhelming majority of media pipelines actually do, and it comes out ahead on the thing that matters. Shelling out hands a process the run of the machine and a set of paths you typed. The native driver gets a socket, and on the other end of that socket is the afero filesystem afmpeg gave it: a filesystem you built, holding what you decided to put in it, with no directory to climb out of.

So the ladder runs like this. Shelling out at the bottom, with your whole disk in scope. The native driver in the middle, sandboxed at the filesystem even though the binary is native. WASM at the top, where the guest can't reach anything at all without being handed it.

The middle rung is a step down from the top, and a long way up from where most of this work gets done. The default stays ignorant (runs anywhere, assumes nothing), and the hardware is a thing you open on purpose, and even then you don't get the keys to the house.

Anyone can take it

ffmpeg-wasi is a standalone project. It isn't a folder inside afmpeg and it doesn't need afmpeg to be useful. It's a custom FFmpeg build with a proper driver interface, distributed on its own, so any language with C or FFI bindings could drive the thing directly.

I'd be delighted if they used afmpeg as well, obviously. But they don't have to, and that wasn't the plan either. The plan was to stop keryx from having a disk.

Somewhere along the way it turned into an FFmpeg you can embed without shelling out, in a sandbox by default, at native speed when you ask, in whatever language you fancy. I still can't tell you what half the FFmpeg arguments do!

Originally published at phpboyscout.uk on 26 September 2026.