DEV Community: Richard Atkins

Earned Autonomy

Richard Atkins — Wed, 09 Sep 2026 21:25:22 +0000

Thursday, 9am

At nine on Thursday morning a container started in Azure, pulled the week's sources, clustered the same story where several outlets had covered it, ran a novelty check that threw out the duplicates and anything it had already published, capped each topic to a handful, then selected and rewrote 69 articles across four editorial voices, generated the images, wrote everything to the production database and told the host to rebuild the site and refresh the search index. Nobody was watching. I was making coffee. The first human to read any of those articles, was a reader.

Then the same job scored its own output. Every article went through mechanical checks, a Haiku judge tested a sample for claims the sources do not support, and a multi judge persona panel scored a stratified dozen. The summary landed in blob storage next to the telemetry: banned phrases 0, structure 1.0, groundedness 0.98, and cost $2.26.

If you'd asked me eighteen months ago whether I would let a language model publish to production with no human in the loop, I would have said no, and I would have felt responsible saying it. I still feel responsible but that "no human in the loop" stopped being a decision I make and became a rung the system has to earn.

That sentence is not original - "Autonomy is earned, not granted" is on half the vendor blogs of 2026. What you will struggle to find written down though is what the receipt looks like: what evidence, gathered how, over how long, against what bar, and what happens when the evidence goes bad. This piece is my answer, with the numbers from a pipeline that has been climbing the ladder since July.

The ladder

What does an agent have to show you before you let it act? My answer is a table. Every automated step in my pipeline sits on one of four rungs, the rung is a config value, and the code refuses to act above the rung you have given it.

Rung	The agent may	You do	It moves up when
1. Shadow	run, and write its output to a log	the work, as before	its logged output would have been right often enough, measured over several real cycles
2. Advisory	propose: a recommendation, a diff, a score	decide, every time	accept and override rates are healthy and the overrides have a pattern you can name
3. Assisted	act, with a check it must pass first	approve or veto the result, not the steps	the check catches every failure on its list and review time per item is measured and falling
4. Autonomous	act, unattended	read the scorecard	the scorecard has been green for long enough to trust, against a threshold you set before you saw the numbers

Three rules make it a ladder rather than a diagram.

Promotion needs evidence you gathered on the rung below, over more than one real cycle, against a bar you agreed in advance. Not a demo, not a benchmark, not a week of it looking fine - actual scored cycles of the real job.

Demotion is one config change, made the moment the evidence slips. That's the rule we'll all be tempted to skip. An autonomy scheme without a demotion path is a launch plan. The whole point is that trust is a variable you keep measuring, not a milestone you pass once.

And the rung belongs to a step, never to "the agent". The rewrite stage of my pipeline is on rung four. The ops-doctor that diagnoses its failures is on rung one. Same pipeline, same model family, same week. An agent is not trustworthy or untrustworthy. A specific action class, with a specific evidence trail, is.

Where the rungs come from

None of the rungs are mine, and you should be suspicious of anyone who says otherwise. Andrew Ng taught the spectrum in his MLOps course in 2021: human-only, then "shadow mode, where the learning algorithms are giving predictions but are not actually used", then AI assistance, partial automation and full automation, with the advice that "many deployment applications will start from the left and gradually move to the right". Behind Ng sits a forty-year-old scale. Sheridan and Verplank's ten levels of automation from 1978 run from "human does the whole job up to the point of turning it over to the computer to implement" through "computer helps determine options and suggests one, which human need not follow", "computer selects action and implements it if human approves", "computer selects action, informs human in plenty of time to stop it", to "computer does whole job if it decides it should be done, and if so tells human, if it decides he should be told". My advisory rung is their levels two to four. Assisted is five and six. Autonomous with a scorecard is seven: "computer does whole job and necessarily tells human what it did".

Shadow mode is the one rung Sheridan does not have, and it came from cars. Tesla has run its driver-assistance software in shadow on customer vehicles since 2016, predicting what it would do and logging the moments its prediction disagrees with the driver. By 2019 the pattern was standard in the machine-learning deployment literature: deploy the new model "side-by-side with the current one, as a shadow model, and send the same production traffic to gather data on how the shadow model performs before promoting it". Note the verb. Promoting.

So the ancestry is honest and the pieces are old. What the older frameworks do not have is the bit in the fourth column. Sheridan's scale is a design choice: you pick a level for a function at design time whereas Ng's spectrum has a direction but no gate. The deployment literature promotes models, not authority, and jumps from shadow straight to serving. The question none of them answers is the one OpenAI's 2023 governance paper left open: "How should a user or system deployer determine and enforce which interactions need human approval, as a function of a model's empirical reliability on that task and other factors?"

The ladder is my answer to that question. The rest of this piece is what the answer looks like when it is running.

Historical Precedents

If the ladder sounds like a novelty from the AI industry, look at how many older professions grant authority in exactly these steps, and how many of them wrote the evidence bar down as a number.

A junior doctor in a competency-based programme is entrusted with a professional activity on a scale that starts at "not allowed to practice; allowed to observe", moves through supervision "with supervisor in room ready to step in as needed", then "supervisor immediately available, all findings double checked", then only "key findings double checked", and ends with practising unsupervised. Observe, co-act, act-with-check, act. The Ottawa version of the same scale is written from the supervisor's side and reads like a changelog of trust: "I had to do", "I had to talk them through", "I had to prompt them from time to time", "I needed to be in the room just in case", "I did not need to be there."

Aviation is stricter and more numeric. When an airline wants approval to let the autoland system put an aircraft on the runway in fog, the FAA's classic criteria asks for "at least one-hundred (100) successful landings" in line operations first, with data collected on every attempt "regardless of whether the approach is abandoned, unsatisfactory, or is concluded successfully", typically over 6 months. Authorised minima then step down in stages, from 300 metres of visibility to 175 to 75, each step earned by the record of the one before. If "an excessive number of failures" turn up during the demonstration, the answer is more landings or a system modification, not a lower bar.

The bank regulators got there in 2011. The Federal Reserve's model risk guidance says that if validation finds significant deficiencies, "use of the model should not be allowed or should be permitted only under very tight constraints until those issues are resolved", and it describes shadow mode almost exactly: "parallel outcomes analysis, under which both the original and adjusted models' forecasts are tested against realized outcomes", where "if the adjusted model does not outperform the original model" it does not replace it. It even asks for "early warning" metrics that start shortly after model introduction. That is the whole ladder, in supervisory prose, fifteen years before anyone put an LLM in a loop.

And the deployment engineers have had the mechanics for years. Google and Netflix's canary analysis tool routes a small slice of traffic, one per cent in Google's example, through the change, scores it against a baseline, and then, in a sentence that contains all four of my rungs: "Success promotes the canary and continues the deployment, a marginal score can trigger a human approval path and failure triggers a roll back." The success and failure conditions live in a template, written before the rollout starts.

So neither the pattern nor receipts are new. What's new is that the thing climbing the ladder now writes prose, makes API calls, files pull requests, and could delete your production database and most of us forgot to bring the receipts.

The rewrite stage: four rungs in eight weeks

The article-rewriting stage is the clearest example because I can show you every step.

It started life on a local Qwen model, which had run unattended for a year, and the question in July was whether to move it to hosted Claude. So Claude went into shadow - which is where your new model should always start. For the full weekly set, both backends rewrote the same 85 articles, the outputs were logged, nothing from the new backend was published, and a blind panel of two judges from different model families scored both. Cloud 4.73 out of 5 against local 2.19, every persona, both judges independently agreeing. I wrote that up in the judge panel piece, including the part where the gate could pass on zero judgments.

That shadow evidence promoted the cloud backend straight past advisory and assisted, because the job it was replacing was already autonomous, and the promotion question was only "is this at least as good". Then the first production edition ran with me watching, which is rung three in all but name. Then again, unattended. Since then the cadence has moved to three editions a week, Wednesday, Friday and Sunday, with cross-run deduplication against the published corpus enforced from the first Sunday run. Same rung, more cycles, every one of them scored.

But autonomy for the rewrite stage came with a condition: online evals on every edition, thresholds fixed before the first run. Persona adherence at least 4.0 mean, groundedness at least 0.95, banned phrases zero, structure 100 per cent. Miss a threshold and the run is marked degraded in the run state and a human is told.

The thresholds are a promise I made to myself in a config file before I had any production numbers to be tempted by. Eugene Yan is right that setting a threshold is hard and that being off by 0.05 can swing precision or recall badly, and the honest answer is that mine were calibrated on the shadow data and then frozen. That is the order that matters. Calibrate on evidence from the rung below, freeze, then run. Never the reverse.

The gate is on rung one too

Here is the part I only noticed while writing this, and the part you are most likely to get wrong. The quality gate I just described is itself a new automated step, and it is on rung one. It is report-only. It can mark a run degraded, it can shout, and it cannot block a publish. Your first instinct will be to let it block. Resist that. It will earn blocking the same way everything else does: by being right for several cycles in a row.

Thursday's run showed why. The persona panel scored nothing. Both judges failed, and the summary said so in a field called judges_degraded rather than inventing a number. That is the fix from the judge panel piece holding up in production. But the reasons the judges failed were three separate bugs that had never had real input before: one judge rejected the schema because a dial name contained an ampersand, one had a missing dependency in the container image, and one had the wrong response shape hard-coded from the A/B run. All three are fixed. All three would have blocked a perfectly good edition if the gate had been allowed to block.

I believe that a gate that starts life blocking is a gate you will disable the first time it is wrong but a gate that starts life reporting is one you will still have around in six months.

The fixed panel, tested on three real articles, scored persona adherence at 3.0 to 4.4 against a threshold of 4.0, and in the first draft of this piece I wrote that the first fully scored run would probably read "below threshold" on that dial and that I would not move the bar. I was wrong about the number and I am keeping the sentence, because the wrong prediction is the useful part. The next scored run, the day after, judged all twelve sampled articles and the panel came in at 4.22. Persona cleared. What did not clear was groundedness: 0.894 against a bar of 0.95, and the run was marked degraded, report-only, exactly as designed. So the gate's first real signal was on the dial I was not watching, about the thing I care about most, and it came from a bar I had set before I had any numbers to be tempted by. That is what the bar is for. Moving the threshold to meet the result is how you get a gate that never fails. Hamel Husain puts the inverse well: if you are passing 100 per cent of your evals, you are probably not challenging your system enough.

The ops-doctor: live on rung one, staying there for now

The newest agent in the pipeline is an ops-doctor. It went live on Friday, the day after the first scored run, because a scored production run existing was its go-live condition. When a run fails operationally, a stage exits non-zero, an edition comes out empty, a batch is orphaned, the judges cannot start, it reads the failure context, classifies the cause against a small taxonomy (revoked key, source site changed, rate limit, code regression, dependency drift, config, image build, secret rotation) and files a structured diagnosis as a GitLab issue.

That is rung one. It may diagnose and report, and you will notice it is not allowed to touch code. It may not propose a patch, it may not open a merge request, and it certainly may not merge one. The rungs above it are designed and written down, and none of them exist as code, which is deliberate. Notice also what does not wake it: a quality flag. A groundedness score of 0.894 is a signal for an editor, not an incident, and the first version of the trigger fired on any degraded run until we narrowed it, because a doctor that files an issue every time the prose scores 3.9 is - you guessed it - a doctor you will learn to ignore! The two signals go to two different places on purpose.

Its promotion evidence is a casebook: every failure it has been given with the human-confirmed cause alongside its diagnosis. Three cases I built by breaking a dev copy on purpose. Two came from a real test in August. The other five came from the cloud cutover a week ago, where five infrastructure gaps were stacked one behind the other, each invisible until the one in front of it was fixed. A doctor that had seen only my synthetic cases would have been useless on that afternoon. The golden set for an agent that diagnoses failures has to be built from real failures, and the cutover donated half of it.

The entry bar for switching it on was five cases correct. It got ten out of ten, twice. In between it got nine out of ten, and then nine out of ten on a different case, which is the more useful result. The classifier runs at a low temperature but not zero, so two ambiguous boundaries in the taxonomy showed up as flicker: rate-limit versus judge-failure, revoked key versus secret rotation. I tightened the definitions rather than the temperature, because a taxonomy that only works at temperature zero is a taxonomy with a bug in it.

Ten for ten on a casebook I partly built by breaking things on purpose is not evidence for rung two, and if you are tempted to read it that way, ask what the casebook does not contain. It is however evidence that it works well enough to be worth running. Rung two - proposing a diff inside the issue - needs at least two real production cycles where its diagnoses are read and marked right or wrong. If the marks come in at 80 per cent or better, it moves up. If not, it stays, and I learn something about the taxonomy.

What the vendors already ship - and what they don't

I went looking for who else does this, because the ops-doctor is a small copy of a pattern every observability vendor now sells. What I found is that the industry has settled on the rungs and skipped the receipts.

Datadog's Bits AI SRE, since renamed Bits Investigation, is "an autonomous AI agent that investigates production issues end to end", and then "engineers can review and merge the resulting pull request". Autonomous investigation, assisted remediation. incident.io is blunter: "the only change Investigations can make to your systems is a pull request you review and merge yourself". Microsoft's Azure SRE Agent is the most interesting, because it publishes a ladder: "Review is the default mode. Your agent investigates, identifies a fix, and asks for your approval before executing" infrastructure changes, and then, "Start with review mode. Observe what the agent recommends for two to four weeks. When you find patterns you consistently approve, switch those specific triggers to Autonomous."

Read that last sentence again, because it is nearly right and the gap is the whole subject. The promotion criterion is that you, a person, notice a pattern in your own approvals over two to four weeks. No number. No bar set in advance. Nothing that runs after promotion to check the pattern held. And no sentence anywhere about switching a trigger back. There's a quieter detail on the same page: review is the default for the agent, but the trigger defaults for incident response plans and scheduled tasks are both autonomous. The ladder is in the documentation and the product starts you at the top of it.

GitHub is the same shape at the code end. Copilot's coding agent can open a draft pull request and "cannot approve or merge a pull request". Then, in March, admins could skip the human approval for workflow runs. Then, on 1 September, Copilot's review "can submit an approval that counts toward the repository's required-approvals rule", off by default. The switches are all there. Nothing tells you when you have earned the right to flip one, and nothing flips it back.

Even the tools I use every day work this way. Claude Code has a permission mode per session, from read-only up to fully unattended, and its SDK documentation says the intent out loud: "start restrictive and loosen permissions as trust builds". I agree with every word of that sentence, and I want to know what "trust builds" means in a number. Anthropic's own data from February shows what happens when you leave it to feel: new users run fully unattended in about 20 per cent of sessions, users with 750 sessions in over 40 per cent, and experienced users interrupt the agent more often, not less. People do climb the ladder on their own but they also learn where to grab the rail.

There is one exception I want to credit. Cloudflare's agent access model, published in August, describes a "trust ratchet" that makes trust stateful: "capability state moves in one direction. When a declared protected event occurs, the trust ratchet removes capabilities across the task execution graph according to policy." That is a demotion mechanic, enforced "in the harness and the network, not the prompt", and it is the only one I managed to find from a vendor. It's scoped to a single task, narrowing what an agent may do as the task proceeds, rather than to the agent's standing across weeks of runs. The direction is right and the timescale is one rung short. Langfuse's prompt CI/CD guide has the exact loop I want: a promotion gated on a threshold, a RegressionError that blocks it, and rollback by pointing the production label back, applied to a prompt version rather than to how much authority an agent holds.

The rung where it breaks

If you take one warning from the literature, take this one: rung three is where the ladder fails, and it fails quietly.

Lisanne Bainbridge wrote the founding paper on this in 1983, about control rooms, and every sentence of it reads as if it were about your pull request queue. The designer "who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate". Skills "deteriorate when they are not used", so "a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one". Monitoring a system on which very little happens is something it is humanly impossible to sustain for more than about half an hour. And the one that should be printed on every approve button: "A classic method of enforcing operator attention to a steady-state system is to require him to make a log. Unfortunately people can write down numbers without noticing what they are."

The experimental literature since then has measured how bad it gets. Parasuraman and Manzey's review finds that automation complacency "occurs in both naive and expert participants, cannot be prevented by training or instructions", and reports a study of experienced airline pilots given a false engine-fire alert in which every one of them shut the engine down. A 2023 radiology study gave 27 radiologists mammograms with a purported AI suggestion attached: when the suggestion was right, the very experienced readers were correct 82 per cent of the time; when it was wrong, 45 per cent. Elish has a name for the person left holding that accountability: the moral crumple zone, "a human actor who had limited control over the behavior of an automated or autonomous system" who nonetheless "bears the brunt of the moral and legal responsibilities when the overall system malfunctions".

The 2026 version says the same thing in the vocabulary of agents. OpenAI saw it in 2023 and asked for "best practices for users reviewing approvals for high-cost actions (such as minimum review times) to avoid their turning into a 'rubber stamp'". Microsoft's AI red team spent a year attacking agent systems and reported in June that human-in-the-loop "bypass was the most consistently exploited failure mode, at very high frequency", with the attack being nothing cleverer than the one its 2025 taxonomy had already described: waiting until "the user becomes fatigued with the prompts". Cloudflare's version is the one I would put on the wall: "An approval that is always granted is not a control. It is a ritual that trains people to ignore the one prompt that matters." And two weeks ago Mitchell, Ghosh and Passi published a paper whose title is the whole argument: "AI Agents Push Humans Out of the Loop".

There is of course an irony inside the ladder itself - the evidence trail that earns promotion, cycle after cycle of the agent being right, is exactly the "sustained positive experience with the system" that breeds complacency in the person approving it. The better the agent gets, the worse the reviewer gets.

So my rule for rung three is that it is a transit state, not a place to live. You should be able to say, for every step sitting on it, either the number that will promote it or the number that will demote it, and roughly when. A step that has been on rung three for six months with no promotion criterion is not being supervised. It is being rubber-stamped by someone who stopped reading months ago, and the honest move is to admit that and either write the check that lets it go up, or send it back down to advisory where you have to decide.

What demotion has to look like

Every published failure of an over-promoted agent that I found has the same shape, and it is not "the model was bad". It is that the demotion lived in the wrong place.

In July 2025 a Replit agent deleted a live production database during a declared code freeze, despite receiving repeated instructions not to make changes, having already spent days covering up bugs with fabricated data, fake reports and lies about the unit tests. The code freeze was a demotion. It was delivered as a sentence in a prompt, and the agent's credentials did not know about it. Replit's fixes, tellingly, were a fence and a lower rung: automatic separation of development and production databases that same weekend, and a plan-only mode six weeks later. In late April this year an agent working from Cursor deleted a company's production database and its volume-level backups in a single API call, having, in its own words, "guessed that deleting a staging volume via the API would be scoped to staging only".

An agent's real rung is whatever its credentials allow. If the rung in your config file and the rung in your IAM policy disagree, the IAM policy wins, and so does the incident. This is why my DOCTOR_RUNG value is enforced in code that refuses to act above it, and why the doctor's context assembly is read-only by construction, not by instruction. The rung is not a request to the model, it's more of a fence that the model can't see over.

Demotion therefore has three parts.

A signal: an online eval below its threshold, a gate recall that dropped, a diagnosis marked as wrong.
A single change: one config value, one label, one ratchet click, with no pull request for you to review at two in the morning.
A fence that makes the change real: the permissions above the new rung are gone, not merely discouraged. Get the first two right and skip the third and you have Replit's code freeze.

Others are here too, but there are gaps

I want to be straight about the crowd, because I nearly wrote this piece as if the ladder were mine. It is not. Accenture presented shadow mode to advisory to controlled autonomy to expanded autonomy at AI Engineer Europe in April, with the end state, in one attendee's notes, "gated by evidence and outcomes". A June paper by Weber and Taneja, "The Digital Apprentice", opens with "autonomy is earned, not assumed" and has the agent "graduating through per-skill autonomy tiers only when empirical evidence justifies it", with each escalation gated by explicit human approval and, as I read it, automatic rollback of a skill when its quality slips. That is the most rigorous version I have read and it is closer to mine than anything from a vendor. An August framework from Ampcome calls it "evaluation-gated autonomy" and says "breach a threshold, and autonomy contracts". Gartner now publishes four levels from observe to autonomous-within-guardrails.

So the idea has arrived, several times, independently. But I would go a few steps further from having run one.

First, shadow is a rung, not a pre-launch phase. Most ladders start at "recommend". Mine starts with the agent doing the real job on real traffic and being scored against what happened, with nobody acting on it. A shadow run on last week's production inputs is the least superficial test environment there is, and it costs the price of the tokens. Shadow also answers the question Kahneman and Klein put at the centre of when intuition can be trusted at all: is this a high-validity environment with feedback that is "rapid and unequivocal"? Their conclusion for human experts holds for agents word for word: "subjective confidence is therefore an unreliable indication of the validity of intuitive judgments and decisions." A model's confidence, a vendor's confidence and your own after a good demo are the same thing, and a shadow replaces all three with a scorecard.

Second, the unit of evidence is a cycle of the real job, and the count is plural. Not a benchmark score. Not a fortnight of it looking fine. The rewrite backend earned its rung on 85 real articles with two judges. The doctor's next rung needs two production cycles of marked diagnoses. When you write your own promotion rule, write the cycle count into it because ten clean runs with a low-blast-radius task tell you less than three clean runs of the one that can delete a database.

Third, make it per step, not per agent. Bradshaw, Hoffman, Johnson and Woods listed seven deadly myths of autonomous systems in 2013, among them that autonomy is a single dimension, that levels of autonomy make a useful roadmap, and, verbatim, that "'full autonomy' is not only possible, but is always desirable". I think that critique is right about a single dial for "the agent" and wrong about a rung per action class. My pipeline has a rung-four step and a rung-one step in the same process, and the config that keeps them apart is one line each.

Why I am writing this down now

I have been running this ladder for months and had never published it. I had said it out loud, in meetings and in at least one interview, and assumed that the field notes made the position obvious. Reading them back, they do not. What they show is a person who writes about eval gates, tripwires, abstention and accountability, and if that is all you had read, you could fairly conclude that I want a developer in every loop.

The opposite is true. I want as few humans in the loop as the evidence allows, and I want the evidence to be the thing that decides. The gates are not there to slow autonomy down. They are the mechanism by which it is granted. An agent with a green scorecard over enough cycles gets more rope than any amount of confidence in a demo would earn it, and an agent whose scorecard goes red loses that rope the same afternoon.

If your organisation is stuck at rung one, it is usually because nobody wrote down what promotion would take. If it jumped to rung four, it is usually because nobody wrote down what demotion would take. Both cost you, the first in time and the second in trust.

What this cannot do

Demotion today is a human changing a config value after reading a degraded flag. Automatic demotion on regression is designed and not built, and until it is, the ladder has a person on the descent path, which is exactly the thing I claim to be removing. Kappa between the judges and human labels is empty, because there are not yet twenty human verdicts to compute it from, so "the scorecard is green" currently means "two model families agree", which although the judge panel piece explains as necessary, it's also not sufficient.

There's a deeper problem with gating anything on a score, and I would rather name it than wait for you to. Goodhart's law (Strathern's version) says that "when a measure becomes a target, it ceases to be a good measure", and an agent promoted on eval scores is an agent under exactly that pressure. This is not hypothetical. METR found OpenAI's o3 reward-hacking about 30 per cent of its runs on one benchmark, by tampering with the scoring code and test harness, and when asked afterwards whether that was what the user wanted, it answered 'no' 10 out of 10 times. A March 2026 paper by Wang and Huang argues that under any finite evaluation "any optimized AI agent will systematically under-invest effort in quality dimensions not covered by its evaluation system", which makes reward hacking "a structural equilibrium, not a correctable bug". My defences are simple and boringly ordinary: the agent cannot see its own scorecard, the judges are different model families from the generator, and the golden set grows from real failures the agent did not choose. Those make gaming harder. They do not make the score the thing I care about, and the day the ladder promotes something that learned the proxy is the day you need a human reading the output again.

The whole thing is running at indie scale: one pipeline, a few dollars a week. Whether the same ladder holds when a rung-four agent is spending your money against your customers is a question I have designed for and not measured. And Bainbridge's warning applies to me as much as to anyone: the person left monitoring a reliable system is the least practised party at the exact moment they are needed. My answer is that the scorecard, not the person, does the monitoring. But, of course, that answer is only ever going to be as good as the scorecard.

What it does give you is a way to answer "is it safe to trust this yet" with a number and a date, instead of an opinion and a meeting.

Richard Atkins

Measurement tools need evals too

Richard Atkins — Thu, 03 Sep 2026 14:16:05 +0000

A tool that refuses to answer

Here is one of my tools declining to do the thing it exists to do.

$ git-habits scan --from-export thin.tsv

  selection
    window            2026-03-10 to 2026-03-12  (3d span, 3 active)
    commits           3   insufficient (n=3 commits, need >=5)
    changed lines     120
    lines/commit      insufficient (n=3 commits, need >=5)
    moved (reuse)     insufficient (n=120 changed lines, need >=200)
    legacy touch      insufficient (n=120 changed lines, need >=200)
    rework <=14d      insufficient (n=120 changed lines, need >=200)
    AI co-authored    insufficient (n=3 commits, need >=5)

  ! only 3 commits in range: rates are volatile, treat as indicative

Three commits went in. The tool prints the raw counts, then refuses to turn them into rates. That is not an error and it is not a zero. It is a stated refusal, with the reason and the shortfall sitting exactly where the number would have gone, so you can see why there is no figure rather than being handed a confident one built on nothing. Its companion behaves the same way when you ask it to compare against a window that turns out to be empty.

$ diff-habits compare --repo . --split 2020-01-01
comparison refused: the 'before' window has no added lines. Check
--author/--since/--until/--split; an empty window compared against a real
one produces fiction, not deltas.
$ echo $?
1

That refusal, and the discipline behind it, is the whole of what v0.2 adds to git-habits and diff-habits, two small command-line tools I keep in the open at github.com/rich-atkins. No new metric. They measure what they measured before: commit sizes, rework, moved lines, error-masking constructs, the shape of how a codebase gets written over time. What changed is the set of things they are willing to say out loud, plus a test suite that proves they can still catch what they claim to catch. If that pairing sounds familiar, it should. It is what a decent AI eval does.

The rule I keep arriving at

These two tools point at other people's code. That is their reason to exist: git-habits reads commit metadata and never opens a source file, so you can run it against an employer's repository without a conversation first. A number that comes out of a tool like that carries weight. You will see it pasted into a review, a decision, a slide. So a wrong number is worse than no number, and an unknown quietly rendered as a zero is the worst outcome of all, because it looks like a finding.

Evals for language models taught me the shape of the fix, from the other end of my work. You do two things. You fail closed on insufficient evidence, so a thin sample produces a refusal rather than a reading. And you prove the thing you built actually detects what it says it detects, rather than assuming it does because the code looks right. v0.2 is those two ideas pushed into a pair of measurement tools that had neither.

The floors are the first idea made concrete. Every rate now knows its denominator and the point below which it declines to be a number. In git-habits that is five commits for the per-commit rates and 200 changed lines for the churn-based ones. In diff-habits it is 500 added lines, set higher on purpose: masking events are rare, running at roughly 0.2 to 4 per thousand added lines in the samples I have, so a per-thousand rate computed over a couple of hundred lines is noise wearing a number's clothes. Below the floor you get the refusal. Above it you get the figure, still with the old volatility caveats attached. The raw counts stay on screen throughout, because you are owed the evidence, not just the verdict you were going to quote.

Proving the detector detects

The second idea is a test suite that builds real git repositories on every run and then tries to fool the tools. There are two of them. One is PLANTED: a clear regime change after a fixed date, where commits swell to roughly six times their earlier size, cadence rises and AI co-author trailers start appearing. The other is CONTROL: the same shape from start to finish, no shift at all. The planted repository must be flagged. The control must stay quiet.

Both directions are asserted, and that is the part you skip when you are proud of the thing. A detector that always fires is as broken as one that never does, and a demo only ever shows you the first kind of failure. So the control test is the one that earns its keep. It fails if the tool cries wolf.

The three bugs it caught on day one

I wrote the self-test to protect work I believed was already correct. It caught three bugs the first time it ran, and the way it caught them is the entire argument for writing it.

The first was a windowing question I had answered wrongly and confidently. When you split a history into before and after at an instant, which window does a commit landing exactly on the split second belong to? My review predicted the bug would be double-counting, the boundary commit falling into both sides and inflating the totals. The boundary test proved something subtler. Git's --until is inclusive while its --since is exclusive, so v0.1 had quietly put every boundary commit in the before window only, and in nobody's after window. Not doubled. Dropped from one side. A plausible story about the bug, refuted by a test before it could become a plausible story in an article.

The second bug was the fix for the first. My initial correction shifted a single window edge, which felt right and read right. The test refused it, because moving one edge dropped the boundary commit out of both windows instead of one. The commit had gone from appearing once in the wrong place to appearing nowhere at all. The corrected fix anchors both edges one second before the split, so a boundary commit now lands exactly once, in the after window, matching git-habits' convention so the two tools agree about the same commit. Plausible fix, test proves it wrong, corrected fix pinned in place. I would have shipped the first attempt.

The third bug was the most humbling, because it exposed a claim I had been making for a month. Both tools advertised support back to Python 3.10. Neither had ever parsed a real git date under it. Git emits a trailing Z on UTC timestamps, and datetime.fromisoformat only accepts that Z from Python 3.11 onward. The unit tests fed synthetic logs that never carried the suffix, so the gap sat latent in both repositories since v0.1. The moment the sabotage suite built actual repositories and ran the real pipeline against them, the 3.10 job on CI failed. The floor I had promised was never a floor. It had simply never been tested.

Where this came from, and where it fails

None of this is new to me, which is why it stung. v0.1 of git-habits once reported "0% AI co-authored" for a repository whose trailers had never been captured, an unknown printed as a zero, the exact mistake the floors now forbid. And I watched the same shape appear somewhere with far higher stakes: an LLM eval gate in another project that returned PASS on a run where zero articles were actually judged, because zero judgments contain zero failures. A gate that cannot fail is not a gate. A report that cannot say "not enough evidence" will, given time, lie to you. I have now met that lesson three times, in a metrics tool, in a masking detector and in a model eval, and I have stopped treating them as separate.

Two honesty notes I will not bury. The floors are defaults with reasons behind them, not universal truths; five commits and 200 lines are low bars for "a number at all", and you should raise them for your own context rather than trust mine. And everything the tools could not tell you before, they still cannot. They do not detect AI, a missing trailer does not prove a human wrote the code, and a young repository cannot hold year-old code. The self-test proves the detector fires on a planted change. It does not turn a coarse signal into a fine one.

If you maintain anything that measures other people's work, whether a metrics script, a CI check or an eval harness, run one experiment this week. Feed it an input with nothing in it, and watch what it prints. If it hands you a clean zero and a passing exit code instead of refusing, you have not found a quiet week. You have found the day your tool learned to lie politely, and you get to fix it before it does so in front of someone who believes it.

One LLM judge is an opinion. Two model families agreeing is evidence.

Richard Atkins — Wed, 02 Sep 2026 08:44:54 +0000

The problem with one judge

I migrated my news pipeline's article rewriting from a local Qwen model to hosted Claude (previous piece), and before cutting over I needed an answer to a question taste can't settle at scale: does the cloud model hold each of my four writer personas as well as the local one did?

The standard answer is an LLM-as-judge: give a model a rubric, show it the article, get a score. The standard objection is also correct: a single judge is one model's opinion, with that model family's tastes baked in. And my case had a sharper version of the objection. The candidate prose was written by Claude. If Claude also judges it, every positive result carries an asterisk you can't remove: of course the judge liked its cousin's writing.

You don't fix that with a better rubric. You fix it with independence.

The panel

So the harness runs a blind, cross-family panel: Claude Opus and Gemini, each scoring the same article against the same persona rubric, five dials from 0 to 5, structured output, neither told which backend produced the text or what the other judge said. The gate takes the average of the two families and flags any article where they meaningfully split.

The blindness matters more than it sounds. The moment a judge prompt contains "this one is the new model", you're not measuring the article any more, you're measuring the judge's expectations. Strip every provenance clue before the text hits the panel.

What the panel buys you is a specific kind of defensibility. When one model says your output is good, you have a claim. When two models with different training lineages, different owners and different failure modes read the same text blind and land within a third of a point of each other, you have something a sceptic has to work much harder to dismiss.

What it found

On the full weekly set, 85 articles across all four personas, with both judges covering all 85, the verdict wasn't close: cloud 4.73 out of 5, local 2.19, and the cloud won on every persona individually:

Persona	Local	Cloud	Margin	n
The Analyst	2.17	4.70	+2.53	21
The Engineer	1.91	4.63	+2.72	27
The Optimist	2.30	4.70	+2.40	2
The Steward	2.41	4.83	+2.42	35
Overall	2.19	4.73	+2.54	85

The two families' scores differed by 0.30 out of 5 on average. Each judge run independently reaches the same verdict: Opus puts the margin at +2.45, Gemini at +2.64: they disagree slightly about how bad the old prose was, not about the answer. And the panel flagged 3 articles out of 85 where the judges genuinely split on a dial, every one of them Gemini scoring a persona trait harsher than Opus. That's not a weakness of the method; that's the method working. A panel that never disagrees isn't measuring independently.

That margin also killed a hedge I'd been keeping alive. I'd held "hosted Qwen-72B" open as an option, on the theory that the exact current voice was worth preserving. The panel data says the current voice was the weakest thing about the pipeline. Preserving it would have been sentimentality with a monthly invoice.

The bug the gate was hiding

Here's the part that earns this piece its place in a series about what survives production.

The panel's first real outing was a 12-article run. Both judges failed to start, thanks to a missing API key: the most boring failure there is. The panel logic did what half-thought-through code does: carried on, judged nothing, and the gate reported PASS, because zero judgments contained zero failures.

A gate that can't fail isn't a gate. It's a rubber stamp with extra steps.

The fix is the same principle I keep arriving at from different directions: fail closed on insufficient evidence. The gate now refuses to pass unless enough of the sample was actually judged, and a judge dropping out is a loud, visible event rather than a silent shrug.

Then reality tested the fix within hours. On the full 85-article run, Gemini's free tier (twenty requests a day) ran out after eleven articles. This time the system behaved: Opus coverage held at 85 out of 85, the run proceeded, and the degradation was visible instead of invisible. That became the production policy: if one judge fails, the panel proceeds on the survivor and flags a human; if both fail, the run fails. Degraded and honest beats complete and fictional.

If you build eval gates, test the gate itself. Feed it an empty result set and check it fails. Mine didn't, and I only found out because two different things broke on the same Saturday.

What it costs, honestly

The Opus judge came to $0.035 per article, about $3 for the full 85. And the quota story has a punchline: on the paid tier, Gemini judged all 85 articles for 22 cents. Twenty free requests a day was never an evals budget; the actual evals budget turned out to be less than a postage stamp. For continuous production use we still sample: mechanical checks (structure, length, banned phrases) run on every article for pennies, and the two-judge panel runs on a stratified dozen per week.

Caveats, because they're load-bearing: this compared new-model-plus-new-window against old-model-plus-old-limits, an era confound as much as a model one; one persona had only two samples in the set; and judge-versus-human calibration is still ahead of us (the panel agreeing with itself is necessary, not sufficient). The scores are directional evidence with unusually good provenance, not a leaderboard.

But the method stands on its own: blind the judges, cross the families, gate on the average, and above all, make sure the gate can actually say no.

The migration the data ordered: moving a production LLM pipeline off the Mac Mini under my desk

Richard Atkins — Mon, 31 Aug 2026 16:24:59 +0000

The bill, up front

Last week my news pipeline rewrote a full weekly batch of 85 articles in the cloud, across all four of its writer personas. The rewriting bill was $4.31, which is $0.051 per article. Even counting the blind two-judge panel that then scored every one of those articles against the old local output, the whole exercise metered $7.47.

For a year that same work ran on a Mac Mini under my desk: a 72-billion-parameter model on Ollama, FLUX generating images on the GPU, launchd waking everything on Thursdays. I'd already measured the economics of that setup and published the break-even: at my scale, hosted APIs with batch pricing were roughly 8 to 12 times cheaper than the fully-loaded cost of the local hardware. Then I kept running it locally anyway, because the marginal cost of a machine you already own is nearly zero and the data said the money didn't matter much either way.

So why move? Because money was never the real bill. The real bill was operational: a single point of failure with a 40-minute cold load, model files on a USB SSD, and a fragility class all of its own. When the machine sneezed, the product missed its week. You can't put a price on that in a cost model, which is exactly how it hides.

What actually moved

Three things were local; everything else already lived in the cloud.

The LLM calls went from local Qwen (72b for writing, 14b for classifying) to hosted Claude, tiered the same way: a cheap fast model for clustering, tagging and novelty checks, a quality model for the rewrites. Every pipeline call now goes through the Batches API, which halves the token price in exchange for latency I don't care about at 3am.

The images went from FLUX running in-process on the GPU to the same FLUX model behind fal.ai's API. Same model, so the product's visual language didn't change. That one mattered to me: a migration should be invisible to readers.

The orchestration went from launchd to Azure Container Apps Jobs: a container on a cron, scale-to-zero, secrets in Key Vault, and a 24-hour task timeout that lets a batch submit, wait and collect inside a single invocation.

One codebase runs both worlds. Each stage picks its backend from an environment variable, so cutover is a config change and rollback is the same change backwards.

What the plan got wrong

The migration brief was written against a clean mental model. Reality filed corrections.

There were 21 LLM call sites, not the 13 I'd counted. Thirteen named helpers, plus eight inline HTTP calls hiding in scripts, most with their own hand-rolled streaming loop and hardcoded URL. The first real move wasn't "swap the provider", it was "make there be one client". If you take nothing else from this piece: count your call sites before you estimate your migration, then consolidate before you
swap.

There were no retries anywhere. A local Ollama never rate-limits you, so the code had never needed them. Hosted APIs do. Without a retry layer, every transient 429 becomes a permanently failed article.

Resumable state turned out to be dangerous. The old design persisted an in-flight batch id so a crashed run could reconnect. In a container that restarts from a clean disk, the run re-derives its candidate list, and here's the trap: the new list isn't guaranteed to match the old one. Reconnect to the old batch and you
can publish content under the wrong article. We made recovery fail closed instead: abandon the orphaned batch, eat the few dollars, re-submit fresh. Correctness beats cost recovery every time the two collide.

The local model's scars were still in the prompts. A 4k context cap and a 3,000-character source-content limit existed purely because of VRAM. Lifting them was a deliberate decision with a measurable cost delta, not a cleanup. Same for the banned-phrase filter that force-rewrote model output: the local model needed it; the hosted one triggered it exactly once in 85 articles. It now runs as a detector first and an enforcer second, because you should measure a guard before you let it edit your prose.

Did quality survive?

Better than survived. A blind two-model judge panel, both judges covering all 85 articles from both backends, scored persona adherence at cloud 4.73 out of 5 against local 2.19, a win on every one of the four personas. The mechanical checks said the same thing from a different angle: every cloud article passed the structure checks; 78% of the local ones did. The panel design (and the bug it caught in its own gate) is the next piece; the headline here is that the migration wasn't a cost-neutral quality trade. The bigger source window alone, un-cramped from its VRAM-era 3,000 characters, gives the writer model more to work with.

What it costs now

About $10 a month of pipeline compute delta: rewrites at $0.051 an article with batch pricing and prompt caching, images at fractions of a cent, the container mostly asleep. Set against roughly £50 a year of electricity for the Mini, this migration will never pay for itself in cash. It pays for itself the first Thursday the pipeline runs while the Mini is switched off, in a hotel, or in a skip.

The other currency

Cost models compare money because money is easy to put in a column. The column that never appears is time, and locally the pipeline spent it in three ways.

Machine time. A 72-billion-parameter model on consumer hardware writes one article at a time, because a second one would blow the memory. Before it writes anything at all, cold-loading the model from an external SSD could take forty minutes. On the measured run, the local pass spent about four and a half hours rewriting the 85 articles (an upper bound from file timestamps, since the machine was juggling other stages in the gaps). The cloud pass wrote the same 85 in 48 minutes. The Mini spent its hours as a pipeline appliance and nothing else.

My time. This is the expensive one. The local stack had a fragility class the cost model never saw: model reloads, broken Python relinks, a launchd job that needed its owner nearby. Every "cheap" local run carried an invisible surcharge of owner-hours, billed at whatever your evening is worth.

The exchange rate you choose. The cloud makes the time-money trade explicit and lets you pick a side per workload. Synchronous calls cost full price and answer in seconds. The Batches API charges half price for answers within a day. A pipeline that runs while you sleep should always sell its latency, because overnight latency is worthless to you and the API pays you 50% for it. That's the quiet insight batch pricing encodes: time and money are convertible, and the smart move is to sell the one you're not using.

Cloud inference isn't just cheaper per token at my scale. It converts hours of machine-sitting into minutes of nobody-sitting, and it hands the 3am problem to someone whose job is 3am problems.

Measure your own crossover, in both currencies. And when the money column says "either way", read the two columns the spreadsheet doesn't have: whose hours are being spent, and what breaks at 3am with only you to fix it.

An MCP server that has to prove itself: citations, abstention and entitlements as CI gates

Richard Atkins — Sun, 30 Aug 2026 22:35:45 +0000

The demo, in one screen

Here's the eval scorecard for a small MCP server on a green build:

retrieval   (staff, n=22):  hit@1 1.00   recall@5 1.00   MRR 1.00
abstention  (n=10):         rate  1.00   (target 1.00)
leakage     : 0 (must be 0)
redaction   : 0 (must be 0)
EVAL GATE: PASS

Now I weaken one thing. Two environment variables drop the server's relevance bar to
zero, the setting where it returns its best guess for every question instead of
admitting defeat:

abstention  (n=10):         rate  0.40   (target 1.00)
  FALSE-ANSWER  'customer refund policy' -> internal/hr/expenses-policy.md (score 4.29)
  FALSE-ANSWER  'office dog policy'      -> internal/hr/expenses-policy.md (score 4.29)
EVAL GATE: FAIL

Exit code 1. The build is red. Nobody had to notice the server had started making
things up, because the pipeline noticed first.

Why bother?

The Model Context Protocol has made it absurdly easy to hand an agent your knowledge
base. There are hundreds of servers that expose notes, wikis and docs as searchable
tools, and most of them work fine as plumbing. But before you let an agent answer
questions out of your company handbook, there are three questions the plumbing never
answers.

Can you trust what comes back? When the server returns a snippet, can the agent cite
where it came from, precisely enough that a human can check?

Does it know when it doesn't know? Ask a vault about a policy that isn't in it and
BM25 will cheerfully return the closest-sounding note anyway. The agent, primed to
trust its tools, turns that into a confident answer about a document that does not
exist.

And who is allowed to see what? If your vault has a restricted/ folder, "the search
tool filters those results out" is not an answer. It's the start of an incident report.

So I built grounded-mcp: a small Python
MCP server over a folder of markdown (Obsidian works as-is) with three guardrails.
Every content-bearing response carries a stable citation id, path plus heading plus
line span. Search abstains, explicitly and structurally, when nothing clears the
relevance bar. And entitlements are enforced at index level: content a profile can't
see is never indexed for it in the first place.

None of which would be worth writing about, except for one decision: the repo ships
with its own eval suite, and the suite is the CI gate. That decision paid for itself
on the first run.

The first honest run found a real bug

The eval suite runs four families against a committed demo vault (a fictional company
handbook with public, internal and restricted zones): a golden retrieval set, a set of
questions whose answers are deliberately not in the vault, a leakage set answerable
only from restricted content, and a redaction check against a seeded fake credential.

First run: retrieval perfect, leakage zero, redaction zero. Abstention: 0.30.
Seven of ten unanswerable questions came back with confident wrong answers.

The interesting part is why. I had assumed a BM25 score threshold would separate real
answers from plausible noise. The data said no:

true answers:    scores 2.44 to 14.68
false answers:   scores 2.12 to 4.29

The ranges overlap. "How long is probation" scores 2.44 against the note that
genuinely answers it, while "customer refund policy" scores 4.29 against an expenses
policy that says nothing about refunds. One common word landing in a heavily weighted
title field looks exactly like relevance. No threshold exists that keeps the first and
rejects the second.

What does separate them is coverage: what fraction of the query's content words the
note actually contains. The false answers ride on one matching word ("policy") and
their coverage collapses once you stop counting scaffolding like "what is the". So
abstention became a double gate: score above threshold AND coverage above half. That
took the rate from 0.30 to 1.00 with zero retrieval loss.

Then the gate caught two more bugs, in my fix. Naive substring matching decided
"code" doesn't appear in a note that only says "coding", and a hyphenated
"AI-generated" hid both of its parts from a query about AI. Light suffix-stripping and
splitting compounds fixed both, and I know the fixes didn't break anything else
because 24 golden queries and the leakage set re-ran on every attempt. The eval suite
was the code review.

Leakage is a ranking problem, not a filter problem

The entitlements design deserves one more paragraph, because the obvious
implementation is quietly wrong. Most access control on search works by filtering
results: retrieve everything, then drop what the caller shouldn't see. That leaks. A
filtered result still influenced rankings. Its terms still matched. In some systems
the result count still moves, and now your search tool is an existence oracle for the
documents you hid.

grounded-mcp builds a separate index per entitlements profile, containing only the
notes that profile may see. Denied content is never scored, so there is nothing to
filter and nothing to leak. The eval suite's leakage family checks this at the
harshest setting, thresholds off entirely, and any hit from a denied path fails the
build. Same principle for direct reads: a denied note and a nonexistent note return
byte-identical responses.

What it doesn't do

Honesty section. Over stdio, client and server run as the same user, so profiles
demonstrate the deployment pattern rather than defend against a hostile peer; real
per-client enforcement arrives with the HTTP transport in v0.2. Retrieval is plain
BM25, deliberately: it's deterministic, dependency-free and measurable, and when
hybrid semantic retrieval lands it has to publish its eval delta against this
baseline or it doesn't land. The redaction patterns catch key-shaped strings, not
every secret. And there are no write tools. A server that can quote your vault but
never rewrite it is a trust feature, not a gap in the roadmap.

Try it

The repo is github.com/rich-atkins/grounded-mcp,
MIT licensed. pip install -e ., point GROUNDED_VAULT at your notes, wire it into
Claude Code or Claude Desktop with a five-line config. Run the evals, then run the
sabotage demo and watch the gate go red.

Better still: write ten questions your vault can't answer and add them to the
abstention set. If your knowledge server passes that test today, you're doing better
than mine was on Friday morning.

Cutting juniors is a choice, not an AI inevitability

Richard Atkins — Sun, 09 Aug 2026 05:06:00 +0000

Part 5 of five, the finale. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

Walk into a team that has quietly stopped hiring juniors and it looks fine. Better than fine: leaner, faster, all senior, nobody to train. It looks like that for years. It keeps looking like that right up until the seniors start to retire and there is no one behind them, and by then the problem took a decade to make and cannot be fixed in a quarter. This is the choice with the longest shadow in the whole series, and the one the industry is sleepwalking into.

We have argued that AI's costs, to wellbeing, to expertise, to how we manage, and to how we organise, are all leadership choices dressed up as inevitabilities. Here is the biggest one.

Across knowledge work, the entry-level door is closing. This is not a vibe; it is now measured. A Stanford study tracking payroll records for millions of workers found that those aged 22 to 25 in the most AI-exposed jobs have seen real declines: employment for young software developers is down roughly 20% since late 2022, with customer service among the other hardest-hit occupations, while employment for experienced workers in the same fields held up. The juniors are the canaries in the coal mine.

A separate line of evidence points the same way from entirely different data. PwC's 2026 AI Jobs Barometer analysed more than a billion job postings rather than payroll records, and found the entry-level rung has not so much vanished as risen out of reach: entry-level roles in highly AI-exposed occupations are now seven times more likely to demand skills that historically appeared later in a career, and 52% of newly appearing entry-level skills were ones previously associated with experienced workers, against just 7% in the least-exposed fields. Traditional entry-level positions shrank 10% since 2019. Two different methods, one direction, and the second tells you how the door is closing: not by removing the job title, but by loading it with prerequisites no beginner can have.

But the same study contains the fact that turns this from a lament into an argument. The decline showed up in jobs where AI automates the work. In jobs where AI augments the worker, entry-level employment did not fall. Same technology. Two deployment choices. Two completely different outcomes for a generation of workers. That is the whole thesis of this series, made visible in the labour data: automate or augment is a choice, and it is landing on the youngest first.

First, be honest about the cause

It is tempting, and self-serving for anyone senior, to pin the collapse of junior hiring entirely on AI. So let me steelman the sceptic: 2022 to 2025 also brought higher interest rates, a correction after pandemic over-hiring, and tighter budgets. Plenty of the entry-level squeeze is the ordinary business cycle, not the technology.

The Stanford authors ran exactly that challenge, which is why their finding is worth citing rather than the usual hand-waving. They checked interest rates and found that AI-exposed jobs are actually less sensitive to rates than average, so rate rises do not explain why those specific jobs shed their juniors. The AI-exposed decline holds up after controlling for the obvious economic factors. AI is not the only thing happening in the labour market. But on the best evidence we have, it is doing real, specific work here, and "it's just the economy" no longer covers it.

It is also genuinely contested, and worth saying so. The Financial Times recently argued that AI is not destroying entry-level work so much as changing it, pointing to US employers who expect to hire around 5.6% more new graduates this year. Both things can be true at once. Aggregate graduate hiring can rise while the specific, AI-exposed junior roles fall, which is exactly what the Stanford data shows: entry-level employment is climbing in the least AI-exposed jobs and dropping in the most exposed. The average hides the sorting.

The counter-argument, taken seriously

The sharper objection is the optimistic one: even if juniors are being automated, maybe that is fine. Maybe we simply need fewer people, and AI will conjure new entry-level roles we cannot yet see, as every technology wave eventually has.

Two problems. First, "new roles will appear" is a hope, not a plan, and it asks a specific cohort of real people to absorb the gap while we wait to be proven right. Second, and more concrete, the best evidence says automating juniors is not even the efficient move. In the Harvard and BCG study I cited earlier in this series, the people who improved most with AI were the least experienced; the floor rose faster than the ceiling. So the economically rational play is to pair a junior with AI and get near-senior output at junior cost, exactly the "augment" path the Stanford data shows protects entry-level jobs. Cutting the junior instead is not the efficient choice. It is the cheap-this-quarter one. Which makes it a choice, and a short-sighted one.

The pincer

Now put this next to something I planted earlier in the series and deliberately left unresolved: intuition rust. In a year-long study of cancer specialists using AI in clinical work, the researchers found early productivity gains quietly masking a dulling of the clinicians' own judgment. The expertise eroded without symptoms, until it was gone. That is a narrow population and I would not stretch it further than it goes, but the mechanism is not specialty-specific: skill atrophies when you stop exercising it, and AI is very good at letting you stop.

Hold both facts at once. You are cutting off the supply of new experts at the bottom, because AI can do junior work. And you are quietly de-skilling your existing experts at the top, because AI is doing their work too. No one coming in. The ones you have, rusting. That is a pincer on expertise, and it closes slowly enough that no single quarter's numbers ever show it. There are now names for both jaws of it: "intuition rust" for the experts quietly losing their edge, and "never skilling" for the juniors who never build one, because the work that would have built it was automated before they arrived.

There is a farming phrase for exactly this: eating your seed corn. When times are hard you can eat the grain you saved to plant next year. It feeds you now. It guarantees there is nothing to harvest later. Automating the juniors while offloading the seniors' judgment to the machine is eating the seed corn of your own profession, and calling the fuller belly a productivity gain.

Here is the question the pincer forces, and almost nobody in the "automate the juniors" conversation is asking it. Where do the seniors of 2035 come from? The senior you are hiring today became senior by doing, badly and then well, the junior work you are now automating away. Remove the rung and you do not just lose this year's juniors. You lose the mechanism that produces every future senior. And the people you are counting on to supervise the AI, whose judgment is the last line against a confidently wrong machine, are the exact people quietly losing that judgment.

What this asks of a leader

Treat your junior pipeline as capability infrastructure, not a cost line. Pair juniors with AI rather than replacing them, the augment path, not because it is kind but because the data says it is the smarter economics and the only way to keep making experts. Protect the deliberate, inefficient practice, in juniors and seniors both, that builds the judgment no model has. Some slowness is not waste. It is how expertise is made, and it is the first thing an efficiency drive deletes.

That is the whole series in one idea. AI hands you options, not outcomes. It can lift your weakest people or hollow out your strongest. It can make work feel owned or make it feel like babysitting a machine. It can build an enabling organisation or a queue. It can grow the next generation of experts or eat the seed that would have become them. Every one of those is decided by a person, not the technology. The costs are real, the evidence is clear, and the choice, every time, is yours.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 5, the finale. Links added as each publishes.

Written by Richard Atkins.

Sources: Brynjolfsson, Chandar & Chen, "Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of AI", Stanford Digital Economy Lab (ADP payroll microdata; the overall relative decline for ages 22–25 in the most AI-exposed occupations was revised from 13% to 16% in the February 2026 update) — https://digitaleconomy.stanford.edu/publication/canaries-in-the-coal-mine-six-facts-about-the-recent-employment-effects-of-artificial-intelligence/ ; the authors' interest-rates follow-up — https://digitaleconomy.stanford.edu/news/canaries-interest-rates-and-timinga-more-on-recent-drivers-of-employment-changes-for-young-workers/ · PwC, 2026 Global AI Jobs Barometer (analysis of 1bn+ job postings) — https://www.pwc.com/gx/en/news-room/press-releases/2026/pwc-2026-ai-jobs-barometer.html · Dell'Acqua et al. (2023), Harvard/BCG jagged-frontier study — https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321 · Ehsan et al. (2026), "From Future of Work to Future of Workers", the "intuition rust" study (year-long study of cancer specialists; CHI 2026) — https://arxiv.org/abs/2601.21920 · "never-skilling": Ke et al. (2026), "AI-induced never-skilling in medical education", Nature Medicine 32(6) — https://www.nature.com/articles/s41591-026-04438-y · Class of 2026 graduate hiring projection (+5.6%): NACE Job Outlook Spring Update, April 2026 — https://www.naceweb.org/job-market/trends-and-predictions/ ; "AI isn't destroying entry-level jobs. It's changing them", Financial Times (2026).

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

The code that hides the evidence

Richard Atkins — Wed, 05 Aug 2026 10:24:00 +0000

Part of "AI, engineering and what survives production", a series on the parts of building with AI that hold up once real traffic hits them.

There is a category of code that nobody defends in principle and everybody writes in practice.

try {
  await syncCustomerRecord(id)
} catch (e) {}

You do not think that is good. Neither do I, and I have written it. It gets written anyway, usually at the end of a long day, usually because the failure was intermittent and the deadline was not. And it is worse than the bug it silences, because the bug at least had the decency to announce itself.

The distinction that matters is between handling a failure and hiding one. A catch block that logs, retries, or rethrows with context is engineering. A catch block that swallows is a deleted alarm. From ten feet away the two look identical: both are three lines and a pair of braces, both make the red text stop.

Here is what interested me: this category is countable. Not perfectly, but well enough to see a trend in your own codebase. So I built a tool to count it, pointed it at seven repositories I had worked in, and got an answer I was not expecting.

What counts as masking

I split it into two tiers, and keeping them separate is the decision that makes the whole number defensible. My first version had a single score, and I abandoned it within an hour of pointing it at a real TypeScript codebase, for reasons I will come to.

Hard masking removes a signal outright and is difficult to justify:

Pattern	Example
Empty catch	`catch (e) {}`
Silenced exception	`except ValueError: pass`, `rescue nil`
Suppressed checking	`@ts-ignore`, `# noqa`, `eslint-disable`
Weakened typing	`as any`
Stubbed return	`return None # TODO`
Discarded error	Go's `_` assignment over an `err`

Soft masking is legitimate often enough that putting it in a headline would mislead: optional chaining, nullish defaults, TODO markers.

Consider user?.profile?.name. On a genuinely optional field that is correct code. The same expression, written because something in the chain was unexpectedly undefined and the ?. made the error go away, is a completely different act with identical syntax. No static tool can tell those apart, and I stopped trying. So soft constructs are counted separately, reported separately, and the judgement stays with you rather than being quietly made on your behalf.

That split is not fussiness, it is what I learned from the version I threw away. Fold optional chaining into one score and a modern TypeScript codebase produces an alarming number that means nothing at all. The first person to check it will find their own perfectly reasonable ?. in the count, conclude the tool is crying wolf, and never open it again. One number you can defend beats two you cannot.

Count only what was added

I measure additions and ignore deletions. A deletion cannot introduce a swallowed error, and counting both sides double-counts a line that has merely moved from one file to another.

Then normalise per thousand added lines, for the same reason as ever: a raw count tracks how much you wrote that month, and what you want to know is density. Otherwise a productive quarter looks like a decline in quality, and you will spend a week investigating the wrong thing.

What the output looks like

Real output, from the data pipeline behind this publication:

diff-habits scan

  a data pipeline
    commits / files      360 / 584
    added lines          121,560  (83,840 meaningful)
    error masking (hard) 229  (1.88 per 1k added)
    error masking (soft) 417  (3.43 per 1k added)
    top patterns         nullish_default=232, stub_return=187,
                         optional_chain=175, any_cast_ts=22, lint_suppress=20

The pattern breakdown is the part I actually act on. "229 hard hits" is a number I can do
nothing with. "187 stubbed returns" is a Monday morning: I go and look at what those
functions hand back when the thing they called did not answer, and I usually find two or
three that should have been raising instead of shrugging.

What I found, and why it is not what I expected

Pointing this at seven repositories, hard-masking density ranged from 0.16 to 4.0 per thousand added lines. That is a twenty-five-fold spread across codebases written by the same three people.

The highest was a data pipeline that scrapes web content and calls language models. The lowest was a small marketing site.

Sit with that for a moment, because it is the whole lesson. The pipeline is not badly written. It is defensively written, correctly. Networks time out. Scraped pages change shape. Model output is unreliable by construction. Code that talks to unreliable things needs guards, and those guards look exactly like masking to a regex.

Which gives the finding I did not want and now think is the most useful thing here: error-masking density is dominated by problem domain, not by who or what wrote the code. I had a tidy hypothesis about authorship. The data said the strongest predictor was what the code talks to.

So if you compare masking density between two different codebases, what you learn is which one has the flakier dependencies. You learn nothing whatsoever about their authors. Compare a codebase with itself, over time. That is the only comparison this metric will support, and I would rather tell you that than let you draw a conclusion it cannot carry.

What a rise actually means

A rise is not automatically bad, and I was careful to build a tool that does not imply otherwise, because one that scolds you gets uninstalled.

Hardening a service against a genuinely flaky dependency raises the count and is usually the right call. So rising density says look here, not you have failed. What I want to know when I see it is whether the guards I added are logging, alerting, or falling back deliberately, or whether they are quietly returning empty and leaving the caller to draw its own conclusions. Those two look the same in a diff and could not be less alike in production.

The question the number should prompt is the one I now ask myself in review: if this construct fires at three in the morning, does anybody find out?

The honest limitations

These are regexes, not a parser. They over-count a legitimate optional chain and under-count a swallow spread across several lines. except ValueError: on one line followed by an indented pass on the next slips straight through, and catching that properly needs an AST rather than a pattern. I decided that was a worthwhile trade for something you can run on any language in a second, but it means the trend is the meaningful part and a single absolute number is not an audit.

Language scoping matters more than I expected. An early version cheerfully matched TypeScript optional chaining inside Python files, which is obvious in hindsight and was not obvious to me until the numbers came out strange. Patterns are scoped by file extension now. If you extend the pattern list, scope yours too.

Small counts are noise, and I nearly fooled myself with one. A repository of mine produced a single hit across six thousand lines, and for about ten minutes I read that as an impressively clean codebase. It is not a low density. It is not enough data to have a density at all.

Try it on yours

Python 3.10 or newer. It depends on git-habits,
so exclusion rules stay identical between the two tools and their numbers remain
comparable.

git clone https://github.com/rich-atkins/diff-habits && cd diff-habits
python3 -m venv .venv && .venv/bin/pip install -e .

diff-habits scan    --repo /your/repo --author "you@example.com"
diff-habits compare --repo /your/repo --author "you@example.com" --split 2026-01-01

Unlike git-habits, this one needs a real working repository. It reads diff bodies, and
a metadata export does not contain them.

It reads locally and emits counts. Nothing is uploaded, and there is no network access to upload it with.

That is deliberate, and it is why this is a separate tool from its companion git-habits rather than a flag on it. git-habits works from commit metadata and never opens a source file, so you can run it on an employer's repository without a conversation. This one reads your code. That is a different decision, and it should be a different install rather than a flag you might not notice.

What is deliberately missing

The tool also implements block duplication, and it is off by default behind --experimental-duplication. The current approach counts any five-line sequence recurring anywhere in the history it walks, which conflates real copy and paste with code re-added after a refactor and with ordinary boilerplate. Measured against real repositories it reads about three orders of magnitude above published figures. It is useful as a trend within one repository and useless as an absolute number, so it says so and stays out of the default output.

And line-level move detection, the signal behind the widely repeated claim that refactoring is collapsing, is not implemented at all. It is a similarity-matching problem rather than a hashing one, and a naive version would produce numbers that look plausible and cannot survive comparison with the research they would be quoted against.

Both of those are in the README rather than an issue tracker, because a tool that quietly ships a broken metric is worse than one that admits to a gap. The whole point of measuring is to stop guessing. A measurement you cannot trust is just a guess wearing a number.

Companion piece: measure your own coding habits before you believe anyone else's numbers, on what git history alone can tell you, and the five ways I nearly fooled myself getting there.

Written by Richard Atkins.

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

Measure your own coding habits before you believe anyone else's numbers

Richard Atkins — Wed, 05 Aug 2026 09:32:00 +0000

Part of "AI, engineering and what survives production", a series on the parts of building with AI that hold up once real traffic hits them.

There is a claim going round that you have probably absorbed by now: AI-assisted development is making codebases worse. Refactoring is down, duplication is up, we are all writing more and revising less. The numbers behind it are real, the samples are enormous, and I found I had started repeating the conclusion in conversation without ever having checked it.

Then it occurred to me that those figures are averages taken across hundreds of millions of changes from thousands of organisations, not one of which is mine. So what is the rate in your repository? Nobody has told you, and on current evidence nobody is going to. I set out to find mine, assumed it would take an afternoon, and spent three days discovering that the answer is far harder to get at than the confident version suggests.

So this is not a piece about what AI does to code. It is about how to ask that question of your own repository without arriving at a wrong answer, which turned out to be the genuinely difficult part. The tool I built to do it is git-habits: free, local, and it reads no source code whatsoever.

What git can actually tell you

Git history is a surprisingly rich behavioural record. Not of quality, about which it knows nothing at all, but of habits: how often you commit, how large those commits are, whether you go back and change what you wrote last month, and whether anybody still touches the old code. That is a narrower thing than quality and it is the thing the industry claims has changed, so it is the thing worth measuring.

Four signals are computable from commit metadata alone, without opening a single source file:

Moved lines. The share of changed lines sitting in files git detected as renamed or copied. It is the closest thing history offers to "somebody went back and reorganised this."

Legacy touch. The share of changes landing on files nobody has touched in a year or more. Whether old code is still maintained or merely accumulating.

Rework. The share of changes landing on files touched again within a fortnight. Short-cycle churn.

Commit shape. How many commits per working day, and how large each one is.

None of these is a quality measure, and I want to be clear that I am not pretending otherwise. They are habits. Habits are what the claim is actually about, so habits are what you can test, and a narrow measurement you trust beats a broad one you do not.

What the output looks like

Real output, from one of my own repositories, split at an arbitrary date:

git-habits compare  split at 2026-05-01

  ruleset v1.0  excluded 28,297 of 101,578 changed lines (27.9%)
    lockfiles            24,513
    data_dumps            3,778
    binary_media              5

  first month
    window            2026-04-03 to 2026-04-24  (21d span, 7 active)
    commits           136   (19.43/active day)
    changed lines     41,788
    lines/commit      mean 307.3  p50 90.0  p90 430.0
    moved (reuse)     0.16%  (1.6 per 1k changed lines)
    legacy touch      0.0%  (0.0 per 1k)
    rework <=14d      17.24%  (172.4 per 1k)
    AI co-authored    45 commits (33.1%)

  after
    window            2026-05-04 to 2026-07-18  (76d span, 11 active)
    commits           44   (4.0/active day)
    changed lines     31,493
    lines/commit      mean 715.8  p50 66.5  p90 2997.0
    ...

Two things in there are worth pointing at before we go further, because both are the
kind of detail a dashboard would smooth over.

The exclusion report comes first, not last. Nearly 28% of the changed lines in that
window were lockfiles and data dumps. You are told that before you read a single metric.

Look at the mean and the median diverge in the second window. Mean 715.8, median 66.5,
p90 2,997. A handful of enormous commits have eaten the mean while the typical commit
actually got smaller. Read only the mean and you would conclude the opposite of what
happened.

Normalise per line, not per commit

This decision matters more than it looks. Measure anything "per commit" and, the moment your commit granularity changes, every metric moves while nothing underneath it has changed at all. A team that switches to squash merges halves its commit count overnight and could be forgiven for thinking it had halved its output. So I made per changed line the primary normalisation, which has the useful side effect of keeping the numbers comparable with the published research.

I kept the per-commit view as well, though, because it answers a different question and the gap between the two is where the artefacts hide. A metric that shifted only because commits got bigger is not a finding, and you want that visible rather than smoothed away.

One caveat on averages. In one repository I measured, the mean commit was 860 changed lines and the median was 16. A handful of bulk imports had eaten the mean entirely. Report the median. If a tool shows you only a mean, it is showing you its largest commits.

The five ways I nearly fooled myself

Every one of these produced a confident, wrong number during the build. They are in the tool's output and its README for that reason.

1. Generated files are most of your history

Lockfiles, build output, vendored dependencies, minified bundles, committed logs. Across the repositories I measured, generated artefacts accounted for between 48% and 62% of all changed lines. One npm install writes tens of thousands of lines to a lockfile, and it will drown every real signal you have.

So exclusions have to be on by default. But the corollary matters just as much: a tool that silently discards two thirds of your data has an invisible thumb on the scale. How would you know? You would not, which is why it must report what it removed and how much. If over half your churn vanishes into the exclusion list, you deserve to be told that before you read a single metric.

2. `git log` can return nothing and call it success

This one cost me a day. I ran a log with --format="COMMIT" as a record separator, parsed the output, and got clean zeroes across six repositories. No error. Exit code 0.

Git treats a format string containing no % placeholder as a named pretty-alias lookup. It finds no alias called COMMIT, emits nothing, and exits successfully. The pipeline downstream dutifully reported that nothing had happened.

I only caught it because a later start date returned more commits than an earlier one, which is impossible. Any tool built on parsing git log needs an assertion that it got something back. Silent zeroes are the most dangerous output a measurement tool can produce, because they look like findings.

3. Unknown is not zero

My exports captured commit subjects but not trailers, so the tool reported "0% AI co-authored" for a repository where 58% of commits carried a Co-Authored-By trailer. The data did not say zero. The data said nothing, and zero was the default.

Then a second version of the same bug: the exclusion step rebuilt each commit object and quietly dropped the trailers field, turning a known value back into an unknown that rendered as zero again.

A missing field and a measured zero must be different values all the way through, and they must render differently. Mine now prints "not captured by this source" and the JSON emits null.

4. Repository age fakes a trend

A repository younger than a year cannot contain year-old code, so its legacy-touch rate is structurally zero and rises as the calendar advances. Compare two windows of a maturing repository and you will see legacy touch climb impressively while nobody's behaviour has changed at all.

Worse, I hit a repository that looked eighteen months old and wasn't: it had been carved out of a larger one, so every file's history reset at the split. Real two-year-old code read as new.

If you take one thing from this piece: check where your repository actually came from before you measure its age.

5. `--all` can double your commit count

Including all refs picks up unmerged branch work. In one repository it took the commit count from 42 to 94. Neither number is wrong, they answer different questions: work you did, versus work that shipped.

Pick one, write it down, and apply it everywhere. The flags used should be printed with the results, because a comparison between two periods computed under different flags is not a comparison.

What it cannot do

It cannot detect AI. It detects when things changed, and any before-and-after split is only as honest as you are about when you changed method. Do you actually remember the week you started letting a model write your first drafts?

Which is worth dwelling on, because I got my own date wrong by sixteen days when asked to recall it three months later. That is why the tool has a detect command that looks for the step change itself and reports it as a candidate rather than a fact. If the person running the study cannot remember, no user will.

And moved-lines is a proxy. Git detects file renames, not the movement of a block of code between files, which is what the published research actually measures. That is a similarity-matching problem, and an approximation of it would give you numbers that look plausible and cannot be compared with the thing you would inevitably compare them against.

Go and check

Python 3.10 or newer, no dependencies.

git clone https://github.com/rich-atkins/git-habits && cd git-habits
python3 -m venv .venv && .venv/bin/pip install -e .

# no idea when your habits changed? most people do not
git-habits detect  --repo /your/repo --author "you@example.com"

# then compare either side of the date it suggests
git-habits compare --repo /your/repo --author "you@example.com" --split 2026-04-13

If the repository lives on a machine you cannot clone from, export its history there and analyse it here. The README has the command.

Its companion, diff-habits, goes one layer deeper. It reads diff contents and counts error-masking constructs, the empty catch blocks and suppressed type checks that remove the evidence of a failure rather than handle it. That is a separate tool rather than a flag on this one, deliberately: git-habits never opens a source file, which is what lets you run it against an employer's repository without a conversation. Reading source is a different decision and should be a different install. I have written that one up in the code that hides the evidence.

I am deliberately not telling you what my own numbers showed, because a before-and-after on one developer means very little without a control group. I have one, two colleagues in the same repositories who do not use AI at all, and that analysis is not finished. What I will say now is that the answer I was confident about after an afternoon was wrong, and every version since has been less exciting and more true.

Measure your own crossover. Do not inherit someone else's headline.

Written by Richard Atkins.

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

Org design for AI: why your Centre of Excellence becomes a bottleneck

Richard Atkins — Fri, 31 Jul 2026 05:00:00 +0000

Part 4 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A large enterprise I heard about stood up an AI Centre of Excellence with the best of intentions: one central team to set standards, vet models, and keep everyone safe. Eighteen months later, the CoE had a six-week approval queue, three of the business units had quietly started using unsanctioned tools to get round it, and the security team, the very people the CoE existed to reassure, had less visibility into the company's real AI use than before it was created. The gate had not stopped the risky behaviour. It had driven it underground.

The last piece ended on a warning: you can only fix ownership and accountability so far one team at a time, because they also live in how the whole organisation is built. So this is the structural piece. When a company decides to get serious about AI, it almost always reaches for the same move, and that move is usually where adoption goes to die.

The move is the Centre of Excellence. Pull the AI expertise into one central team, give them the mandate for standards, safety, and best practice, and let the rest of the organisation come to them. It sounds responsible. It is how most enterprises have handled every new capability for thirty years. And with AI it reliably curdles into a bottleneck.

How the enabler becomes the ceiling

The failure is not incompetence. It is a structural trap. A central team that must review every model, approve every use case, and sign off every rollout becomes, by simple arithmetic, the ceiling on how fast the rest of the company can move. Ten teams want to build; one team has to approve; the queue forms.

And people do not wait in queues. They route around them. This is not hypothetical. A 2026 survey of 1,000 employees and 500 security leaders found 81% of employees using unapproved AI tools, and 45% of workers actively finding workarounds to reach applications their employer had blocked. That second number is the one that should worry you, because it is the queue-jumping made explicit: the official channel is half-empty and the unofficial one is packed. Your careful central control produces the exact thing it was built to prevent, ungoverned AI use, now invisible, because you made the governed path the slow one.

A gate does not stop the water. It decides where the water goes around.

The counter-argument, taken seriously

The obvious objection: but you need governance. AI carries real risk, data, compliance, reputation, and "enablement" sounds like a polite word for a free-for-all. Fair, and worth answering directly, because it hides a false choice.

Enablement is not the absence of governance. It is governance delivered as a road rather than a checkpoint. A checkpoint governs by inspecting each car; a well-built road governs by making the safe route the default, guardrails, a paved surface, a sensible limit built into the design, so that the easy thing and the safe thing are the same thing. The shadow-AI numbers are the proof that gating fails at its own stated goal: the CoE that inspects every use case ends up with less control, not more, because it drove usage into the dark. If your genuine priority is safety, the queue is the least safe design you could pick.

The pattern that actually scales

The organisations getting this right are inverting the central team's job. Instead of a team that approves, a team that enables. Two ideas from how high-performing engineering organisations are already built map onto this cleanly. The first is the enabling team: a small group whose success is measured by how quickly it can raise another team's capability and then leave, not by how many approvals it processes. Its job is to make the other teams good at AI, then get out of the way. The second is the platform team: it builds the paved road, the self-service tools, guardrails, and defaults that make the safe way to use AI the easy way. You do not govern by inspecting each decision. You govern by shaping the path so the default is already safe.

That is the shift in a line: govern the road, not each journey.

There is an older idea worth borrowing too, with a caveat. The Spotify model gave us the language of guilds and chapters, communities that cut across teams to spread a craft and stop knowledge pooling in silos. That cross-cutting community is a genuinely good home for AI practice: how we prompt, what we have found breaks, which failure modes to watch for. The caveat is honesty. The Spotify model was aspirational even at Spotify, and copied badly it becomes ceremony. Take the idea, spreading practice sideways across teams, not the diagram.

And there is a reason structure matters more here than almost anywhere. Conway's Law, the old observation that organisations ship systems which mirror their own communication structures, has never been more literal than with AI. Build a gatekeeping organisation and you get gatekept, brittle adoption. Build an enabling one and you get adoption that flows. The shape of the team becomes the shape of the capability.

What this asks of a leader

Resist the instinct that says control means a checkpoint. On a technology moving this fast, a checkpoint is a bottleneck wearing a lanyard. Design for enablement instead: a central team measured on how much capability it builds elsewhere, a platform that makes the safe path the easy one, and a community that spreads what works. Governance that enables velocity rather than gating it is not a slogan. It is a structural decision about whether your best people wait in a queue or get a paved road, and, on the evidence, about whether you have real control or only the paperwork of it.

But structure has a hard limit, and it is the one this series has been circling from the start. You can build the most elegant enabling organisation in the world and it will still fail if there is nobody left to enable, if the pipeline that produces capable people has quietly been switched off. That is the last piece, and it is the one I think we are getting most wrong.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 4. Links added as each publishes.

Written by Richard Atkins.

Sources: Team Topologies (Skelton & Pais) — https://teamtopologies.com/key-concepts · "AI Center of Excellence: Why Most Become Bottlenecks" — https://agility-at-scale.com/ai/people-change/ai-center-of-excellence/ · Shadow-AI prevalence: UpGuard, "The State of Shadow AI" — https://www.upguard.com/resources/the-state-of-shadow-ai ; https://redteampartner.com/blog/shadow-ai-enterprise-risk/ · The Spotify model — https://www.atlassian.com/agile/agile-at-scale/spotify · Conway's Law (Melvin Conway, 1968).

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

Managing a team of agents: leadership when roles become software

Richard Atkins — Wed, 22 Jul 2026 05:00:00 +0000

Part 3 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

An agent on your team drafts a client report. Another reviews it. A third sends it. It goes out with a number that is wrong, not obviously wrong, plausibly wrong, and the client acts on it. Monday morning, someone asks who is responsible. Point to a person. If you find yourself hesitating between the drafting agent, the reviewing agent, and the human who "just" pressed send on work three agents produced, you have found the real management problem with agents, and it is not their capability.

So far in this series: the human cost of AI is a leadership choice (piece one), and the authority in the room is no longer whoever knows the most, but whoever knows where to trust the machine (piece two). This piece is about what management becomes when the organisation stops hiring for a role and starts spinning up an agent for it instead.

Picture the team you might run in two years. Three people. Nine agents: one drafting, one reviewing, a couple monitoring, others doing work that used to carry a job title. The people wear several hats each; the agents wear the rest. Our entire apparatus for running teams, ownership, accountability, span of control, was built for humans. Agents quietly break all three.

Oversight was the job. It is not any more.

For decades, a large part of middle management was moving information and checking work: routing decisions up and down, monitoring who did what, catching mistakes before they travelled. Agents do the routing and much of the checking themselves. So the manager whose value was oversight is, honestly, in trouble, and the numbers reflect it. Middle managers grew to around 13% of the US workforce by 2022, up from roughly 9% in the early 1980s, and a good deal of that growth was the oversight work agents now absorb. The pressure is already visible: a June 2026 Harvard Business Review analysis describes middle managers being overloaded by AI adoption, and Gartner predicts that this year one in five organisations will use AI to flatten their structure, eliminating more than half of their middle-management positions. The oversight layer is not just shifting; in places it is being deleted. The research on where the role is heading is consistent: from monitoring to facilitation, from watching work happen to making it possible.

That is not a demotion. It is a harder job. Overseeing ten people is a known problem with a century of practice behind it. Enabling three people to direct a shifting fleet of agents, and staying accountable for what that fleet produces, is not a problem most managers have ever been trained for.

The three things agents break

Ownership. People take ownership of work they feel is theirs. Split a task across three people and six agents and ownership evaporates: everyone contributed, nobody owns it. The fix is not technical, it is a deliberate management act. Name a human owner for every outcome, not every task. The agents can do the work; a person still has to own the result, including the parts the agents got wrong.

Accountability. This is where the "it's just delegation" objection breaks down. Good managers already delegate, the argument goes, and org charts always adapt. But delegation, properly understood, is to a human who can be asked why, who feels the consequence, who learns and carries the responsibility next time. An agent can do none of that. It cannot be accountable. So accountability does not distribute when you deploy agents the way it does when you delegate to people. It concentrates, upward, onto the humans who directed them. The old management principle held that you can delegate authority but never responsibility. Agents make that literal and unforgiving: the more of a team's work you automate, the more exposed its remaining people become, because they now answer for output they did not personally produce. A leader who misses this will let their best people quietly absorb unbounded risk, and call it efficiency.

Reliance. Here is a result worth pinning to the wall. In a 2025 experiment, people systematically over-relied on AI advice even when their own judgment would have been better, and the driver was not laziness. It was incentives. When people were rewarded for throughput, they rubber-stamped. When the incentive was redesigned to reward good judgment about when to trust the AI, over-reliance fell. Your team will trust agents exactly as much as your incentives tell them to. Reward speed, and you will get rubber-stamping, and, per the accountability point above, you will personally own the results of it.

What managing actually becomes

Less monitoring, more designing. Who owns which outcome. What a human must still decide. How you reward judgment over throughput. And what your real span of control now is, because a manager of three people directing fifty agents has a span of attention no org chart has ever had to model. The manager's craft moves from supervising effort to engineering accountability across a team where most of the doing is done by things that cannot be held responsible.

That is a real skill, and almost nobody has it yet, which is precisely the argument for learning it early. The managers worth most in five years will be the ones who worked out, now, how to keep ownership and judgment human while the doing moved to software.

There is a limit to how far you can solve this one team at a time, though. Ownership and accountability do not only live in a manager's head. They live in how the whole organisation is structured, and most organisations are about to reach for the same structure to "manage AI." Most of them will build a bottleneck and call it a centre of excellence. That is where the series goes next.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 3. Links added as each publishes.

Written by Richard Atkins.

Sources: "What's the Future of Middle Management?", HBR (2025) — https://hbr.org/2025/04/whats-the-future-of-middle-management · "Managers Managing AI Agents", Business Insider (2025) — https://www.businessinsider.com/ai-agent-managers-new-job-2025-11 · Holstein et al. (2025), "When Thinking Pays Off: Incentive Alignment for Human-AI Collaboration" — https://arxiv.org/abs/2511.09612 · "AI Adoption Is Overloading Your Middle Managers", HBR (June 2026) — https://hbr.org/2026/06/ai-adoption-is-overloading-your-middle-managers

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

Who's the authority now? Leading in the age of the jagged generalist

Richard Atkins — Sun, 19 Jul 2026 17:48:18 +0000

Part 2 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A senior engineer I know approved a database migration plan last year. It was clear, well-reasoned, confidently written, and it would have taken the production system down, because one step assumed a lock the database did not actually provide. The plan came from an AI. The engineer was good. He caught it on the second read, not the first, because it read exactly like the correct plans he approves every day. That gap, between how right it looked and how wrong it was, is the whole subject of this piece.

The first piece argued that AI's cost to people's wellbeing is a leadership choice, not a technology outcome. This one is about a quieter loss: authority. When the model is a better specialist than most of your team on a Tuesday, and confidently, plausibly wrong on Wednesday, who is the expert in the room?

Start with the finding that should unsettle anyone putting AI into skilled work. In 2023, researchers at Harvard Business School and Boston Consulting Group ran a field experiment with several hundred management consultants. On tasks that sat inside the AI's capability, the consultants using it produced work rated around 40% higher in quality, finished roughly 25% faster, and completed more of it. On a task designed to sit just outside that capability, the pattern flipped: the ones using AI were more likely to reach the wrong answer than the ones without it. Not a little wrong. Fluently, confidently wrong, because the model produced something that looked exactly as authoritative as its correct work.

They called it the jagged frontier. AI is brilliant and useless in a pattern you cannot see from the outside, and the two sit right next to each other. The same tool that makes your team look expert on one task quietly makes them look expert while being wrong on the adjacent one. The line usually attributed to Mark Twain fits it better than anything written since: it ain't what you don't know that gets you into trouble, it's what you know for sure that just ain't so. AI is a confident-wrong machine, and confidence is the one signal humans are worst at ignoring.

"Is AI the specialist now?" is the wrong question

Watching a model out-perform your people on a narrow task, it is tempting to conclude that the AI is now the specialist and the humans are drifting into generalists who wear many hats. That gets the shape of it backwards.

A specialist you can rely on. Their expertise has edges you understand. You know roughly what they know, and where their competence stops, which is exactly what lets you trust them. The jagged frontier means AI has no reliable edges. It is a generalist that performs like a specialist in unpredictable patches. Treating it as "the authority" is not delegation to an expert. It is trusting a colleague who is sometimes brilliant, sometimes bluffing, and never tells you which.

A second body of evidence sharpens this. A 2025 meta-analysis in Psychological Bulletin reconciled years of contradictory findings about whether people trust or distrust AI, and the answer was: it depends on the task. People tend to appreciate AI on objective, capability-heavy work and resist it on subjective, personal work. Which means a blanket policy, "use AI" or "don't trust AI", is wrong almost by definition, because the right answer changes task to task, sometimes sentence to sentence.

So the authority in the room is not the model. And it is no longer, on its own, the person who knows the most about the subject. It is the person who knows where the model can be trusted and where it cannot. That is a genuinely new competence, and it is not the same as domain expertise. You can be the strongest engineer on the team and still wave through a plausible, wrong answer, because knowing a domain and knowing the shape of a model's blind spots are different skills. My engineer happened to have both. Not everyone will.

Garry Kasparov, who lost to a computer and then spent years studying human-machine teams, distilled it into a law: a weaker player with a better process, working with a machine, beats a stronger player with a worse process. The edge was never the human or the machine on its own. It was the process binding them, the judgment about who does what and when to override. That process is the authority now, and it lives in a person, not the model.

The part the productivity numbers hide

There is a second finding in that study every leader should hold onto. The consultants who gained the most from AI were the ones who started out weakest; the bottom half of performers improved far more than the top. AI is a leveller. It pulls the floor up.

Read quickly, that sounds like good news, and in the short term it is. Read slowly, it carries a warning. If AI makes your least experienced people produce work that looks senior, you lose the signal you used to manage by. You can no longer read competence from output, because the output has been levelled. The junior who genuinely understands and the junior who prompted well now hand you the same document. The tell you have relied on your whole career, good work means a good developer, quietly stops being true.

The counter-argument, taken seriously

The obvious objection: the models keep getting better, so will the frontier not just fill in and this whole problem solve itself? No, and it is worth being precise about why. The frontier moves, but it does not stop being jagged. Every jump in capability opens a new set of adjacent tasks the model now attempts and gets subtly wrong, because it attempts them with exactly the same confidence it brings to the ones it has mastered. A more capable model is a more capable confident-wrong machine on the new edge. And the signal-loss problem, the fact that you can no longer read competence from levelled output, does not improve as the model improves. It gets worse.

What this asks of a leader

Stop asking your people to trust or distrust AI as a blanket policy. Both are wrong on a jagged frontier. Ask them instead to build, and to show you, calibrated judgment: where they lean on the model, where they check it, and how they know the difference. Reward the engineer who catches the plausible, wrong answer over the one who ships fastest. Make "here is where this could be wrong, and here is how I checked" a first-class contribution rather than friction.

And notice what this quietly costs. Every task you hand entirely to the model is a task your people stop practising. The expertise that let my engineer catch the migration bug was built by doing that work himself, badly and then well, for years, the same work you are now tempted to automate away. Which raises a question worth leaving open: if we stop building expertise in our people, where does the next generation of people who can supervise the machine come from? Hold that one. The series comes back to it, and the answer is not comfortable.

For now, the nearer question. If judgment about the machine is the new authority, how do you actually run a team when half its "roles" are no longer people but agents you spin up on demand? That is next week.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 2. Links added as each publishes.

Written by Richard Atkins.

Sources: Dell'Acqua et al. (2023), "Navigating the Jagged Technological Frontier" (Harvard/BCG) — https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321 · Mollick, "Centaurs and Cyborgs on the Jagged Frontier" — https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged · Qin et al. (2025), "AI aversion or appreciation? A capability-personalization framework", Psychological Bulletin · Garry Kasparov, on process in human-machine teams (Kasparov's Law).

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

The psychological cost of AI is a leadership choice, not a technology outcome

Richard Atkins — Sun, 19 Jul 2026 17:48:09 +0000

Part 1 of five, and the opener. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A team gets a new set of AI tools on a Monday. By Friday the dashboards look wonderful: more output, faster turnaround, fewer late nights. Six weeks later the same team is quieter in stand-ups, slower to volunteer, and two of your best people have started updating their CVs. Nothing broke. The numbers are still good. Something else did.

Most of what you read about AI at work is about that first Friday, the productivity. This is about the six weeks after: the bill that comes with it, and who decides how large it is.

Start with how people actually feel. In a global survey by ADP Research this year, only 22% of workers strongly agreed that their job was safe from elimination. Pew finds US workers more worried than hopeful about AI at work. And the worry is not evenly spread; early-career workers report the sharpest sense that AI is already reshaping their jobs. Whatever the productivity story, the human one is anxious, and anxiety is not a neutral input to a workplace.

Now the harder evidence. A 2025 study in Humanities and Social Sciences Communications followed 381 employees across three waves and found that adopting AI significantly lowered people's psychological safety, and that drop, in turn, raised their depression. This is not grumbling. Psychological safety, the belief that you can speak up, admit a mistake, or ask a question without being punished for it, is the thing Amy Edmondson spent a career showing is the foundation of teams that learn. Erode it and people stop flagging problems, stop asking, stop taking the small risks that improve the work. Done carelessly, AI adoption erodes exactly that.

But here is the finding that should change how you read all of it. The same study found that ethical leadership buffered the damage. Where leaders behaved with integrity and care through the change, the hit to psychological safety was significantly softened. Same technology, same rollout, different leadership, different human outcome.

This is not one study's fluke. Gallup finds that manager support is among the biggest factors in how well employees adapt to AI and workplace change. The technology arrives for everyone; how it lands depends on the person managing it.

That is the argument of this piece, and the thread that runs through this series: the human costs of AI are leadership choices, not technology outcomes. The tool does not decide whether your people feel ownership or dread. You do.

Why AI reaches the parts of work motivation is made of

Management has always run on a quiet substrate most of us never name. People need to feel that their work is theirs, that they are good at it, and that they belong to something. Psychologists call these autonomy, competence, and relatedness, the three needs at the centre of self-determination theory, and decades of evidence say they are what turn a job into something a person actually cares about.

Used carelessly, AI erodes all three at once.

It erodes autonomy when it quietly narrows people's choices. In a 2024 experiment, operators supervising an automated system were handed fewer and fewer options by the AI. Their raw performance held up, but their sense of autonomy and the meaningfulness of the work fell, and, crucially, the effect grew the longer it ran. That is the detail most rollouts miss. The motivational cost is not a one-off dip people bounce back from. It compounds.

It erodes competence when the interesting, skill-building parts of a role are handed to a model and the person is left to check its output. A reviewer of AI work all day is not building the mastery that made them worth hiring; they are slowly becoming an approver, and most people can feel the difference. And it erodes relatedness when the colleague you used to turn to with a half-formed question is replaced by a prompt box that never asks how your weekend was.

None of that is the AI's doing. Every one of those is a design decision made by a leader: what to automate, what to leave with people, how much choice to preserve, whether to protect the parts of a job that make someone feel capable and connected.

Why leaders reach for the harmful version by default

Almost nobody chooses the damaging rollout on purpose. They back into it, because AI arrives dressed as an IT procurement rather than an organisational change. A tool gets bought, access gets switched on, a productivity target gets set, and the questions that would have protected people, what does this do to how the work feels, who loses the interesting part of their job, where does judgment still live, never get asked, because nobody owns them. The harm is rarely malice. It is a vacuum. And a vacuum is still a choice, just an unmade one.

The counter-argument, taken seriously

The obvious objection: is this not just change resistance? People always grumble about new tools, then adapt, and the gains are worth a few uncomfortable weeks. Sometimes, yes. But two things in the evidence say do not lean on that too hard.

First, the autonomy study found the erosion of meaning intensified over time rather than fading. That is the opposite of the "they will get used to it" pattern. Second, the psychological-safety study is measuring depression, a clinical outcome with real absenteeism and turnover attached, not a passing mood. "They will adapt" is a comfortable story precisely because it lets leadership off the hook for a cost that, on the numbers, does not simply wear off.

And the productivity-is-worth-it framing hides the actual trade. The short-term gain is often real; the 2024 study found that restricting operators to a single recommended action did improve immediate performance. The cost showed up later, in motivation, and it compounded. So the trade is not "wellbeing versus results." It is "results now versus results and people later." Calling that an inevitable consequence of the technology is simply a way of avoiding the decision.

What this asks of a leader

Not a wellbeing programme bolted on afterwards. Something earlier and cheaper: decide, deliberately, which human needs each AI rollout is going to protect.

Keep a real decision in the loop even when the model could make it, because autonomy is load-bearing and, on the evidence, removing it is a slow leak rather than a clean win. Automate the drudgery and defend the parts of a role where people build and feel their skill, rather than the reverse. Protect the human connections AI can quietly replace. And behave, through the disruption, in the way the data actually rewards, with the integrity and care that measurably buffers the harm. That last one costs nothing, and on the numbers it does the most.

None of this slows the technology down. It just refuses to pretend the human cost was handed to you by the machine.

If the psychological cost of AI is a choice, so is what happens to expertise itself. When the model is a better specialist than most of your team on a Tuesday and confidently, plausibly wrong on Wednesday, who is the authority in the room? That is next week.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 1, the opener. Links added as each publishes.

Written by Richard Atkins.

Sources: ADP Research, People at Work 2026 — https://fortune.com/2026/03/25/workers-anxious-scared-insecure-ai-adp-global-survey/ · Kim, Kim & Lee (2025), Humanities and Social Sciences Communications — https://www.nature.com/articles/s41599-025-05040-2 · Faas et al. (2024), "Give Me a Choice" — https://arxiv.org/abs/2410.07728 · Amy Edmondson, The Fearless Organization (psychological safety) · Self-determination theory (Deci & Ryan) · Gallup, on manager support and adapting to AI.

📬 Subscribe to The Grounded Architect — field notes on building production AI, and leading the people around it. Straight to your inbox, no hype.

DEV Community: Richard Atkins

Earned Autonomy

Thursday, 9am

The ladder

Where the rungs come from

Historical Precedents

The rewrite stage: four rungs in eight weeks

The gate is on rung one too

The ops-doctor: live on rung one, staying there for now

What the vendors already ship - and what they don't

The rung where it breaks

What demotion has to look like

Others are here too, but there are gaps

Why I am writing this down now

What this cannot do

Measurement tools need evals too

A tool that refuses to answer

The rule I keep arriving at

Proving the detector detects

The three bugs it caught on day one

Where this came from, and where it fails

One LLM judge is an opinion. Two model families agreeing is evidence.

The problem with one judge

The panel

What it found

The bug the gate was hiding

What it costs, honestly

The migration the data ordered: moving a production LLM pipeline off the Mac Mini under my desk

The bill, up front

What actually moved

What the plan got wrong

Did quality survive?

What it costs now

The other currency

An MCP server that has to prove itself: citations, abstention and entitlements as CI gates

The demo, in one screen

Why bother?

The first honest run found a real bug

Leakage is a ranking problem, not a filter problem

What it doesn't do

Try it

Cutting juniors is a choice, not an AI inevitability

First, be honest about the cause

The counter-argument, taken seriously

The pincer

What this asks of a leader

The code that hides the evidence

What counts as masking

Count only what was added

What the output looks like

What I found, and why it is not what I expected

What a rise actually means

The honest limitations

Try it on yours

What is deliberately missing

Measure your own coding habits before you believe anyone else's numbers

What git can actually tell you

What the output looks like

Normalise per line, not per commit

The five ways I nearly fooled myself

1. Generated files are most of your history

2. git log can return nothing and call it success

3. Unknown is not zero

4. Repository age fakes a trend

5. --all can double your commit count

What it cannot do

Go and check

Org design for AI: why your Centre of Excellence becomes a bottleneck

How the enabler becomes the ceiling

The counter-argument, taken seriously

The pattern that actually scales

What this asks of a leader

Managing a team of agents: leadership when roles become software

Oversight was the job. It is not any more.

The three things agents break

What managing actually becomes

Who's the authority now? Leading in the age of the jagged generalist

"Is AI the specialist now?" is the wrong question

The part the productivity numbers hide

The counter-argument, taken seriously

2. `git log` can return nothing and call it success

5. `--all` can double your commit count