DEV Community: Richard Atkins

The code that hides the evidence

Richard Atkins — Wed, 05 Aug 2026 10:24:00 +0000

Part of "AI, engineering and what survives production", a series on the parts of building with AI that hold up once real traffic hits them.

There is a category of code that nobody defends in principle and everybody writes in practice.

try {
  await syncCustomerRecord(id)
} catch (e) {}

You do not think that is good. Neither do I, and I have written it. It gets written anyway, usually at the end of a long day, usually because the failure was intermittent and the deadline was not. And it is worse than the bug it silences, because the bug at least had the decency to announce itself.

The distinction that matters is between handling a failure and hiding one. A catch block that logs, retries, or rethrows with context is engineering. A catch block that swallows is a deleted alarm. From ten feet away the two look identical: both are three lines and a pair of braces, both make the red text stop.

Here is what interested me: this category is countable. Not perfectly, but well enough to see a trend in your own codebase. So I built a tool to count it, pointed it at seven repositories I had worked in, and got an answer I was not expecting.

What counts as masking

I split it into two tiers, and keeping them separate is the decision that makes the whole number defensible. My first version had a single score, and I abandoned it within an hour of pointing it at a real TypeScript codebase, for reasons I will come to.

Hard masking removes a signal outright and is difficult to justify:

Pattern	Example
Empty catch	`catch (e) {}`
Silenced exception	`except ValueError: pass`, `rescue nil`
Suppressed checking	`@ts-ignore`, `# noqa`, `eslint-disable`
Weakened typing	`as any`
Stubbed return	`return None # TODO`
Discarded error	Go's `_` assignment over an `err`

Soft masking is legitimate often enough that putting it in a headline would mislead: optional chaining, nullish defaults, TODO markers.

Consider user?.profile?.name. On a genuinely optional field that is correct code. The same expression, written because something in the chain was unexpectedly undefined and the ?. made the error go away, is a completely different act with identical syntax. No static tool can tell those apart, and I stopped trying. So soft constructs are counted separately, reported separately, and the judgement stays with you rather than being quietly made on your behalf.

That split is not fussiness, it is what I learned from the version I threw away. Fold optional chaining into one score and a modern TypeScript codebase produces an alarming number that means nothing at all. The first person to check it will find their own perfectly reasonable ?. in the count, conclude the tool is crying wolf, and never open it again. One number you can defend beats two you cannot.

Count only what was added

I measure additions and ignore deletions. A deletion cannot introduce a swallowed error, and counting both sides double-counts a line that has merely moved from one file to another.

Then normalise per thousand added lines, for the same reason as ever: a raw count tracks how much you wrote that month, and what you want to know is density. Otherwise a productive quarter looks like a decline in quality, and you will spend a week investigating the wrong thing.

What the output looks like

Real output, from the data pipeline behind this publication:

diff-habits scan

  a data pipeline
    commits / files      360 / 584
    added lines          121,560  (83,840 meaningful)
    error masking (hard) 229  (1.88 per 1k added)
    error masking (soft) 417  (3.43 per 1k added)
    top patterns         nullish_default=232, stub_return=187,
                         optional_chain=175, any_cast_ts=22, lint_suppress=20

The pattern breakdown is the part I actually act on. "229 hard hits" is a number I can do
nothing with. "187 stubbed returns" is a Monday morning: I go and look at what those
functions hand back when the thing they called did not answer, and I usually find two or
three that should have been raising instead of shrugging.

What I found, and why it is not what I expected

Pointing this at seven repositories, hard-masking density ranged from 0.16 to 4.0 per thousand added lines. That is a twenty-five-fold spread across codebases written by the same three people.

The highest was a data pipeline that scrapes web content and calls language models. The lowest was a small marketing site.

Sit with that for a moment, because it is the whole lesson. The pipeline is not badly written. It is defensively written, correctly. Networks time out. Scraped pages change shape. Model output is unreliable by construction. Code that talks to unreliable things needs guards, and those guards look exactly like masking to a regex.

Which gives the finding I did not want and now think is the most useful thing here: error-masking density is dominated by problem domain, not by who or what wrote the code. I had a tidy hypothesis about authorship. The data said the strongest predictor was what the code talks to.

So if you compare masking density between two different codebases, what you learn is which one has the flakier dependencies. You learn nothing whatsoever about their authors. Compare a codebase with itself, over time. That is the only comparison this metric will support, and I would rather tell you that than let you draw a conclusion it cannot carry.

What a rise actually means

A rise is not automatically bad, and I was careful to build a tool that does not imply otherwise, because one that scolds you gets uninstalled.

Hardening a service against a genuinely flaky dependency raises the count and is usually the right call. So rising density says look here, not you have failed. What I want to know when I see it is whether the guards I added are logging, alerting, or falling back deliberately, or whether they are quietly returning empty and leaving the caller to draw its own conclusions. Those two look the same in a diff and could not be less alike in production.

The question the number should prompt is the one I now ask myself in review: if this construct fires at three in the morning, does anybody find out?

The honest limitations

These are regexes, not a parser. They over-count a legitimate optional chain and under-count a swallow spread across several lines. except ValueError: on one line followed by an indented pass on the next slips straight through, and catching that properly needs an AST rather than a pattern. I decided that was a worthwhile trade for something you can run on any language in a second, but it means the trend is the meaningful part and a single absolute number is not an audit.

Language scoping matters more than I expected. An early version cheerfully matched TypeScript optional chaining inside Python files, which is obvious in hindsight and was not obvious to me until the numbers came out strange. Patterns are scoped by file extension now. If you extend the pattern list, scope yours too.

Small counts are noise, and I nearly fooled myself with one. A repository of mine produced a single hit across six thousand lines, and for about ten minutes I read that as an impressively clean codebase. It is not a low density. It is not enough data to have a density at all.

Try it on yours

Python 3.10 or newer. It depends on git-habits,
so exclusion rules stay identical between the two tools and their numbers remain
comparable.

git clone https://github.com/uxdw/diff-habits && cd diff-habits
python3 -m venv .venv && .venv/bin/pip install -e .

diff-habits scan    --repo /your/repo --author "you@example.com"
diff-habits compare --repo /your/repo --author "you@example.com" --split 2026-01-01

Unlike git-habits, this one needs a real working repository. It reads diff bodies, and
a metadata export does not contain them.

It reads locally and emits counts. Nothing is uploaded, and there is no network access to upload it with.

That is deliberate, and it is why this is a separate tool from its companion git-habits rather than a flag on it. git-habits works from commit metadata and never opens a source file, so you can run it on an employer's repository without a conversation. This one reads your code. That is a different decision, and it should be a different install rather than a flag you might not notice.

What is deliberately missing

The tool also implements block duplication, and it is off by default behind --experimental-duplication. The current approach counts any five-line sequence recurring anywhere in the history it walks, which conflates real copy and paste with code re-added after a refactor and with ordinary boilerplate. Measured against real repositories it reads about three orders of magnitude above published figures. It is useful as a trend within one repository and useless as an absolute number, so it says so and stays out of the default output.

And line-level move detection, the signal behind the widely repeated claim that refactoring is collapsing, is not implemented at all. It is a similarity-matching problem rather than a hashing one, and a naive version would produce numbers that look plausible and cannot survive comparison with the research they would be quoted against.

Both of those are in the README rather than an issue tracker, because a tool that quietly ships a broken metric is worse than one that admits to a gap. The whole point of measuring is to stop guessing. A measurement you cannot trust is just a guess wearing a number.

Companion piece: measure your own coding habits before you believe anyone else's numbers, on what git history alone can tell you, and the five ways I nearly fooled myself getting there.

Written by Richard Atkins.

Measure your own coding habits before you believe anyone else's numbers

Richard Atkins — Wed, 05 Aug 2026 09:32:00 +0000

Part of "AI, engineering and what survives production", a series on the parts of building with AI that hold up once real traffic hits them.

There is a claim going round that you have probably absorbed by now: AI-assisted development is making codebases worse. Refactoring is down, duplication is up, we are all writing more and revising less. The numbers behind it are real, the samples are enormous, and I found I had started repeating the conclusion in conversation without ever having checked it.

Then it occurred to me that those figures are averages taken across hundreds of millions of changes from thousands of organisations, not one of which is mine. So what is the rate in your repository? Nobody has told you, and on current evidence nobody is going to. I set out to find mine, assumed it would take an afternoon, and spent three days discovering that the answer is far harder to get at than the confident version suggests.

So this is not a piece about what AI does to code. It is about how to ask that question of your own repository without arriving at a wrong answer, which turned out to be the genuinely difficult part. The tool I built to do it is git-habits: free, local, and it reads no source code whatsoever.

What git can actually tell you

Git history is a surprisingly rich behavioural record. Not of quality, about which it knows nothing at all, but of habits: how often you commit, how large those commits are, whether you go back and change what you wrote last month, and whether anybody still touches the old code. That is a narrower thing than quality and it is the thing the industry claims has changed, so it is the thing worth measuring.

Four signals are computable from commit metadata alone, without opening a single source file:

Moved lines. The share of changed lines sitting in files git detected as renamed or copied. It is the closest thing history offers to "somebody went back and reorganised this."

Legacy touch. The share of changes landing on files nobody has touched in a year or more. Whether old code is still maintained or merely accumulating.

Rework. The share of changes landing on files touched again within a fortnight. Short-cycle churn.

Commit shape. How many commits per working day, and how large each one is.

None of these is a quality measure, and I want to be clear that I am not pretending otherwise. They are habits. Habits are what the claim is actually about, so habits are what you can test, and a narrow measurement you trust beats a broad one you do not.

What the output looks like

Real output, from one of my own repositories, split at an arbitrary date:

git-habits compare  split at 2026-05-01

  ruleset v1.0  excluded 28,297 of 101,578 changed lines (27.9%)
    lockfiles            24,513
    data_dumps            3,778
    binary_media              5

  first month
    window            2026-04-03 to 2026-04-24  (21d span, 7 active)
    commits           136   (19.43/active day)
    changed lines     41,788
    lines/commit      mean 307.3  p50 90.0  p90 430.0
    moved (reuse)     0.16%  (1.6 per 1k changed lines)
    legacy touch      0.0%  (0.0 per 1k)
    rework <=14d      17.24%  (172.4 per 1k)
    AI co-authored    45 commits (33.1%)

  after
    window            2026-05-04 to 2026-07-18  (76d span, 11 active)
    commits           44   (4.0/active day)
    changed lines     31,493
    lines/commit      mean 715.8  p50 66.5  p90 2997.0
    ...

Two things in there are worth pointing at before we go further, because both are the
kind of detail a dashboard would smooth over.

The exclusion report comes first, not last. Nearly 28% of the changed lines in that
window were lockfiles and data dumps. You are told that before you read a single metric.

Look at the mean and the median diverge in the second window. Mean 715.8, median 66.5,
p90 2,997. A handful of enormous commits have eaten the mean while the typical commit
actually got smaller. Read only the mean and you would conclude the opposite of what
happened.

Normalise per line, not per commit

This decision matters more than it looks. Measure anything "per commit" and, the moment your commit granularity changes, every metric moves while nothing underneath it has changed at all. A team that switches to squash merges halves its commit count overnight and could be forgiven for thinking it had halved its output. So I made per changed line the primary normalisation, which has the useful side effect of keeping the numbers comparable with the published research.

I kept the per-commit view as well, though, because it answers a different question and the gap between the two is where the artefacts hide. A metric that shifted only because commits got bigger is not a finding, and you want that visible rather than smoothed away.

One caveat on averages. In one repository I measured, the mean commit was 860 changed lines and the median was 16. A handful of bulk imports had eaten the mean entirely. Report the median. If a tool shows you only a mean, it is showing you its largest commits.

The five ways I nearly fooled myself

Every one of these produced a confident, wrong number during the build. They are in the tool's output and its README for that reason.

1. Generated files are most of your history

Lockfiles, build output, vendored dependencies, minified bundles, committed logs. Across the repositories I measured, generated artefacts accounted for between 48% and 62% of all changed lines. One npm install writes tens of thousands of lines to a lockfile, and it will drown every real signal you have.

So exclusions have to be on by default. But the corollary matters just as much: a tool that silently discards two thirds of your data has an invisible thumb on the scale. How would you know? You would not, which is why it must report what it removed and how much. If over half your churn vanishes into the exclusion list, you deserve to be told that before you read a single metric.

2. `git log` can return nothing and call it success

This one cost me a day. I ran a log with --format="COMMIT" as a record separator, parsed the output, and got clean zeroes across six repositories. No error. Exit code 0.

Git treats a format string containing no % placeholder as a named pretty-alias lookup. It finds no alias called COMMIT, emits nothing, and exits successfully. The pipeline downstream dutifully reported that nothing had happened.

I only caught it because a later start date returned more commits than an earlier one, which is impossible. Any tool built on parsing git log needs an assertion that it got something back. Silent zeroes are the most dangerous output a measurement tool can produce, because they look like findings.

3. Unknown is not zero

My exports captured commit subjects but not trailers, so the tool reported "0% AI co-authored" for a repository where 58% of commits carried a Co-Authored-By trailer. The data did not say zero. The data said nothing, and zero was the default.

Then a second version of the same bug: the exclusion step rebuilt each commit object and quietly dropped the trailers field, turning a known value back into an unknown that rendered as zero again.

A missing field and a measured zero must be different values all the way through, and they must render differently. Mine now prints "not captured by this source" and the JSON emits null.

4. Repository age fakes a trend

A repository younger than a year cannot contain year-old code, so its legacy-touch rate is structurally zero and rises as the calendar advances. Compare two windows of a maturing repository and you will see legacy touch climb impressively while nobody's behaviour has changed at all.

Worse, I hit a repository that looked eighteen months old and wasn't: it had been carved out of a larger one, so every file's history reset at the split. Real two-year-old code read as new.

If you take one thing from this piece: check where your repository actually came from before you measure its age.

5. `--all` can double your commit count

Including all refs picks up unmerged branch work. In one repository it took the commit count from 42 to 94. Neither number is wrong, they answer different questions: work you did, versus work that shipped.

Pick one, write it down, and apply it everywhere. The flags used should be printed with the results, because a comparison between two periods computed under different flags is not a comparison.

What it cannot do

It cannot detect AI. It detects when things changed, and any before-and-after split is only as honest as you are about when you changed method. Do you actually remember the week you started letting a model write your first drafts?

Which is worth dwelling on, because I got my own date wrong by sixteen days when asked to recall it three months later. That is why the tool has a detect command that looks for the step change itself and reports it as a candidate rather than a fact. If the person running the study cannot remember, no user will.

And moved-lines is a proxy. Git detects file renames, not the movement of a block of code between files, which is what the published research actually measures. That is a similarity-matching problem, and an approximation of it would give you numbers that look plausible and cannot be compared with the thing you would inevitably compare them against.

Go and check

Python 3.10 or newer, no dependencies.

git clone https://github.com/uxdw/git-habits && cd git-habits
python3 -m venv .venv && .venv/bin/pip install -e .

# no idea when your habits changed? most people do not
git-habits detect  --repo /your/repo --author "you@example.com"

# then compare either side of the date it suggests
git-habits compare --repo /your/repo --author "you@example.com" --split 2026-04-13

If the repository lives on a machine you cannot clone from, export its history there and analyse it here. The README has the command.

Its companion, diff-habits, goes one layer deeper. It reads diff contents and counts error-masking constructs, the empty catch blocks and suppressed type checks that remove the evidence of a failure rather than handle it. That is a separate tool rather than a flag on this one, deliberately: git-habits never opens a source file, which is what lets you run it against an employer's repository without a conversation. Reading source is a different decision and should be a different install. I have written that one up in the code that hides the evidence.

I am deliberately not telling you what my own numbers showed, because a before-and-after on one developer means very little without a control group. I have one, two colleagues in the same repositories who do not use AI at all, and that analysis is not finished. What I will say now is that the answer I was confident about after an afternoon was wrong, and every version since has been less exciting and more true.

Measure your own crossover. Do not inherit someone else's headline.

Written by Richard Atkins.

Org design for AI: why your Centre of Excellence becomes a bottleneck

Richard Atkins — Fri, 31 Jul 2026 05:00:00 +0000

Part 4 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A large enterprise I heard about stood up an AI Centre of Excellence with the best of intentions: one central team to set standards, vet models, and keep everyone safe. Eighteen months later, the CoE had a six-week approval queue, three of the business units had quietly started using unsanctioned tools to get round it, and the security team, the very people the CoE existed to reassure, had less visibility into the company's real AI use than before it was created. The gate had not stopped the risky behaviour. It had driven it underground.

The last piece ended on a warning: you can only fix ownership and accountability so far one team at a time, because they also live in how the whole organisation is built. So this is the structural piece. When a company decides to get serious about AI, it almost always reaches for the same move, and that move is usually where adoption goes to die.

The move is the Centre of Excellence. Pull the AI expertise into one central team, give them the mandate for standards, safety, and best practice, and let the rest of the organisation come to them. It sounds responsible. It is how most enterprises have handled every new capability for thirty years. And with AI it reliably curdles into a bottleneck.

How the enabler becomes the ceiling

The failure is not incompetence. It is a structural trap. A central team that must review every model, approve every use case, and sign off every rollout becomes, by simple arithmetic, the ceiling on how fast the rest of the company can move. Ten teams want to build; one team has to approve; the queue forms.

And people do not wait in queues. They route around them. This is not hypothetical. A 2026 survey of 1,000 employees and 500 security leaders found 81% of employees using unapproved AI tools, and 45% of workers actively finding workarounds to reach applications their employer had blocked. That second number is the one that should worry you, because it is the queue-jumping made explicit: the official channel is half-empty and the unofficial one is packed. Your careful central control produces the exact thing it was built to prevent, ungoverned AI use, now invisible, because you made the governed path the slow one.

A gate does not stop the water. It decides where the water goes around.

The counter-argument, taken seriously

The obvious objection: but you need governance. AI carries real risk, data, compliance, reputation, and "enablement" sounds like a polite word for a free-for-all. Fair, and worth answering directly, because it hides a false choice.

Enablement is not the absence of governance. It is governance delivered as a road rather than a checkpoint. A checkpoint governs by inspecting each car; a well-built road governs by making the safe route the default, guardrails, a paved surface, a sensible limit built into the design, so that the easy thing and the safe thing are the same thing. The shadow-AI numbers are the proof that gating fails at its own stated goal: the CoE that inspects every use case ends up with less control, not more, because it drove usage into the dark. If your genuine priority is safety, the queue is the least safe design you could pick.

The pattern that actually scales

The organisations getting this right are inverting the central team's job. Instead of a team that approves, a team that enables. Two ideas from how high-performing engineering organisations are already built map onto this cleanly. The first is the enabling team: a small group whose success is measured by how quickly it can raise another team's capability and then leave, not by how many approvals it processes. Its job is to make the other teams good at AI, then get out of the way. The second is the platform team: it builds the paved road, the self-service tools, guardrails, and defaults that make the safe way to use AI the easy way. You do not govern by inspecting each decision. You govern by shaping the path so the default is already safe.

That is the shift in a line: govern the road, not each journey.

There is an older idea worth borrowing too, with a caveat. The Spotify model gave us the language of guilds and chapters, communities that cut across teams to spread a craft and stop knowledge pooling in silos. That cross-cutting community is a genuinely good home for AI practice: how we prompt, what we have found breaks, which failure modes to watch for. The caveat is honesty. The Spotify model was aspirational even at Spotify, and copied badly it becomes ceremony. Take the idea, spreading practice sideways across teams, not the diagram.

And there is a reason structure matters more here than almost anywhere. Conway's Law, the old observation that organisations ship systems which mirror their own communication structures, has never been more literal than with AI. Build a gatekeeping organisation and you get gatekept, brittle adoption. Build an enabling one and you get adoption that flows. The shape of the team becomes the shape of the capability.

What this asks of a leader

Resist the instinct that says control means a checkpoint. On a technology moving this fast, a checkpoint is a bottleneck wearing a lanyard. Design for enablement instead: a central team measured on how much capability it builds elsewhere, a platform that makes the safe path the easy one, and a community that spreads what works. Governance that enables velocity rather than gating it is not a slogan. It is a structural decision about whether your best people wait in a queue or get a paved road, and, on the evidence, about whether you have real control or only the paperwork of it.

But structure has a hard limit, and it is the one this series has been circling from the start. You can build the most elegant enabling organisation in the world and it will still fail if there is nobody left to enable, if the pipeline that produces capable people has quietly been switched off. That is the last piece, and it is the one I think we are getting most wrong.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 4. Links added as each publishes.

Written by Richard Atkins.

Sources: Team Topologies (Skelton & Pais) — https://teamtopologies.com/key-concepts · "AI Center of Excellence: Why Most Become Bottlenecks" — https://agility-at-scale.com/ai/people-change/ai-center-of-excellence/ · Shadow-AI prevalence: UpGuard, "The State of Shadow AI" — https://www.upguard.com/resources/the-state-of-shadow-ai ; https://redteampartner.com/blog/shadow-ai-enterprise-risk/ · The Spotify model — https://www.atlassian.com/agile/agile-at-scale/spotify · Conway's Law (Melvin Conway, 1968).

Managing a team of agents: leadership when roles become software

Richard Atkins — Wed, 22 Jul 2026 05:00:00 +0000

Part 3 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

An agent on your team drafts a client report. Another reviews it. A third sends it. It goes out with a number that is wrong, not obviously wrong, plausibly wrong, and the client acts on it. Monday morning, someone asks who is responsible. Point to a person. If you find yourself hesitating between the drafting agent, the reviewing agent, and the human who "just" pressed send on work three agents produced, you have found the real management problem with agents, and it is not their capability.

So far in this series: the human cost of AI is a leadership choice (piece one), and the authority in the room is no longer whoever knows the most, but whoever knows where to trust the machine (piece two). This piece is about what management becomes when the organisation stops hiring for a role and starts spinning up an agent for it instead.

Picture the team you might run in two years. Three people. Nine agents: one drafting, one reviewing, a couple monitoring, others doing work that used to carry a job title. The people wear several hats each; the agents wear the rest. Our entire apparatus for running teams, ownership, accountability, span of control, was built for humans. Agents quietly break all three.

Oversight was the job. It is not any more.

For decades, a large part of middle management was moving information and checking work: routing decisions up and down, monitoring who did what, catching mistakes before they travelled. Agents do the routing and much of the checking themselves. So the manager whose value was oversight is, honestly, in trouble, and the numbers reflect it. Middle managers grew to around 13% of the US workforce by 2022, up from roughly 9% in the early 1980s, and a good deal of that growth was the oversight work agents now absorb. The pressure is already visible: a June 2026 Harvard Business Review analysis describes middle managers being overloaded by AI adoption, and Gartner predicts that this year one in five organisations will use AI to flatten their structure, eliminating more than half of their middle-management positions. The oversight layer is not just shifting; in places it is being deleted. The research on where the role is heading is consistent: from monitoring to facilitation, from watching work happen to making it possible.

That is not a demotion. It is a harder job. Overseeing ten people is a known problem with a century of practice behind it. Enabling three people to direct a shifting fleet of agents, and staying accountable for what that fleet produces, is not a problem most managers have ever been trained for.

The three things agents break

Ownership. People take ownership of work they feel is theirs. Split a task across three people and six agents and ownership evaporates: everyone contributed, nobody owns it. The fix is not technical, it is a deliberate management act. Name a human owner for every outcome, not every task. The agents can do the work; a person still has to own the result, including the parts the agents got wrong.

Accountability. This is where the "it's just delegation" objection breaks down. Good managers already delegate, the argument goes, and org charts always adapt. But delegation, properly understood, is to a human who can be asked why, who feels the consequence, who learns and carries the responsibility next time. An agent can do none of that. It cannot be accountable. So accountability does not distribute when you deploy agents the way it does when you delegate to people. It concentrates, upward, onto the humans who directed them. The old management principle held that you can delegate authority but never responsibility. Agents make that literal and unforgiving: the more of a team's work you automate, the more exposed its remaining people become, because they now answer for output they did not personally produce. A leader who misses this will let their best people quietly absorb unbounded risk, and call it efficiency.

Reliance. Here is a result worth pinning to the wall. In a 2025 experiment, people systematically over-relied on AI advice even when their own judgment would have been better, and the driver was not laziness. It was incentives. When people were rewarded for throughput, they rubber-stamped. When the incentive was redesigned to reward good judgment about when to trust the AI, over-reliance fell. Your team will trust agents exactly as much as your incentives tell them to. Reward speed, and you will get rubber-stamping, and, per the accountability point above, you will personally own the results of it.

What managing actually becomes

Less monitoring, more designing. Who owns which outcome. What a human must still decide. How you reward judgment over throughput. And what your real span of control now is, because a manager of three people directing fifty agents has a span of attention no org chart has ever had to model. The manager's craft moves from supervising effort to engineering accountability across a team where most of the doing is done by things that cannot be held responsible.

That is a real skill, and almost nobody has it yet, which is precisely the argument for learning it early. The managers worth most in five years will be the ones who worked out, now, how to keep ownership and judgment human while the doing moved to software.

There is a limit to how far you can solve this one team at a time, though. Ownership and accountability do not only live in a manager's head. They live in how the whole organisation is structured, and most organisations are about to reach for the same structure to "manage AI." Most of them will build a bottleneck and call it a centre of excellence. That is where the series goes next.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 3. Links added as each publishes.

Written by Richard Atkins.

Sources: "What's the Future of Middle Management?", HBR (2025) — https://hbr.org/2025/04/whats-the-future-of-middle-management · "Managers Managing AI Agents", Business Insider (2025) — https://www.businessinsider.com/ai-agent-managers-new-job-2025-11 · Holstein et al. (2025), "When Thinking Pays Off: Incentive Alignment for Human-AI Collaboration" — https://arxiv.org/abs/2511.09612 · "AI Adoption Is Overloading Your Middle Managers", HBR (June 2026) — https://hbr.org/2026/06/ai-adoption-is-overloading-your-middle-managers

Who's the authority now? Leading in the age of the jagged generalist

Richard Atkins — Sun, 19 Jul 2026 17:48:18 +0000

Part 2 of five. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A senior engineer I know approved a database migration plan last year. It was clear, well-reasoned, confidently written, and it would have taken the production system down, because one step assumed a lock the database did not actually provide. The plan came from an AI. The engineer was good. He caught it on the second read, not the first, because it read exactly like the correct plans he approves every day. That gap, between how right it looked and how wrong it was, is the whole subject of this piece.

The first piece argued that AI's cost to people's wellbeing is a leadership choice, not a technology outcome. This one is about a quieter loss: authority. When the model is a better specialist than most of your team on a Tuesday, and confidently, plausibly wrong on Wednesday, who is the expert in the room?

Start with the finding that should unsettle anyone putting AI into skilled work. In 2023, researchers at Harvard Business School and Boston Consulting Group ran a field experiment with several hundred management consultants. On tasks that sat inside the AI's capability, the consultants using it produced work rated around 40% higher in quality, finished roughly 25% faster, and completed more of it. On a task designed to sit just outside that capability, the pattern flipped: the ones using AI were more likely to reach the wrong answer than the ones without it. Not a little wrong. Fluently, confidently wrong, because the model produced something that looked exactly as authoritative as its correct work.

They called it the jagged frontier. AI is brilliant and useless in a pattern you cannot see from the outside, and the two sit right next to each other. The same tool that makes your team look expert on one task quietly makes them look expert while being wrong on the adjacent one. The line usually attributed to Mark Twain fits it better than anything written since: it ain't what you don't know that gets you into trouble, it's what you know for sure that just ain't so. AI is a confident-wrong machine, and confidence is the one signal humans are worst at ignoring.

"Is AI the specialist now?" is the wrong question

Watching a model out-perform your people on a narrow task, it is tempting to conclude that the AI is now the specialist and the humans are drifting into generalists who wear many hats. That gets the shape of it backwards.

A specialist you can rely on. Their expertise has edges you understand. You know roughly what they know, and where their competence stops, which is exactly what lets you trust them. The jagged frontier means AI has no reliable edges. It is a generalist that performs like a specialist in unpredictable patches. Treating it as "the authority" is not delegation to an expert. It is trusting a colleague who is sometimes brilliant, sometimes bluffing, and never tells you which.

A second body of evidence sharpens this. A 2025 meta-analysis in Psychological Bulletin reconciled years of contradictory findings about whether people trust or distrust AI, and the answer was: it depends on the task. People tend to appreciate AI on objective, capability-heavy work and resist it on subjective, personal work. Which means a blanket policy, "use AI" or "don't trust AI", is wrong almost by definition, because the right answer changes task to task, sometimes sentence to sentence.

So the authority in the room is not the model. And it is no longer, on its own, the person who knows the most about the subject. It is the person who knows where the model can be trusted and where it cannot. That is a genuinely new competence, and it is not the same as domain expertise. You can be the strongest engineer on the team and still wave through a plausible, wrong answer, because knowing a domain and knowing the shape of a model's blind spots are different skills. My engineer happened to have both. Not everyone will.

Garry Kasparov, who lost to a computer and then spent years studying human-machine teams, distilled it into a law: a weaker player with a better process, working with a machine, beats a stronger player with a worse process. The edge was never the human or the machine on its own. It was the process binding them, the judgment about who does what and when to override. That process is the authority now, and it lives in a person, not the model.

The part the productivity numbers hide

There is a second finding in that study every leader should hold onto. The consultants who gained the most from AI were the ones who started out weakest; the bottom half of performers improved far more than the top. AI is a leveller. It pulls the floor up.

Read quickly, that sounds like good news, and in the short term it is. Read slowly, it carries a warning. If AI makes your least experienced people produce work that looks senior, you lose the signal you used to manage by. You can no longer read competence from output, because the output has been levelled. The junior who genuinely understands and the junior who prompted well now hand you the same document. The tell you have relied on your whole career, good work means a good developer, quietly stops being true.

The counter-argument, taken seriously

The obvious objection: the models keep getting better, so will the frontier not just fill in and this whole problem solve itself? No, and it is worth being precise about why. The frontier moves, but it does not stop being jagged. Every jump in capability opens a new set of adjacent tasks the model now attempts and gets subtly wrong, because it attempts them with exactly the same confidence it brings to the ones it has mastered. A more capable model is a more capable confident-wrong machine on the new edge. And the signal-loss problem, the fact that you can no longer read competence from levelled output, does not improve as the model improves. It gets worse.

What this asks of a leader

Stop asking your people to trust or distrust AI as a blanket policy. Both are wrong on a jagged frontier. Ask them instead to build, and to show you, calibrated judgment: where they lean on the model, where they check it, and how they know the difference. Reward the engineer who catches the plausible, wrong answer over the one who ships fastest. Make "here is where this could be wrong, and here is how I checked" a first-class contribution rather than friction.

And notice what this quietly costs. Every task you hand entirely to the model is a task your people stop practising. The expertise that let my engineer catch the migration bug was built by doing that work himself, badly and then well, for years, the same work you are now tempted to automate away. Which raises a question worth leaving open: if we stop building expertise in our people, where does the next generation of people who can supervise the machine come from? Hold that one. The series comes back to it, and the answer is not comfortable.

For now, the nearer question. If judgment about the machine is the new authority, how do you actually run a team when half its "roles" are no longer people but agents you spin up on demand? That is next week.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 2. Links added as each publishes.

Written by Richard Atkins.

Sources: Dell'Acqua et al. (2023), "Navigating the Jagged Technological Frontier" (Harvard/BCG) — https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321 · Mollick, "Centaurs and Cyborgs on the Jagged Frontier" — https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged · Qin et al. (2025), "AI aversion or appreciation? A capability-personalization framework", Psychological Bulletin · Garry Kasparov, on process in human-machine teams (Kasparov's Law).

The psychological cost of AI is a leadership choice, not a technology outcome

Richard Atkins — Sun, 19 Jul 2026 17:48:09 +0000

Part 1 of five, and the opener. "AI, leadership and the human structures of work" is a series on what AI actually changes about leading people, and why those changes are choices, not inevitabilities.

A team gets a new set of AI tools on a Monday. By Friday the dashboards look wonderful: more output, faster turnaround, fewer late nights. Six weeks later the same team is quieter in stand-ups, slower to volunteer, and two of your best people have started updating their CVs. Nothing broke. The numbers are still good. Something else did.

Most of what you read about AI at work is about that first Friday, the productivity. This is about the six weeks after: the bill that comes with it, and who decides how large it is.

Start with how people actually feel. In a global survey by ADP Research this year, only 22% of workers strongly agreed that their job was safe from elimination. Pew finds US workers more worried than hopeful about AI at work. And the worry is not evenly spread; early-career workers report the sharpest sense that AI is already reshaping their jobs. Whatever the productivity story, the human one is anxious, and anxiety is not a neutral input to a workplace.

Now the harder evidence. A 2025 study in Humanities and Social Sciences Communications followed 381 employees across three waves and found that adopting AI significantly lowered people's psychological safety, and that drop, in turn, raised their depression. This is not grumbling. Psychological safety, the belief that you can speak up, admit a mistake, or ask a question without being punished for it, is the thing Amy Edmondson spent a career showing is the foundation of teams that learn. Erode it and people stop flagging problems, stop asking, stop taking the small risks that improve the work. Done carelessly, AI adoption erodes exactly that.

But here is the finding that should change how you read all of it. The same study found that ethical leadership buffered the damage. Where leaders behaved with integrity and care through the change, the hit to psychological safety was significantly softened. Same technology, same rollout, different leadership, different human outcome.

This is not one study's fluke. Gallup finds that manager support is among the biggest factors in how well employees adapt to AI and workplace change. The technology arrives for everyone; how it lands depends on the person managing it.

That is the argument of this piece, and the thread that runs through this series: the human costs of AI are leadership choices, not technology outcomes. The tool does not decide whether your people feel ownership or dread. You do.

Why AI reaches the parts of work motivation is made of

Management has always run on a quiet substrate most of us never name. People need to feel that their work is theirs, that they are good at it, and that they belong to something. Psychologists call these autonomy, competence, and relatedness, the three needs at the centre of self-determination theory, and decades of evidence say they are what turn a job into something a person actually cares about.

Used carelessly, AI erodes all three at once.

It erodes autonomy when it quietly narrows people's choices. In a 2024 experiment, operators supervising an automated system were handed fewer and fewer options by the AI. Their raw performance held up, but their sense of autonomy and the meaningfulness of the work fell, and, crucially, the effect grew the longer it ran. That is the detail most rollouts miss. The motivational cost is not a one-off dip people bounce back from. It compounds.

It erodes competence when the interesting, skill-building parts of a role are handed to a model and the person is left to check its output. A reviewer of AI work all day is not building the mastery that made them worth hiring; they are slowly becoming an approver, and most people can feel the difference. And it erodes relatedness when the colleague you used to turn to with a half-formed question is replaced by a prompt box that never asks how your weekend was.

None of that is the AI's doing. Every one of those is a design decision made by a leader: what to automate, what to leave with people, how much choice to preserve, whether to protect the parts of a job that make someone feel capable and connected.

Why leaders reach for the harmful version by default

Almost nobody chooses the damaging rollout on purpose. They back into it, because AI arrives dressed as an IT procurement rather than an organisational change. A tool gets bought, access gets switched on, a productivity target gets set, and the questions that would have protected people, what does this do to how the work feels, who loses the interesting part of their job, where does judgment still live, never get asked, because nobody owns them. The harm is rarely malice. It is a vacuum. And a vacuum is still a choice, just an unmade one.

The counter-argument, taken seriously

The obvious objection: is this not just change resistance? People always grumble about new tools, then adapt, and the gains are worth a few uncomfortable weeks. Sometimes, yes. But two things in the evidence say do not lean on that too hard.

First, the autonomy study found the erosion of meaning intensified over time rather than fading. That is the opposite of the "they will get used to it" pattern. Second, the psychological-safety study is measuring depression, a clinical outcome with real absenteeism and turnover attached, not a passing mood. "They will adapt" is a comfortable story precisely because it lets leadership off the hook for a cost that, on the numbers, does not simply wear off.

And the productivity-is-worth-it framing hides the actual trade. The short-term gain is often real; the 2024 study found that restricting operators to a single recommended action did improve immediate performance. The cost showed up later, in motivation, and it compounded. So the trade is not "wellbeing versus results." It is "results now versus results and people later." Calling that an inevitable consequence of the technology is simply a way of avoiding the decision.

What this asks of a leader

Not a wellbeing programme bolted on afterwards. Something earlier and cheaper: decide, deliberately, which human needs each AI rollout is going to protect.

Keep a real decision in the loop even when the model could make it, because autonomy is load-bearing and, on the evidence, removing it is a slow leak rather than a clean win. Automate the drudgery and defend the parts of a role where people build and feel their skill, rather than the reverse. Protect the human connections AI can quietly replace. And behave, through the disruption, in the way the data actually rewards, with the integrity and care that measurably buffers the harm. That last one costs nothing, and on the numbers it does the most.

None of this slows the technology down. It just refuses to pretend the human cost was handed to you by the machine.

If the psychological cost of AI is a choice, so is what happens to expertise itself. When the model is a better specialist than most of your team on a Tuesday and confidently, plausibly wrong on Wednesday, who is the authority in the room? That is next week.

The series: AI, leadership and the human structures of work

The psychological cost of AI is a leadership choice, not a technology outcome
Who's the authority now? Leading in the age of the jagged generalist
Managing a team of agents: leadership when roles become software
Org design for AI: why your Centre of Excellence becomes a bottleneck
Cutting juniors is a choice, not an AI inevitability

You are reading part 1, the opener. Links added as each publishes.

Written by Richard Atkins.

Sources: ADP Research, People at Work 2026 — https://fortune.com/2026/03/25/workers-anxious-scared-insecure-ai-adp-global-survey/ · Kim, Kim & Lee (2025), Humanities and Social Sciences Communications — https://www.nature.com/articles/s41599-025-05040-2 · Faas et al. (2024), "Give Me a Choice" — https://arxiv.org/abs/2410.07728 · Amy Edmondson, The Fearless Organization (psychological safety) · Self-determination theory (Deci & Ryan) · Gallup, on manager support and adapting to AI.

The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

Richard Atkins — Sat, 18 Jul 2026 15:11:28 +0000

The gif, and the honest headline

Here's a multi-stage LLM pipeline being killed mid-run and started again:

It picks up exactly where it stopped. No duplicate model calls, no re-spend. That's the engineering payoff. The headline I want to be honest about up front is the economics one, because it runs against the usual pitch:

This is not "run it locally and save a fortune." At a real content pipeline's scale, batched cloud inference is genuinely cheap. The valuable thing is a reproducible way to measure the local-vs-cloud crossover, and the answer of when local does and doesn't pay off.

Here's the real number, from an instrumented production run: one weekly run of this pipeline moves 641,671 input and 77,409 output tokens across nine local model stages, plus 134 locally-generated images. Run it on batched cloud models instead and that's about $0.88, under a dollar a week. So the interesting question was never "how much does local save." At this scale, cloud is already pocket change. It's when that stops being true, and how you would know.

The pains everyone hits on week two

The first version of any LLM pipeline is a for loop over a list. It works in the demo. Then production happens:

It dies at item 800 of 1000 and you do not want to pay for the first 800 again.
The same run, re-triggered, re-does work and double-writes downstream.
Near-duplicate inputs get processed twice, because nothing deduplicates them.
You have no idea what it cost, because nobody recorded the tokens.
A rate limit or a credit ceiling hits and the run dies dirty, mid-item.

None of these is hard on its own. Together they're the difference between a script and a pipeline. resumable-llm-pipeline is the small, framework-light reference that wires them together, extracted (patterns only) from a real weekly content pipeline at Cedar & Bloom. It runs offline on the stdlib alone via a deterministic mock router, so you can see all of it in five minutes.

The build

Two pieces carry the weight.

ProgressStore, for resumability. A crash-safe {item_id -> {status, ...}} written with a temp-file-and-rename atomic swap after every item. Each stage filters to unfinished items and reads the previous stage's output. That decoupling is the resumability: kill the process anywhere and re-running recomputes nothing that already finished. The test suite proves it in numbers. Across a killed run and its resume, total model calls equal the number of kept items exactly. No item is processed twice.

ModelRouter, for cost-aware routing. complete(prompt, tier) and embed(text) behind one uniform response that carries token counts. Two tiers (cheap and quality) map to a backend (mock, local ollama, or hosted) by one config line. Because both local and hosted return (text, tokens_in, tokens_out), cost and metrics are backend-agnostic, and swapping local for hosted to re-run the benchmark is a one-liner. A Budget(max_tokens, max_cost) guard checks the paid stage as it runs; on breach it checkpoints and exits cleanly. Resume is the retry.

Routing between a cheap and an expensive model is not a new idea.
RouteLLM
(paper) showed a learned router can cut cost by around 85% on some benchmarks while holding quality. This router is the boring, explicit version: you assign each stage a tier by hand. The goal is not to be clever, it is to make the cost of that choice measurable.

The measurement: the tokens were being thrown away

Here's the detail that started this. The production pipeline it generalises calls local models via Ollama, and Ollama's response includes prompt_eval_count and eval_count, the exact input and output token counts, in the final chunk. The original code threw them away. It did real token work every week and recorded none of it.

So the router captures those counts on every call (and usage.input_tokens / usage.output_tokens on the hosted path), and a fail-open metrics module writes one JSON line per call: tokens, latency, backend, model. Fail-open matters. The thing that measures the pipeline must never be the thing that breaks it. rlp report rolls it up.

Here's the actual per-stage scorecard from one instrumented weekly run (qwen2.5:14b is the cheap tier, qwen2.5:72b the quality tier):

stage          model            calls   tok_in   tok_out
novelty        qwen2.5:14b        273   417647    12860
rewrite        qwen2.5:72b         67   112516    48673
image_prompt   qwen2.5:14b         67    52193    11094
enrich         qwen2.5:14b         67    43759     3289
cluster        qwen2.5:14b          4    10264      700
timelines      qwen2.5:14b         10     2568       60
companies      qwen2.5:72b         20     1445       661
edition_intro  qwen2.5:14b          1     1279        72
image_gen      FLUX.1-schnell     134        0        0   (images — timing only)
--------------------------------------------------------------
TOTAL                                    641671    77409

Until this run, every one of those token counts was None. The pipeline did the work weekly and recorded nothing. Notice novelty on its own: 273 calls, 417k input tokens, 65% of the entire run's input. Hold that thought, it matters for the cost.

The benchmark: measure the crossover, don't guess it

rlp benchmark reads the tokens the run actually recorded, maps each tier to its hosted equivalent, projects to an annual workload, and compares hosted cost (list price, and with the Batch API's 50% discount) against a fixed local cost. Fed the real weekly run, with the cheap tier priced as claude-haiku-4-5 ($1/$5 per 1M) and the quality tier as claude-sonnet-5 ($3/$15 per 1M):

Measured (weekly run): 641,671 input + 77,409 output tokens across 9 stages + 134 images.
Modeled:  hosted prices as above, Batch 50% off, hosted image ~$0.003/img, local $560/yr amortised (hardware capex + power), 52 weekly runs/yr.

| line                         | per run | per year (x52) |
|------------------------------|---------|----------------|
| hosted text (list)           | $1.75   | $91            |
| hosted text (batch, 50% off) | $0.88   | $46            |
| hosted images (modeled)      | $0.40   | $21            |
| local (fixed, amortised)     | —       | $560           |

Break-even is about 640 weekly runs per year, roughly 12x the real cadence. Below that, batched cloud wins on cash.

Add the monthly evergreen run (measured: 47,596 in / 33,920 out, so $0.47 a run at list, $0.24 batched, roughly $3 to $6 a year) and the whole Cedar & Bloom LLM workload lands around $50 to $70 a year on batched cloud (about $110 at list). Against a $560/yr amortised local box, cloud is 8 to 12 times cheaper on cash at the real volume.

The discipline that makes this trustworthy, and the whole reason the tool exists: token counts are measured; hosted prices, the per-image cloud rate, and the local yearly cost are modeled and labelled. "The hardware's a sunk cost" is not a fair comparison. An honest break-even needs the amortised capex line, which is why it's in there.

The counter-intuitive bit: the cost isn't where the tokens are

Look again at that weekly scorecard through a cost lens:

stage	share of input tokens	share of hosted cost
`novelty` (cheap tier)	65%	~28%
`rewrite` (quality tier)	18%	~61%

novelty moves two-thirds of the tokens but stays cheap, because it runs on the cheap model and emits almost nothing. rewrite is the reverse: modest token volume, but its output tokens on the quality model at $15/1M dominate the bill. The money is in output tokens on the quality tier, not where the token volume is. Knuth's old warning that "premature optimization is the root of all evil" has a neat LLM corollary: don't guess which stage is expensive. The tokens pile up in novelty; the money leaves through rewrite. You only see that by measuring, and it's what tells you which stage to move to cloud first if you ever wanted a hybrid.

One genuinely measured cloud data point

Everything above prices local tokens as if they'd run on the cloud. But one stage in the wider Cedar & Bloom stack already is hosted: SteadyPath's quiz runs on Anthropic Haiku, and instrumenting it captured 4 real calls, 1,490 input and 1,861 output tokens, actual usage from the API, not a projection. Same metrics module, same JSONL, local and hosted side by side. That's the point of the backend-agnostic router. The moment you move a stage to the cloud, its real cost lands in the same ledger as the local estimate.

The honest conclusion the data supports: at Cedar & Bloom's real scale, batched cloud would cost tens of dollars a year, and a weekly run is under a dollar. Local's edge is control, privacy, and no rate limits, and it only wins on cash at roughly 10 times the volume, or once you bolt on a token-hungry agentic layer. The number that tells you which regime you're in is one you can now measure instead of argue about.

Take the pattern

git clone https://github.com/uxdw/resumable-llm-pipeline && cd resumable-llm-pipeline
python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
rlp reset && rlp run && rlp run       # watch the 2nd run resume with 0 model calls
rlp benchmark                          # your own local-vs-cloud crossover

Point the router at your models, feed the benchmark your real per-stage tokens and verified prices, and read your own break-even. Measure your crossover. Don't inherit someone else's headline.

(Companion piece: Stop shipping AI Agents you can't measure: evals observability from scratch, on evals and observability as a CI gate. This pipeline's metrics core is the same telemetry idea, pointed at cost.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.

Stop shipping AI agents you can't measure: evals + observability from scratch

Richard Atkins — Sat, 18 Jul 2026 15:00:00 +0000

The demo, in one screen

Here's an agent's eval scorecard on a green build:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          95.00%    95.00%   +0.00%
tool_correctness     100.00%   100.00%   +0.00%
schema_validity      100.00%   100.00%   +0.00%
groundedness         100.00%   100.00%   +0.00%
PASS: no regressions vs baseline.

Now I make one change. I stop the agent from actually reading the documents it cites, so it answers from search snippets alone. That's one environment variable, AES_DISABLE_FETCH=1. Re-run the exact same eval:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          75.00%    95.00%  -20.00%  <-- REGRESSION
tool_correctness      15.00%   100.00%  -85.00%  <-- REGRESSION
schema_validity      100.00%   100.00%   +0.00%
groundedness          15.00%   100.00%  -85.00%  <-- REGRESSION
FAIL: 3 metric(s) regressed vs baseline: task_success, tool_correctness, groundedness

The command exits non-zero. CI goes red, and the PR is blocked. One behavioural regression, caught by three independent metrics, before it ever reached a user.

Would your agent have caught that? For most teams I talk to, the honest answer is no. Not because they don't care, but because the tooling that makes it easy is all paid SaaS, and the DIY version never quite becomes a priority. This post is that DIY version, built once so you can copy it.

The gap

Everyone can build an agent now. Almost no one can prove it still works after the next prompt tweak, model bump, or refactor. The tools that help (Braintrust, LangSmith, DeepEval) are mostly paid platforms you bolt on later. For the deep version of why evals matter and how to design domain-specific ones, Hamel Husain's Your AI Product Needs Evals is the canonical read. This post is the smallest running skeleton of that idea, wired into CI.

There's no clean, vendor-neutral, framework-light reference that shows the whole loop: a working agent, traced, evaluated, and guardrailed, with evals running as CI so a regression fails the build like a broken unit test. So I built one. The repo, agent-eval-starter, runs end-to-end with zero API keys and zero network on a deterministic mock provider, so you can clone it and see all of the above in under five minutes.

The build

Four pieces, each deliberately boring:

A real agent loop using a JSON-action protocol rather than a vendor's native tool-calling. The agent emits {"action": "...", ...} and the harness parses it. That keeps it portable across providers and trivially mockable, with no framework lock-in.
OpenTelemetry tracing. The same spans that draw the trace tree roll up into a cost, latency, and token scorecard, and export to any OTel backend. It isn't a bespoke tracer: the OpenTelemetry GenAI semantic conventions are standardising exactly this shape of LLM span, so you're not locked into my format.
An eval harness that scores a committed, labelled dataset on four pass-rate metrics and compares against a committed baseline.json.
Guardrails: a deterministic banned-phrase post-filter, plus an optional LLM-as-verifier review pass.

The four metrics, each a pass-rate over the dataset:

task_success: did it answer correctly, or correctly abstain on the unanswerable cases? (Abstention counts as a correct answer.)
tool_correctness: did it use tools honestly, citing only documents it actually fetched?
schema_validity: is the final answer well-formed? (pydantic-validated)
groundedness: the hallucination guard. Is every claim backed by fetched text?

The insight worth stealing

The single most useful thing I've learned shipping this stuff:

Prompt-level guardrails ("never say X", "always cite your sources") are unreliable. A deterministic post-filter is the dependable fallback, and an LLM-as-verifier is the second layer, not the first.

The model will, eventually, ignore the instruction. So the banned-phrase check runs in code, after generation, every time. It cannot be talked out of it. The LLM-as-verifier pass then catches the fuzzier failures a deterministic filter can't express. Two layers,
cheap one first. It's why the demo works: groundedness isn't a line in the system prompt asking nicely, it's a computed check over what the agent actually fetched.

The numbers (and their caveats)

Everything above is measured on the repo's committed synthetic 20-case suite, not a production workload. I'm flagging that on purpose. The credibility of an eval story is the honesty about what got measured.

Baseline pass-rates: task_success 95%, tool_correctness 100%, schema_validity 100%, groundedness 100%.
One seeded regression (disabling document fetch) drops 3 of 4 metrics, the suite exits non-zero, and CI posts the scorecard on the PR.
Cost and latency per run: about $0 on the mock provider (no network). Point it at claude or ollama and the OpenTelemetry roll-up reports tokens, cost, and p95 latency from real calls, so accuracy and spend are tracked over time from the same spans.

None of this proves the agent is correct. As Edsger Dijkstra put it, testing "can be used to show the presence of bugs, but never to show their absence." What evals give you is the other thing production needs: a tripwire. The moment a change makes the agent measurably worse, the build stops.

So the point isn't "95% is good." It's that the 95% is a number you can defend, version, and regress against. A gate, not a vibe.

Take the pattern

git clone https://github.com/uxdw/agent-eval-starter && cd agent-eval-starter
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
agent-eval eval          # score the suite -> scorecard
make demo                # green, then a seeded regression turns it red

Swap the corpus and dataset for your domain, point the provider at your model, and commit a new baseline. The pattern is the product; the example is just a runnable seed. Make your agent's accuracy a CI gate.

(Companion piece: The real economics of a Production LLM
pipeline - resumability cost aware routing & measuring, on resumability, cost-aware routing, and measuring when local beats the API.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.

DEV Community: Richard Atkins

The code that hides the evidence

What counts as masking

Count only what was added

What the output looks like

What I found, and why it is not what I expected

What a rise actually means

The honest limitations

Try it on yours

What is deliberately missing

Measure your own coding habits before you believe anyone else's numbers

What git can actually tell you

What the output looks like

Normalise per line, not per commit

The five ways I nearly fooled myself

1. Generated files are most of your history

2. git log can return nothing and call it success

3. Unknown is not zero

4. Repository age fakes a trend

5. --all can double your commit count

What it cannot do

Go and check

Org design for AI: why your Centre of Excellence becomes a bottleneck

How the enabler becomes the ceiling

The counter-argument, taken seriously

The pattern that actually scales

What this asks of a leader

Managing a team of agents: leadership when roles become software

Oversight was the job. It is not any more.

The three things agents break

What managing actually becomes

Who's the authority now? Leading in the age of the jagged generalist

"Is AI the specialist now?" is the wrong question

The part the productivity numbers hide

The counter-argument, taken seriously

What this asks of a leader

The psychological cost of AI is a leadership choice, not a technology outcome

Why AI reaches the parts of work motivation is made of

Why leaders reach for the harmful version by default

The counter-argument, taken seriously

What this asks of a leader

Next

The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

The gif, and the honest headline

The pains everyone hits on week two

The build

The measurement: the tokens were being thrown away

The benchmark: measure the crossover, don't guess it

The counter-intuitive bit: the cost isn't where the tokens are

One genuinely measured cloud data point

Take the pattern

Stop shipping AI agents you can't measure: evals + observability from scratch

The demo, in one screen

The gap

The build

The insight worth stealing

The numbers (and their caveats)

Take the pattern

2. `git log` can return nothing and call it success

5. `--all` can double your commit count