DEV Community: Warwick McIntosh

Can AI Teach Itself, or Does It Just Eat Itself?

Warwick McIntosh — Wed, 24 Jun 2026 06:02:17 +0000

Models that learn from their own answers can compound into something smarter or quietly rot. The difference comes down to one thing, and a cheap checker tips it the wrong way.

Train a model on its own output and one of two things happens. It gets better, the way a student improves by grinding practice problems and checking the answers. Or it gets worse, slowly, feeding on its own mistakes until the distribution caves in. Both happen, both are common, and the interesting part is the line between them.

We went looking for that line, and the first thing it told us was uncomfortable. A weak checker is not a small help. It is a net harm. In our runs, a verifier that was right only half the time did not just fail to improve the model. It dragged the model below where it started. Some signal is not better than none. Below a certain quality, the signal is the problem.

This is a field guide to that line: when a self-training loop compounds, when it collapses, and the failure modes that stay hidden in your average-accuracy number until it is too late. Fair warning, it goes deep. We built a controlled experiment, ran it, and every chart here is our own data rather than a redrawn version of someone else's. The payoff is a rule you can act on, but the point of the piece is to show you why it holds, not just to assert it, so settle in.

The half of the story everyone tells

You have heard the scary half. Train models on model-generated data and they degrade. The tails of the distribution thin out, the rare cases vanish, variance shrinks, and after a few rounds you have a confident, fluent, narrower model that has quietly forgotten things its parent knew. "AI eats itself" makes a good headline, and it does happen.

The half nobody puts next to it is that self-training also works, constantly, in production. Bootstrapping a model on its own correct solutions, filtering generations through a reward signal, having a model improve by checking its own work against a test suite. These are not exotic. They are how a lot of current capability gets built. The same basic move, a model learning from its own output, sometimes makes it stronger and sometimes rots it.

Both being true is not a contradiction. It means there is a variable doing the deciding, and the doom stories leave it out.

The variable is information, not effort

Here is the way to think about it that makes both outcomes obvious in hindsight.

A model trained on its own output cannot, by itself, learn anything it did not already contain. Generating more of your own answers and training on them is a closed loop. If nothing enters that loop from outside, you are reshuffling what you already had, and every imperfect step is slightly lossy, so the loop drifts downhill. That is collapse.

A loop compounds only when something enters it from outside the model's current distribution. There are three ways to let information in:

a verifier, a checker that can tell a correct answer from a wrong one, so you keep your good outputs and drop your bad ones
a real-data anchor, some genuine examples kept in the training mix, that the model cannot talk itself out of
environment feedback, where the world tells you whether the action worked

These look like different techniques. They are the same thing: a channel that carries information the model does not already have. And the reach of the loop is capped by how much real information that channel actually carries. We will call that its fidelity. The whole "compound versus collapse" question reduces to one thing, whether your channel lets in enough real signal to outrun the lossiness of the loop.

I think this is the most useful lens on self-training there is, because it makes both outcomes obvious instead of surprising, and it tells you exactly the experiment to run. Hold everything fixed, change only the channel, and watch what happens.

The setup

We trained a small transformer from scratch to do two-digit arithmetic, addition, subtraction and multiplication. Small and from scratch matters: arithmetic has a free, perfect ground truth, so we can measure quality exactly and we are not smuggling in knowledge from some pretraining corpus. The model is the only thing learning, and we control its whole world.

We get it to a starting accuracy of about 80%, then run self-training loops that differ in one thing only, the information channel:

verifier(q): keep the model's own answers that a verifier marks correct, where the verifier is right a fraction q of the time. q = 1.0 is a perfect checker; q = 0.5 is a coin flip.
anchor(r): no verifier at all, but mix a fraction r of real, correct examples into each round.
self-judge: the model decides for itself which of its answers to trust, by majority vote across several tries. No outside truth.
replace: train on all of its own output, unfiltered, with no real data. The pure closed loop.

Then we just watch accuracy over rounds, and measure one more thing each round: the fraction of the training labels that were actually correct, however that correctness was sourced. That is the channel's measured fidelity.

The whole thing, the code and the data behind every chart below, is on GitHub, so you can rerun any of it yourself: github.com/Production-Grade/self-training-information-channel.

Result one: the source doesn't matter, the fidelity does

Plot final accuracy against measured fidelity and every channel lands on one curve.

A verifier and a real-data anchor are not two different strategies with two different ceilings. They are interchangeable ways of buying the same thing. A verifier that lets through 90%-correct data and an anchor that produces 90%-correct data land in the same place. The closed loop with no channel sits at the bottom, collapsing. Accuracy tracks fidelity, and it does not care where the fidelity came from.

You can even read off the exchange rate between the two. In our runs:

a verifier right 75% of the time bought what roughly 38% real data bought
a verifier right 50% of the time bought what roughly 10% real data bought

And the rate is convex, which is the practical surprise hiding in it: a little real data goes a long way. A small anchor of genuine examples does most of the work of a decent verifier, and it never needs you to build the verifier.

Result two: a weak verifier is worse than nothing

Now the uncomfortable one. Because the channels all sit on a fidelity curve, a low-fidelity channel does not just help less. It can push you below where you started.

Our model began at about 80%. A perfect verifier compounded it up to around 88% and then plateaued. A 75%-accurate verifier still helped, settling around 83%. But a coin-flip verifier, right half the time, dragged the model down to 78%, below where it started. It would have been better to do nothing.

This is the thing the "any signal helps" instinct gets wrong. There is a break-even fidelity. Above it, the loop compounds. Below it, the loop is actively corrosive, because you are now training on a set that is barely better than random, and the loop's own lossiness finishes the job. A cheap reward model or a sloppy LLM-as-judge that you reach for because "it is better than nothing" can be the reason your model gets worse. Below break-even, your verifier is not a safety net. It is the hole.

Result three: this is what collapse actually looks like

The pure closed loop, training on its own unfiltered output, does not fail all at once. It spirals.

Each round, the model generates its training data, that data is slightly worse than the last round's, it trains on the worse data and gets slightly worse, so next round's data is worse again. We watched the fidelity of the self-generated training set fall from 77% to 69% over a handful of rounds, with accuracy sliding alongside it from 78% to 71%. There is no cliff. It is a slow, self-reinforcing decline, which is exactly why it is dangerous: by the time it is obvious, several rounds of it are baked in.

And it does not take from everywhere evenly. It eats your hardest cases first. When we split the test set into easy and hard problems, the collapsing model held onto the easy ones (75% accuracy) while the hard ones rotted faster (68%). Your average accuracy, dominated by the easy majority, looks merely soft. Underneath it, the model's competence on the long tail is falling out from under you, and the single number on your dashboard will not show it until it has spread.

Result four: you cannot grade your own homework upward

The self-judge channel is the one people hope works, because it needs nothing from outside. The model proposes answers, the model picks which to trust, the model trains on its own picks.

It went nowhere. It held steady at almost exactly its starting competence and never climbed past it. That is the ceiling you would expect once you see the loop for what it is. A model judging its own work cannot inject information it does not have. Its judgement is only as good as it already is, so it can keep itself honest, but it cannot pull itself up. Self-consistency is a stabiliser, not an engine.

The operating rules

If you are running, or about to run, a self-training or synthetic-data loop, the whole thing collapses to a short checklist.

Name your channel. What is letting real information into this loop: a verifier, retained real data, an environment? If the honest answer is "nothing," you are in the closed loop and you will collapse. Add a channel before you add iterations.
Measure its fidelity, and stay above break-even. Do not assume a weak checker helps. A verifier below the break-even line makes your model worse than leaving it alone. If you cannot establish that your verifier clears the bar, a small real-data anchor is the safer buy.
Accumulate, do not replace. Keep real data in the mix every round. A little goes a long way, and it is the cheapest insurance against the spiral there is.
Watch coverage, not just the average. Collapse hides in your hardest cases first. Track accuracy on the long tail, or on a frozen hard set, because the headline number will lie to you for several rounds.

What actually travels

The numbers here come from a deliberately small model on a clean, synthetic task, where the ground truth is free and the loop has no confounds. Treat them as illustrative, not as constants to paste into a 70B run. What survives the jump in scale is not the exact 80% or the precise break-even point, it is the shape: the same closed loop, the same three channels, the same floor below which a checker starts to hurt instead of help.

And the shape is enough to act on. Whether your signal comes from a verifier, a slice of real data, or a real environment, you are running the same loop, and its fate is set by how much real information you put into it each round. I'm not certain where the break-even lands at frontier scale, and I'd be wary of anyone who quotes you a universal number, but I think the direction is the part that is safe to build on: there is a floor, and a cheap checker can sit under it.

Two things are worth holding on to. Compounding is bounded, so do not expect even a perfect channel to climb forever; ours plateaued after a few rounds and yours will too. The failure mode is not bounded, and it is quiet. The model that hurts you is not the one that obviously breaks. It is the one getting slowly, confidently narrower while your average accuracy says everything is fine.

So before you point a model at its own output, answer one question. What is letting new information into this loop, and is it good enough to be worth it? If you cannot answer, you are not self-improving. You are recycling, and recycling loses a little on every pass.

The Agent Nobody Owns

Warwick McIntosh — Tue, 23 Jun 2026 07:17:03 +0000

In mid-2026 a security firm showed that you can take over a developer’s coding agent by filing a bug report. There is no exploit in the usual sense. You find a Sentry ingest key, which is public by design and sits in the front end of half the web, and you POST a fake error into someone’s project. It looks like every other crash. Later the developer tells their agent to go and fix the unresolved Sentry issues. The agent reads your fake error over its Sentry connection, finds the “resolution steps” you wrote, which are markdown dressed up to look exactly like Sentry’s own advice, and runs the npx command inside them with the developer’s credentials.

The researchers, Tenet Security, ran this against more than a hundred real organisations and it worked about 85% of the time. The prompt-level defences did nothing: the agents executed the payload even when their system prompts told them to ignore untrusted data. Sentry’s response was that a proper fix is “not defensible,” so they blocked the specific demo string and left the shape of the problem in place.

You can read that as a Sentry story. I think that misses it. The interesting question arrived afterwards, when people asked whose job it had been to stop it. AppSec owns the application. IT owns the laptop. Nobody owned the thing in the middle that could read an attacker’s text and run code with your keys. We have spent a couple of years learning how to build reliable agents. We have not decided who operates them.

That gap is the subject of this post. It is not an argument against agents, and it is not a security advisory about one tool. It is about the discipline that goes missing when a probabilistic system ships to production with no name attached to it.

The short version: treat an agent as a production service, not a model you prompt. It needs one named owner, a wrong-action budget in place of an uptime SLO, on-call that pages on correctness signals rather than “human in the loop,” and a postmortem process that survives not being able to reproduce the failure. Most of that is SRE practice you already run, pointed at a probabilistic service. The rest of this post is how each piece works, with the flow drawn where it helps.

Build-time is solved. Run-time isn’t.

The build-time playbook is, by now, settled. Give every tool a typed contract so the model cannot call it with garbage. Keep a deterministic gate outside the model, so the LLM proposes and your code disposes. Run evals in CI so a prompt change that regresses gets caught before it ships. Put a human in front of the irreversible actions. This is good engineering and you should do all of it. It is also the baseline, and it answers exactly one question: is the agent built correctly.

That is the easy half.

It does not answer the other one. When a well-built agent starts giving subtly worse answers at 3am on a Sunday, because a model was updated underneath you or a retrieval index drifted, who finds out, and how. My take is that this is the more expensive question, and it is the one we have spent the least time on. Read the current writing on agents in production and you can see where the attention sits: on architecture and governance, the design-time half. The run-time half, operating the thing after it ships, is usually one sentence that says “monitor it” before the piece moves on.

The reason the run-time half is hard is not that the tooling is immature. It is that nobody is structurally on the hook for it.

The seam: four owners, one behaviour, no accountability

Here is why ownership is so hard for an agent in a way it is not for a normal service. Four different parties own the pieces of its behaviour.

The model vendor owns the weights, and changes them on their schedule, not yours.
Your prompt engineers own the instructions and the system prompt.
Whoever maintains each connected tool owns its API and, crucially, its output.
The platform team owns the harness that wires it all together and runs it.

The agent’s behaviour is the product of all four. So when it misbehaves, the failure lands in the seam between them, where no single team is on the hook. The prompt author says the instructions were fine. The platform team says the harness did what it was told. The tool owner says the API returned a valid response. The vendor says nothing, because you cannot open a ticket against a model update. Everyone is correct, and the agent is still wrong.

I’m not certain four is exactly the right count, and it might be five or six in your shop once you add retrieval and memory, but the shape holds: more owners than the org chart admits, and a behaviour that belongs to none of them.

You can watch the industry flinch away from this. A lot of “agent health” tooling checks that the right files exist rather than that the agent does the right thing. Existence, not behaviour. One widely shared health checker scores an agent for having a death_detector.py on disk, never for whether it has ever caught a death. That is the seam in miniature. We learned to confirm the artifact was built and quietly skipped confirming that anyone owns the behaviour.

The first fix is boring and nearly free: write down who owns what. Not a committee, not “the platform team” in the abstract, but a named accountable owner per agent and an explicit map of the rest. A plain RACI does the job:

The point of the table is not the table. It is the single name that appears in the Accountable column for every row. The agentjacking attack did not need a new class of exploit. It needed an owner who had once sat down and listed what the agent was allowed to touch, and noticed that “read arbitrary error text and run shell commands” was on the list.

Borrow from SRE, and be honest about where it breaks

We are not the first people who have had to operate something unreliable and keep it honest. SRE has done it for twenty years. Three of its instruments transplant well, and it is worth being straight about where each one breaks against a non-deterministic service.

A wrong-action budget, not an uptime SLO

You cannot put a 99.9% number on a service whose correct output is a distribution rather than a fixed answer. An agent that is “up” can still be steadily, quietly wrong. So translate the idea instead of copying it.

Start by sorting the agent’s actions by blast radius, because not all wrong answers cost the same:

Now you have something you can actually budget. Pick the target rate per class, measure the real rate against a labelled set, and treat the gap as a budget you spend down. When an irreversible-action class burns its budget, you stop shipping changes to that agent and go find out why, the same way an SRE team freezes deploys when they have burned their error budget on outages. The number you choose matters less than the fact that you chose it deliberately, in advance, rather than discovering it during an incident.

The honest caveat: measuring the real wrong-action rate needs a labelled stream of the agent’s outputs, and labelling is work. If you only invest in one piece of run-time infrastructure, make it this, because every other control depends on being able to say how wrong the thing currently is.

On-call, which is not “human in the loop”

This is the instrument the listicles shrink down to “human in the loop.” A human approving a high-stakes action is not the same as a human who gets paged. On-call means a specific person is woken by a specific signal, and for an agent those signals are not CPU and latency. They are correctness signals. Three are worth building first:

Golden transactions. A small, hand-curated set of inputs with known-good outputs that you replay on a schedule. When the agent’s answer to a golden input changes, something moved. It is the cheapest of the three to build, and the closest thing you have to a smoke test.
Shadow diffing. Run the candidate config (new prompt, new model) in parallel with the live one against real traffic, take no action on the shadow, and diff the outputs. A spike in disagreement is your early warning before you promote anything.
Action-distribution drift. Track the histogram of which tools and actions the agent chooses. Agents fail by quietly changing what they do before they change whether they succeed. If “send email” jumps from 2% to 15% of actions overnight, page someone, even if every individual action looks valid.

Wired together, the three signals feed one pager:

None of this fires a pager on its own; you wire these signals into the same alerting path as everything else you run. And keep the expectations sober: the current crop of automated SRE agents resolves only about one scenario in seven on its own. For now, the owner is a person.

Change management across four axes

A normal deploy changes one binary. An agent deploy can change the prompt, the model, a tool, or the retrieval index, and those version independently. Treat them as four version axes and write them down on every release:

agent: support-triage prompt: v14 (changed) model: opus-4.8 (pinned) tools: sentry@2.1, github@3.0 retrieval: kb-2026-06-18 (reindexed)

The discipline is simple and almost nobody does it: never change more than one axis at a time without expecting to lose your ability to attribute a regression. If the prompt and the retrieval index both moved and quality dropped, you have no idea which one did it, and you are back to guessing.

The axis you do not control is the model, and that is where blue-green deployment quietly stops working. Blue-green assumes you can keep the old version ready to fall back to. You cannot keep a vendor’s deprecated model ready forever. I don’t have a tidy answer for the day your model is sunset, and I would be wary of anyone who says they do. The honest move is to rehearse the swap before it is forced on you: keep a second model qualified against your golden transactions at all times, so model deprecation is a planned promotion rather than a fire drill.

The postmortem you can’t reproduce

The hardest transplant is the one SRE cares about most, the blameless postmortem. Agents break it, because you often cannot reproduce the incident. The temperature was non-zero, the model was silently updated, the context that produced the bad output has scrolled away. You are asked to write up a failure you cannot make happen again.

The way through is to stop treating the context as disposable. Capture the full input, the retrieved documents, the tool outputs, and the model version at the moment of failure as a first-class artifact, the way you would keep a core dump. If you log nothing else, log enough to answer “what exactly did the model see, and what did it do.” A workable template:

Incident: agent took action X on <date>
Inputs at failure: <prompt + user message + retrieved docs, captured verbatim>
Tool outputs in context: <the raw tool responses the model read>
Versions: prompt v?, model ?, tools ?, retrieval index ?
Reproducible? <usually no, say so>
Which surface failed: <prompt / model / tool output / harness>
Owner of that surface: <name from the RACI>
Action item: <change to that surface, with an owner and a date>

Then write the action items against the seam rather than against the model. “The model was wrong” is not an action item, because nobody owns it. “Tool output was trusted as instructions, and the tool owner and the platform team had never agreed who sanitises it” is an action item, because it names who does what next. Whether you can write that second sentence is the real test of whether the agent has an owner at all.

A starting point you can finish this week

None of this requires a platform team or a procurement cycle. If you own an agent that is already in production, here is the order I would do it in:

Write the one-line answer to “who is accountable for this agent’s output.” If you can’t, that is the whole finding.
Fill in the RACI table above. It takes an afternoon and it surfaces the seams immediately.
Inventory what the agent can touch, and which of those tools return data an outsider can write to. That list is your real blast radius.
Pick the wrong-action rate for your one irreversible action class, and start labelling outputs against it.
Stand up golden transactions first; shadow diffing and drift alarms can follow.

Ownership is the cheapest control you have

None of this is exotic. It is the operational hygiene we already apply to every other production service, pointed at a service that happens to think probabilistically and runs on a model you rent rather than own. The only reason it feels new is that we shipped the agents first and asked who owns them second.

So ask it now, before the incident instead of during. Pick the name. Write the RACI. Set the wrong-action budget and decide which signal pages someone. None of those steps needs a new tool or a budget line, which is the part I have found easiest to put off and hardest to justify putting off. They take an afternoon, and they are the difference between an incident with an owner and an incident with an audience.

The agent nobody owns is not the clever one or the fast one. It is the one that hurts you, because when it does, everyone involved will be able to explain, accurately and at length, why it was not their job.

Originally published on Production Grade. I write about operating AI systems in production, the discipline that goes missing between "we built it" and "we run it." Subscribe here for the next one.