Model collapse: what happens when AI trains on AI, generation after generation — and the one-line fix

For years, models trained on human-made text and images. But the web is now filling with AI-generated content, and the next models scrape that web — so increasingly a model's training data was written by an earlier model. That raises a sharp question: if you keep training models on the output of previous models, generation after generation, what happens to what they know? The answer is model collapse, and I built a demo that runs the whole thing live — a rich "real world" distribution, a simple fit-and-resample model, and the recursive loop that grinds it down to a bland blob. Here's the mechanism.

The recursive loop, made precise

Strip it to essentials. Generation 0 is a finite sample of real data. To make generation t+1, you fit a model to generation t, then sample fresh data from that model — and that synthetic data becomes the next round's training set. The real data is never looked at again:

let sample = Array.from({length:n}, sampleReal);   // gen 0: real data
for (let g = 1; g <= generations; g++){
  const model = fit(sample);        // train on what we currently have
  sample = generate(model, n);      // its output BECOMES the next training set
}

My "real world" is a mixture with two big populations, a rare subgroup, and a heavy tail — genuine structure for a collapse to destroy. The "model" is deliberately simple: estimate a mean and variance, then draw from that single Gaussian.

Three errors that compound

Collapse isn't one bug; it's three, reinforcing each other every round.

Finite samples miss the rare cases. Any finite sample is a lottery, and rare events almost never win. A 3% subgroup sampled 30 times comes up empty ~40% of the time. Fit a model to that sample and it gives those cases ~zero probability, so they can't reappear next round. The tails die first.

A simple model smears structure toward the average. Fit a single Gaussian to two-humped data and it can only produce one hump, centred on the average of the two — a bland middle that matches neither mode.

The spread shrinks a little every generation. This is the quiet killer. Draw n points from a distribution of variance V and the expected ML-estimated variance is (n-1)/n · V — a hair smaller. Feed it back as the new truth and it compounds:

// var_after ≈ (n-1)/n * var_before        (compounds every round)
// var_t     ≈ ((n-1)/n)^t * var_0   ->  0  (collapse)
// smaller n -> factor further below 1 -> collapses FASTER

After t rounds the spread is ((n-1)/n)ᵗ of the original, heading to zero. And there's no correction signal: in normal training real data keeps the model honest, but in the loop each generation's errors become the next generation's ground truth. The demo's two gauges tell the story — rare-case survival hits zero well before the spread finishes narrowing.

The fix: keep real data in the mix

The defence is simple and provable: don't train purely on your own output. Replace a fraction p of every generation with fresh real samples, and those genuine points re-inject the tails and the spread each round, cancelling the compounding shrink:

function nextGen(sample, n, p){       // p = fraction of REAL data mixed in
  const model = fit(sample);
  const kReal = Math.round(p * n), kSyn = n - kReal;
  const out = generate(model, kSyn);              // (1-p) from the model
  for (let i=0;i<kReal;i++) out.push(sampleReal()); // p from the REAL world
  return out;
}
// p = 0   -> spread -> 0     (collapse)
// p = 0.2 -> spread plateaus (it keeps remembering the real world)

In the demo, two recursions run from the same start: the pure synthetic loop slides toward zero, while the one with even ~20% real data mixed in plateaus at a stable floor. Same recursion — one forgets the world, one remembers it.

This isn't just a toy. As synthetic content floods the internet, naively scraping the web risks training on your predecessors' output, degrading quality and diversity over model generations — which is why verified human data is becoming precious (some compare pre-2023 data to "low-background" steel, clean because it predates the contamination). It also amplifies bias: whoever is under-represented today gets erased faster tomorrow. The practical rule: keep real data in the mix, filter and verify synthetic data, and accumulate rather than replace.

Watch a rich world collapse, then flip on "mix in real data":
https://dev48v.infy.uk/ai/days/day44-model-collapse.html