Synthetic data and the self-improvement loop: a model trains on its own output to improve — or collapses without a filter

You've run out of human data. So the model makes its own, grades it, and retrains on it. Does that actually work? I built two browser demos that answer both sides of the question, and the whole thing hinges on one word: filter. With a verifier in the loop, quality climbs each round. Without it, the model slowly eats its own tail until it collapses to a single note. Same engine, opposite outcomes.

The asymmetry that makes it possible

For many tasks, verifying an answer is far cheaper than generating one — checking a proof, running unit tests, matching a known answer key. That asymmetry is the whole opening. A mediocre model can spray out many candidates, a cheap verifier keeps only the good ones, and the model retrains on that cleaner-than-average set, getting a little better each round. That's STaR (Self-Taught Reasoner) / self-training.

Step 1: generate many candidates

Data is the bottleneck, so make your own. For each prompt, sample several answers with temperature > 0 so they differ. One try might be wrong; among k tries, some are usually right.

def generate(model, prompts, k=8, temp=0.8):
    pool = []
    for p in prompts:
        for _ in range(k):                  # k diverse attempts each
            pool.append((p, model.sample(p, temperature=temp)))
    return pool                             # many candidates, quality varies

Step 2: verify and filter — the step that makes it all work

The whole idea rests on verification. Prefer a ground-truth checker (unit tests, an answer key, a compiler) over an LLM judge, and an LLM judge over nothing. Keep only what passes — the kept set is purer than the model's average output.

def keep_good(pool, verify):
    return [(p, a) for (p, a) in pool if verify(p, a)]
# verify() is the filter. ground-truth > LLM-judge > nothing.
# no verify() -> you retrain on your own average -> NO improvement.

Step 3 and the loop: retrain, repeat, quality ticks up

Fine-tune the model on its own verified-correct answers — including the reasoning that led there — then iterate. As the model improves, more of its candidates pass the filter, so the kept set grows and improves: a virtuous cycle that plateaus when the model saturates what the verifier can confirm.

def self_improve(model, prompts, verify, rounds=5):
    for r in range(rounds):
        pool  = generate(model, prompts, k=8)
        kept  = keep_good(pool, verify)          # <-- the filter is everything
        model = retrain(model, kept)
    return model
# filter ON -> quality climbs.   filter OFF -> it flat-lines.

In the first demo, a filter slider drives this directly. At 70% the model's quality bars step up round over round. Drag the filter to 0% — keep everything — and the line flat-lines, because the training set is exactly as good as the model already is (purity = quality). Garbage in, garbage out.

The mirror image: model collapse

Now remove the verifier and train generation after generation on unfiltered output. Sampling over-weights the peak of a distribution and drops the rare tails, so each generation is a little narrower than the last. The second demo shows a healthy 9-mode distribution collapsing toward a single spike as diversity leaks away — model autophagy, sometimes called MAD.

# sampling sharpens: peak grows, tails fade, generation after generation
def sharpen(p, alpha=1.6):
    return normalize([x**alpha for x in p])

The defence is to keep real data in the loop, which re-anchors the tails:

# NEVER train purely on your own unfiltered output.
train_set = mix(real_data, synthetic_kept, real_frac=0.3)   # anchor tails
model     = finetune(model, train_set)
div = effective_modes(model.sample_many())
assert div > MIN_DIVERSITY, "distribution collapsing — inject real data!"

Slide even a little real data back in each generation and the distribution stays healthy.

Distillation is the same family

Same "generate data to train" idea, different roles: a strong teacher produces high-quality answers (still verified and filtered), and a small cheap student trains on them. You bottle a big model's skill into one you can actually afford to serve. Self-improvement is teacher == student; distillation is teacher > student.

The lesson from both demos: synthetic data is only as good as the filter in front of it, and you must keep real data in the loop. Verify hard, mix in reality, cap the self-training generations, and watch a diversity tripwire — and the loop compounds instead of rotting.

Run both loops — improvement and collapse — in your browser:
https://dev48v.infy.uk/ai/days/day39-synthetic-data.html