AI OpenFree

Posted on Oct 6

How Model-Level RSI Works: Darwin-180B-RSI Learns From Problems It Can Check, Not From Human Answers

#ai #machinelearning #research #llm

TL;DR

Most language models get better because people write more training data for them. Darwin-180B-RSI gets better a different way. It attempts problems whose answers can be checked automatically, keeps only the self-generated solutions that pass the check, and then retrains on that verified set. No human writes the answer key, and no human grades the output. We call this model-level recursive self-improvement, or model-level RSI, to separate it from harness-level self-improvement where only the surrounding scaffolding changes. This article explains the idea at a conceptual level: what the loop is, why verifiability is the whole trick, where it stops, and what it does not claim.

What is model-level RSI in one sentence?

Model-level RSI is a training loop in which the model generates its own candidate solutions, an automatic checker separates the correct ones from the wrong ones, and the model is retrained on the correct ones, so the weights of the model itself improve across rounds.

The important words are "the model itself." There is a related and useful idea where you leave the model fixed and only improve the prompt, the tool list, the retrieval step, or the voting rule around it. That is real and it works, but it is harness-level: the brain does not change, the manual around the brain does. Model-level RSI changes the brain. That distinction matters because the two approaches have different ceilings, different failure modes, and different things you can claim about them.

Why can a model learn from its own answers at all?

On the surface this sounds circular. If a model already knew the right answer, why would retraining on its own output teach it anything new?

The resolution is that solving a problem and recognizing a correct solution are not the same difficulty. For a large class of problems, checking is far easier and far more reliable than producing. A proof can be verified step by step even when it was hard to discover. A program can be run against tests even when writing it took many tries. A numeric answer can be plugged back into the original constraints. When the model samples many attempts, some fraction will be correct even if its first instinct is usually wrong. The checker finds those correct attempts. Retraining then shifts the model so that the behavior which used to appear only occasionally, under lucky sampling, becomes its default behavior.

In short: the model is not grading itself on opinion. An external, deterministic check decides what counts as correct, and only verified work survives into the next round. That is what keeps the loop from simply amplifying the model's own confidence.

What does the loop actually look like?

At press-release level, one round has four stages.

Pose verifiable problems. The loop runs on tasks where correctness can be decided mechanically. Think of domains with an unambiguous right answer: a computation that resolves to a value, a statement that either follows or does not, a program that either passes its tests or fails them.
Generate many candidate solutions. The model attempts each problem repeatedly, producing a spread of candidate answers rather than a single guess.
Keep only what passes the checker. Every candidate is run through the automatic verifier. Solutions that pass are retained. Solutions that fail are discarded. Crucially, failed attempts are not relabeled or rationalized into the training set.
Retrain on the verified set, then repeat. The surviving, proven-correct solutions become the new training signal. After retraining, the improved model poses the next round, and the cycle continues.

Each pass ratchets the model toward solutions it can prove, using problems it can check. There is no point in the loop where a human supplies the correct answer.

pose verifiable problem
        |
        v
model generates many candidate solutions
        |
        v
automatic checker: pass or fail?
        |-- fail --> discard (never relabeled)
        |
       pass
        |
        v
retrain model on verified-correct solutions
        |
        v
improved model poses the next round  --> (repeat)

How is this different from ordinary fine-tuning?

Ordinary supervised fine-tuning needs a dataset that people have already written and labeled. The quality ceiling is set by the humans who produced it, and scaling it up means paying for more human labeling. Model-level RSI moves the bottleneck. Instead of "how much labeled data can we buy," the question becomes "how many problems can we state whose answers a machine can check." Where such problems are plentiful, the training signal can grow without a human writing each answer.

It is also different from naive self-training, where a model's own outputs are fed back in wholesale. Naive self-training tends to reinforce whatever the model already does, including its mistakes, because nothing filters the output by correctness. The verifier is exactly that missing filter. Remove it and the loop degrades. Keep it honest and the loop has a reason to climb.

Why does verifiability do all the heavy lifting?

Because the verifier is the only thing standing between "self-improvement" and "self-delusion."

If the check is sound, the loop can only promote behavior that genuinely satisfies the problem. If the check is weak, gameable, or misaligned with what you actually care about, the model will happily learn to satisfy the check rather than the intent behind it. This is the single most important engineering constraint in the whole approach: the loop is only as trustworthy as its checker. That is why model-level RSI is a good fit for domains with clean, hard-to-fake correctness conditions, and a poor fit for tasks where "correct" is a matter of taste or where the only judge is another model's unverified opinion.

It also explains why the gains do not transfer for free. A model trained this way gets strong at the kinds of problems its checkers cover. It does not magically become better at everything. Honest reporting of model-level RSI means being specific about which verifiable domains drove the improvement.

Where does the loop stop?

A self-improvement loop that never stopped would be a remarkable claim, so it is worth being blunt about the limits.

The loop slows and plateaus when the model stops producing new correct solutions the checker has not already rewarded. If every problem in the pool is one the model already solves reliably, there is nothing left to promote. Progress depends on there being a frontier: problems the model solves sometimes but not always. As that frontier gets consumed, each round yields less. Pushing further requires harder verifiable problems, not just more rounds of the same ones. The loop is a way to climb a hill efficiently, not a perpetual motion machine.

What model-level RSI does not claim

To keep expectations grounded:

It is not a model that trains with zero data of any kind. It is a model that trains without human-written answer keys inside the loop. The problems and the checkers still have to exist.
It is not general self-awareness or open-ended autonomy. It is a specific, bounded training procedure.
It is not a guarantee of correctness on unverifiable tasks. Strength is concentrated where the checks are.
It is not a substitute for careful evaluation. The results still have to be measured on independent benchmarks that the loop did not train against.

FAQ

Is "model-level RSI" the same as the recursive self-improvement people worry about in AI safety discussions?
No. The safety-discussion version usually imagines an unbounded, open-ended agent rewriting itself without limit. Model-level RSI here is a concrete, bounded training loop: fixed problem types, an external checker, retraining, and a plateau. It improves measured skills on verifiable problems; it does not grant autonomy.

How is this different from harness-level self-improvement?
Harness-level self-improvement leaves the model's weights fixed and improves the scaffolding around it: the prompt, the tools, the retrieval, the voting. Model-level RSI changes the weights of the model itself. One improves the manual, the other improves the brain. Both are legitimate; they just make different claims.

Could the model cheat by learning to fool its own checker?
That is the central risk, and it is why the checker has to be sound and hard to game. If the check can be satisfied without actually solving the problem, the loop will find that shortcut. Model-level RSI is therefore only as good as the verification it rests on, which is why it suits domains with clean, mechanical correctness conditions.

If no human provides answers, where does the ground truth come from?
From verifiability, not from an answer key. The correctness of a solution is decided by an automatic check of the solution against the problem's own constraints. Humans define the problem space and the checking rules, but no human marks individual outputs right or wrong.

Does this work for every kind of task?
No. It works best where correctness can be decided mechanically and cheaply. Open-ended writing, matters of taste, and tasks whose only judge is unverified opinion are poor fits, because there is no trustworthy signal to filter on.

Does the model keep improving forever?
No. It plateaus once it has learned the correct solutions its checkers can surface. Going further needs harder verifiable problems, not simply more rounds.

DEV Community