A paper titled Language Models Need Sleep landed a few weeks ago. The claim is direct: a model that consolidates its short-term context into its weights during a "sleep" phase, then dreams synthetic samples to rehearse, learns new information better than a model that just sees it in context, and forgets less than one that is plainly fine-tuned on it.
The results in the paper are large. Class-incremental learning gains over ICL and SFT. Near-zero forgetting on continual translation. Long-context performance on BABILong holding up to 10 million tokens.
I do not have a cluster. I have an M4 Pro Mac mini with 64 GB of unified memory, and I had a weekend. So the question I actually asked was narrower than "is the paper right":
If I build the cheapest faithful version of consolidation plus filtered dreams, does it beat a fair SFT baseline on hardware I own, with 7B parameters and a LoRA budget?
The answer is yes at 400 iterations, 37.3% against 11.9% across three seeds. The more useful answer is what the same table says about the rest of the claim, which is that in-context learning wins outright and none of the paper's headline mechanisms were tested at all.
Here is exactly what I built, the numbers, and the places where I think this kind of replication goes wrong.
First, what the paper actually does
Reading the method section matters more than reading the abstract, because the gap between them is where most "implementations" quietly overclaim.
The sleep phase has three moves:
- Consolidate. The model rewrites accumulated context into compact self-contained facts, then updates its own weights on that distilled form.
- Dream. It generates synthetic samples that combine pieces of knowledge from different contexts, then trains on them. These are not restatements of what it saw, they are combinations it did not see.
- Filter. Not every dream is useful. Samples that carry no learning signal get dropped.
Architecturally the paper does this with unlockable low-rank experts inside a mixture-of-experts model, trained with generalized knowledge distillation and reinforcement learning, over contexts up to 10 million tokens.
I can do exactly one of those things honestly: the filter. Everything else has a cheaper stand-in, and the stand-ins are the reason this post has a limitations section instead of a benchmark table.
Here is the mapping I committed to before writing any code:
| Paper mechanism | What I built | What it cannot show |
|---|---|---|
| Custom MoE with unlockable experts | LoRA on a dense 7B | Capacity growth, expert routing |
| GKD and RL from trajectories | SFT on teacher-written targets | Anything about reward or policy |
| 10M-token training context | 768-token sequences | Every long-context claim |
| Model consolidates its own memory | Hosted teacher writes the targets | Self-consolidation |
Writing that table first is the difference between a test and a demo. If the delta I measure cannot distinguish the mechanism from my substitute, the result is not evidence about the paper.
The pipeline
flowchart LR
A["6 synthetic sessions<br/>90 invented facts"] -->|teacher: consolidate| B["97 atomic facts"]
A -->|teacher: recombine facts<br/>from different sessions| C["96 dream questions"]
C -->|student answers,<br/>keep only failures| D["40 kept dreams"]
A -->|teacher: verbatim QA<br/>baseline only| E["96 raw QA pairs"]
B --> F["sleep arm<br/>93 rows"]
D --> F
E --> G["sft_raw arm<br/>93 rows"]
A --> H["raw arm<br/>6 prose sessions"]
F --> I["LoRA, 400 iters,<br/>identical budget"]
G --> I
H --> I
I --> J["evaluate.py<br/>42 held-out + 14 general"]Two stages, two machines in my case. The teacher stage needs an API key. The student stage needs none, which is why the two are separate scripts and the teacher output is committed as JSON. My training box never held a credential.
The corpus has to have a floor
This is the step people skip, and it invalidates everything downstream.
If the model can answer your questions from pretraining, you are measuring how much fine-tuning destroys or preserves prior knowledge, not whether it learned anything new. So every fact is invented. Six fictional services, each with its own cluster name, region, freeze window, rotation interval, on-call length, memory cap, and release train.
SUBS = ["ingest", "billing", "search", "growth", "sre", "portal"]
CLUSTERS = ["quartz-dw", "flint-07", "marl-2", "basalt-3", "onyx-5", "shale-11"]
REGIONS = ["eu-central-1", "us-east-2", "ap-south-1", "eu-west-2", "us-west-1", "ap-northeast-1"]
The second design choice matters more than the first: all six services share the same ten fact templates with different values. The task is therefore not "remember a fact", it is "bind the right value to the right service". One fact per topic measures nothing; six facts that interfere measure the thing that actually breaks fine-tuning.
I verified the floor before training anything:
Q: How often is the CI token rotated?
base model: "Every 12 months, with a 30-day overlap window."
truth: 45 days
A confident, fluent, wrong answer. That is exactly what I wanted, and it is worth recording in the write-up, because it proves any later correct answer came from the data.
Held-out evaluation is in three tiers: 36 direct single-fact questions, 6 two-hop questions that require joining two sessions, and 14 general-knowledge probes whose only job is to detect forgetting.
Consolidation and dreaming, done by the teacher
The teacher gets all six transcripts and is asked for atomic, self-contained facts, and then for two-hop questions that combine facts from different sessions. 90 raw facts become 97 atomic ones. The recombination step produces 96 candidate dream questions.
Then the part I think is the most faithful bit of the whole build.
The filter runs on the student
The paper drops dreams that carry no learning signal. A gradient would be the direct measure. I do not have one for a 7B with a 400-iteration budget, so I used the cheapest correct proxy: ask the student, keep what it gets wrong.
kept = []
for d in drafts:
got = norm(student(d["question"]))
if key_of(d["answer"]) not in got:
kept.append(d)
rng.shuffle(kept)
kept = kept[:args.cap]
print(f"student failed {len(kept)} usable / {len(drafts)} drafted dreams (cap {args.cap})")
40 of 96 survive. That number is itself a diagnostic: a keep-rate near zero means the dreams are too easy, near 100 means they are unusable, and either way the method is not doing what you think.
Notice this step has to run on the training machine, because it needs the student weights. That is a real constraint on any faithful reimplementation, and it is why the teacher and student stages are separate executables rather than one notebook.
The baseline is where most replications cheat
Raw-prose LoRA is not a fair SFT baseline. It teaches the model the text, not the task, and it will lose for reasons that have nothing to do with the method under test. I ran it anyway (raw, 19%) because it is the thing everyone tries first, but the comparison that matters is sft_raw:
- Same supervision format: explicit question and answer pairs.
- Same example count: 93 rows in both arms.
- Same hyperparameters: iters, rank, layers, LR, sequence length.
- Generated from the same transcripts, with no consolidation and no dreaming.
The only difference between the two arms is the recipe. That is a structural guarantee, not a discipline I promise to keep:
ARMS="${ARMS:-raw sleep}" MODEL="${MODEL:-mlx-community/Qwen2.5-7B-Instruct-4bit}" ITERS="${ITERS:-400}" ./train.sh
train.sh loops over the arms with identical flags for each. If I had wanted to fudge the result I would have to edit the loop, which is exactly the kind of edit a reader can spot in a diff.
Scoring, without a judge
For factual recall on invented facts, an LLM judge is a source of noise and an unaccounted token cost. Must-contain substring matching against hand-written keys is deterministic, free, and lets anyone re-check a row.
def squash(s: str) -> str:
return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", " ", s.lower())).strip()
def hit(ans: str, must: list[str]) -> bool:
a = squash(ans)
return any(squash(m) in a for m in must)
Every generated answer is written into the results JSON next to its key and its verdict, which is how I can say "the baseline binds the value to the wrong service" instead of guessing why it scored 12%.
Six conditions run in the same harness. Three need no training, and they are the ceiling the method has to beat before you can call it needed:
condition all direct derived general ctx tok
------------------------------------------------------
base 0% 0% 0% 100% 0
icl_all 100% 100% 100% 100% 1032
icl_recent 33% 33% 33% 100% 344
raw 19% 17% 33% 93% 0
sft_raw 12% 14% 0% 64% 0
sleep 45% 39% 83% 93% 0
The result, and how I broke it
One seed said sleep 45% against sft_raw 12%. That is a demo, not a result. So the seed was parameterized and three more were run, each rebuilding both arms from a fresh dream sample and retraining both at identical budget.
| condition | n | all | direct | 2-hop | general |
|---|---|---|---|---|---|
sft_raw |
3 | 11.9 ± 0.0 | 13.0 ± 1.6 | 5.6 ± 9.6 | 76.2 ± 8.2 |
sleep |
3 | 37.3 ± 7.7 | 33.3 ± 7.3 | 61.1 ± 19.2 | 73.8 ± 8.2 |
Three things worth saying about this table.
The baseline is identical to the decimal, three times. 11.9% in every seed, which is 5 of 42 questions. That is not stability, it is a systematic failure: it learns the values and attaches them to the wrong services, so it gets the same wrong answers in the same places regardless of what data it saw. Failure that consistent is more damning than failure that varies.
The treatment varies a lot and never comes close to overlapping. 28.6, 40.5, 42.9. With 42 questions, one question is 2.4 points, so a 5-point delta would be noise. A 25-point gap is not.
Two-hop is where the mechanism shows. 61% against 5.6%. Dreams that combine facts from different sessions are the only supervision in the study that trains a join, so this is the one number I would expect to hold up if someone rebuilt the whole thing.
And the correction: my first write-up said sleep "forgot less", 93% against 64% general retention. Across seeds that difference disappears, 73.8% against 76.2%. It was one seed's artifact. The README now says retention is a wash, because deleting a claim that replication demoted to a tie is the whole point of running the seeds.
The budget is part of the comparison
At 150 iterations the two arms are indistinguishable.
sft_raw 12% (val loss 0.286, train 0.441)
sleep 14% (val loss 0.781, train 0.525)
Both are underfit at that budget. The gap appears at 400 because the sleep data is still being absorbed while the verbatim-QA arm has already saturated into its wrong answers. Both tables are in the repo. Reporting only the 400-iteration table would have made a hyperparameter look like a finding.
The agent-level proxy, and why it lost
Half of this project is a separate artifact: the same loop implemented at the memory layer with LangChain, no gradients, to see what consolidation and dreaming are worth when you cannot touch weights.
It has a two-tier store, an arbiter that decides whether a new fact supersedes an old one or coexists with it, a dreaming step that recombines stored memories, and pruning that deletes raw transcripts after consolidation.
Against a plain chunk-retrieval baseline:
| condition | all | direct | derived | ctx tokens/query |
|---|---|---|---|---|
no_memory |
0% | 0% | 0% | 0 |
transcript_rag |
95.5% | 100% | 83.3% | 227 |
chunk_rag |
100% | 100% | 100% | 77 |
sleep |
90.9% | 100% | 66.7% | 361 |
The sleep loop uses 4.7x the context of chunk RAG and still loses 9 points. Honest conclusion: at the agent level, a good retriever plus a valid_from column dominates the elaborate loop.
It does win one thing. When facts are revised over time, the arbiter retires superseded nodes and correctness goes to 100% against 85.7% for retrieval. That is a real interference result, and it is also the cheapest thing here to reimplement in your own product: version your facts, do not just append them.
What this does not test
The paper is not one claim, it is five. I tested one and a half.
- Long-context scaling. Nothing here touches 10M tokens. The corpus fits in 768 tokens.
- Capacity growth. LoRA rank is not an unlockable MoE expert. The paper's argument that consolidation needs capacity is untested by construction.
- Self-consolidation. My teacher is a hosted frontier model, not the student. That makes the treatment arm better fed than the paper's, so a win here is weaker evidence than theirs, and a loss would have been damning.
- GKD and RL. Pure SFT on teacher targets. No reward, no policy.
- Generality. One model family, one corpus, 42 questions, one machine.
So: 37.3% against 11.9% is evidence that consolidation plus student-filtered dreams beats a fair SFT baseline at 7B on a Mac mini. It is not evidence that Google's sleep mechanism works as described, and the repo, the README, and the title of this post all say so. The honest framing is "I tried to test the claim and here is what held", not "an implementation of the sleep framework".
Running it yourself
Everything is on GitHub. A 0.5B smoke run reproduces the ordering in about 40 seconds per arm on an 8 GB machine, which is enough to check the harness before spending an hour on the 7B:
git clone https://github.com/<you>/sleep-replication
cd sleep-replication/slm-test
python3 -m venv .venv && . .venv/bin/activate && pip install mlx-lm
python make_corpus.py
python assemble.py --model mlx-community/Qwen2.5-0.5B-Instruct-4bit
MODEL=mlx-community/Qwen2.5-0.5B-Instruct-4bit ITERS=150 ./train.sh
python evaluate.py --model mlx-community/Qwen2.5-0.5B-Instruct-4bit --tag 0.5b
The committed JSON means you can rebuild the arms and retrain without ever calling the teacher API. That was deliberate: the expensive, credentialed, non-deterministic step happens once, and everything downstream of it is free and re-runnable.
Three 7B seeds in parallel take 33 minutes on an M4 Pro. Sequentially they take about 19 minutes each, so the parallel win is 1.7x rather than 3x, because the GPU is the shared resource. Each run peaks at 5.2 GB, so memory is not the constraint.
What I would build next
The most interesting missing arm is sleep without dreams: consolidated facts only, no synthetic rows. That splits the 25-point gain into "consolidation helps" and "the filter helps", and right now I cannot say which is doing the work.
After that, a proper two-hop generator, and a retrieval baseline at the same token budget as icl_all, so the ceiling is a tuned one instead of a brute-force paste.
Both are listed as starter issues in the repo. If you fork it, the rule is simple: the results JSON goes in the PR with any change that moves a number.
*Code & more: *https://www.dailybuild.xyz/project/299-sleep-replication
Top comments (0)