I’ve been building an independent Python SDK around the recently published Dream-RSI idea: instead of changing model weights, you record how an agent explores a problem, replay that history offline, let an LLM rewrite the exploration policy, validate the new policy, and then deploy it on future tasks.
The project is called dream-rsi-sdk. It is unofficial and independent from Google and Google DeepMind. My goal is not just to reproduce the basic loop from the paper, but to answer a question that is easy to ignore in self-improving systems: does the improvement actually pay for itself?
That distinction matters because deployment savings can be misleading. I ran an earlier experiment with gpt-6-luna using xhigh reasoning where the same model both solved tasks and developed executable exploration policies. On six held-out tasks, the learned policies reduced deployment from 24 model requests to 6. Look only at deployment and that is a 75% reduction. But the system had already spent 70 requests on training and validation and another 24 on policy development. The full Dream-RSI path was therefore 100 model requests versus 24 ** for the baseline. Mean reported quality was also slightly lower. So the honest result was not “24 → 6*”. It was “24 → 100*”. The deployment policy was cheaper, but the full self-improvement process had not paid back its own preparation cost.
I kept that negative result public because this is exactly the accounting problem I want the SDK to handle properly. If developing a better search policy costs 100 calls and saves one call per task, it may be a terrible idea for ten tasks and a very good idea for ten thousand. You cannot answer that by reporting deployment cost alone.
The next controlled experiment used Ternary-Bonsai-27B-Q2_g64 locally through llama.cpp. In this experiment Bonsai was the real LLM responsible for writing and revising executable exploration-policy code, while the discovery environment itself was a deterministic fixture. Before policy generation I fixed two training tasks, two separate validation tasks, and 64 fresh evaluation tasks.
The policy developer made eight request attempts. Seven returned model responses; one exceeded the configured context window and failed. I still counted that failed request in the budget. The sixth returned response produced the policy that was eventually promoted.
Preparation cost was 24 counted operations: 8 from the two training tasks, 8 developer attempts, and 8 from validation. The fixed policy then required 256 operations across the 64 fresh tasks. The learned policy required 224 deployment operations. Including the full preparation cost, the learned path cost 248 versus 256 for the baseline. Raw quality stayed at 0.9 on all 64 fresh tasks.
So the final result was:
Fixed policy: 256
Dream-RSI preparation: 24
Dream-RSI deployment: 224
Dream-RSI all-in: 248
That is only a 3.125% all-in reduction. It is not a dramatic number, and that is exactly why I like it. The result is much more meaningful than a large deployment-only percentage because the preparation cost is included instead of being hidden outside the chart.
There is also an important amortization effect. The baseline averaged 4.0 operations per fresh task, while the learned policy averaged 3.5. That is a saving of 0.5 operation per task. With a 24-operation upfront preparation cost, the policy crosses break-even after roughly 48 comparable tasks. At 64 tasks, the one-time development cost has already been recovered.
This is where the architecture becomes more interesting than a single benchmark. A useful exploration policy should not have to be recreated from zero every time.
That is why dreamrsi 0.2.0a1 added AdaptivePolicyMemory. The SDK can persist successful policies in SQLite, associate them with a task family, and try them again on related future tasks. The application defines what “related” means through family_of, and it also defines a raw-quality floor. If a remembered policy still performs acceptably, the SDK can keep using it instead of immediately paying for another policy-development campaign. If the policy is missing or degrades on the new task, an improvement pass can be triggered.
In other words, the intended lifecycle is closer to this:
first tasks
↓
collect discovery history
↓
pay for policy development once
↓
validate and store the useful policy
↓
reuse it on related tasks
↓
only retrain when quality degrades or no suitable policy exists
The important word is can. The SDK supports this reuse mechanism, but I am not claiming that a policy will automatically generalize to every new task in the same family. That has to be measured. AdaptivePolicyMemory deliberately checks performance again instead of treating every saved policy as permanently valid.
The memory layer also separates saving and reusing worlds from saving and reusing policies. Applications can decide which historical worlds are allowed into replay, which policy versions are acceptable, and whether a stored policy was a real holdout-validated promotion or merely a successful incumbent. Old policy versions are not silently overwritten.
I also changed replay accounting after finding a subtle problem during earlier experiments. When a replay policy reaches the edge of a recorded discovery tree, there is no historical continuation to reveal. But in a real live run, attempting that expansion could still have cost an agent request. If replay treats that boundary as free, it can create fake savings and promote a policy that would be more expensive in production. The current replay path can conservatively count those attempted expansions so the offline score is harder to game accidentally.
This made some earlier results worse. That was intentional.
The SDK now tries to separate three questions that are often mixed together: did the policy improve replay score, did it preserve raw task quality, and did it actually reduce work? Promotion can use held-out worlds and compare quality and cost separately instead of hiding everything inside one reward number.
The current state of the project is therefore pretty specific. A real local LLM has successfully written, repaired, validated, and deployed executable exploration policies. A controlled fixture experiment has crossed all-in break-even after counting preparation. A separate real-LLM discovery-agent experiment showed strong deployment savings but still lost badly once training and policy-development overhead were included.
So I am not claiming that Dream-RSI is now universally cheaper, or that I reproduced Google’s published benchmark results. I have not yet shown a positive all-in result where the discovery agent itself is an expensive LLM and the full preparation cost is recovered.
That is the next target.
What I want to eventually demonstrate is simple:
training
- + policy development
- + validation
- + failed attempts
- + deployment
- < baseline cost
while preserving task quality, and preferably across multiple task families rather than one controlled fixture.
The SDK itself is model-agnostic. You bring an agent and evaluator; the library handles discovery trees, strict replay, exploration policies, executable LLM policy development, sandboxed policy execution, holdout validation, promotion gates, cost accounting, persistent worlds, and policy reuse. The core requires Python 3.11+ and has zero required third-party runtime dependencies.
Install the current alpha:
pip install dreamrsi==0.2.0a1
Repository: TheAstrayDev/dream-rsi-sdk
Controlled Bonsai Q2 experiment: ternary-bonsai-diverse-2026-09-24.md
Full-model Luna experiment: luna-xhigh-discovery-v1.md
Top comments (0)