An agent watches a game, learns to hallucinate the next frame, then plays inside its own dream — but only the model that knows players react to each other stays true.
TL;DR: The hottest idea in agents right now: don't feed them the real world — let them dream it. An agent watches some footage, learns to hallucinate the next frame, and practices inside its own head. I built a tiny one, and found the catch: a dream only stays true if it knows the players react to each other. Runs on a laptop.
The world
Two players in an 11-cell corridor: a predator steps toward the prey, the prey steps away. Every move is a reaction. An agent watches random games, then closes its eyes and dreams 15 frames ahead, feeding each prediction back in as the next input. I built two dreamers from the exact same footage:
- single-player — predicts each player from its own position alone
- multiplayer — predicts both positions together
Training an agent inside its own learned dream goes back to Ha & Schmidhuber's World Models (2018); the open frontier is making that dream multiplayer — modeling agents reacting to each other, not just physics.
The 10-second version
% of the dream still matching reality, this many frames ahead:
| frames ahead | 1 | 3 | 5 | 10 | 15 |
|---|---|---|---|---|---|
| single-player dream | 11 | 0 | 0 | 9 | 0 |
| multiplayer dream | 100 | 100 | 100 | 100 | 100 |
The single-player dream falls apart almost immediately; the multiplayer one stays locked to reality. That gap is the whole point — and, as we'll see, it comes down to what each model is even able to represent.
How it works
The real world is one tiny rule — predator steps toward prey, prey steps away:
def real_next(p, q): # p = predator cell, q = prey cell
return clamp(p + sign(q - p)), clamp(q - sign(q - p))
The agent never sees that rule. It just watches random games and counts what follows what. The only design choice is the shape of the memory it keeps — and that's the whole story:
multi[(p, q)][(p2, q2)] += 1 # multiplayer: "when BOTH are here, both go there"
one_p[p][p2] += 1 # single-player: "when the predator is here, it goes there"
one_q[q][q2] += 1 # single-player: "when the prey is here, it goes there"
To dream, it rolls its own predictions forward — feeding each guessed frame back in as the next input — and we check how long the dream keeps matching reality:
real = dream = start
for _ in range(15):
real = real_next(*real) # what actually happens
dream = model(*dream) # the model predicting on its OWN last frame
match += (dream == real)
Same footage, same loop. The only difference is whether the model remembered the two players together or apart.
Why single-player collapses
The prey moves because the predator moved. A single-player model looks at each player in isolation, so it structurally can't represent that reaction — its errors compound each frame until the dream is fiction. The multiplayer model conditions on both, so it captures the coupling. To be fair, this isn't a surprising empirical result so much as a demonstration: the corridor is deterministic, so the outcome really follows from what each model is allowed to see. That's exactly why the framing matters.
A dream you can act in is a superpower — an agent can practice a thousand risky moves for free. But a dream that forgets everyone else reacts to you isn't practice. It's a delusion.
Why it's exciting
The proven part: training agents inside a learned dream works — from World Models (2018) to DreamerV3 (2023) mastering 150+ tasks, and Genie (2024) learning playable worlds from video alone. Dreamed worlds let agents rehearse infinitely, safely, at zero real-world cost.
Where it's heading: those dreams are mostly single-agent today. The 2026 push is multiplayer — worlds where agents model each other. The lesson from this demo is the whole ballgame: model the reactions or the dream drifts. Get it right and agents can plan against each other entirely in imagination.
How faithful is this?
A real world model learns from raw pixels with a neural net, in a noisy, stochastic world. This is that mechanism stripped to its core — a frequency table over a tiny, deterministic game. It's built to make the intuition concrete, not to reproduce the papers; those (linked below) do the heavy, learned version.
Try it
git clone https://github.com/Shridhar-2205/secret-lives-of-agents
cd secret-lives-of-agents/03-dreamed-world && python demo.py
The series — The Secret Lives of AI Agents
- Agents invent their own language
- Agents build a culture on a decaying notepad
- Agents that live inside dreamed-up worlds (you're here)
Shridhar Shah — Senior Software Engineer on the AI team at Cisco. GitHub · LinkedIn
Sources & further reading: Ha & Schmidhuber, World Models (2018) — the "train inside a dream" idea · Hafner et al., Mastering Diverse Domains through World Models (DreamerV3, 2023) · Bruce et al., Genie: Generative Interactive Environments (2024).
Top comments (1)
This is a really clean demo of something people underestimate: the environment is not just physics, it is other actors reacting back.
The line "model the reactions or the dream drifts" feels like the core lesson. A single-agent world model can look stable in isolation, then fall apart the moment another policy starts adapting. That probably applies beyond games too: agent workflows, public chat, negotiation, code review, support loops, anywhere the next state depends on another mind or model.
I would love to see this idea pushed into messier social simulations where agents are not only predicting positions, but tone, trust, incentives, and when to stop talking.