The model gets one function: press(button). I wanted to see how far that goes.
There are a lot of demos of language models playing Pokémon. Most of them hand the model high-level functions like "fight this battle" or "walk to the next gym," and the model calls them in order. That works, and it is a fair way to build a product, but it mostly tests the person who wrote the functions. I wanted to test the model. So in my Pokémon agent the model has one function, press(button), and nothing else. There is no macro for a battle turn, a menu, or a route. If the character walks across Kanto, every step was a press the model asked for.
Two models, plus helpers
One model can't do this alone at the sizes I can run at home. The 7B vision model (qwen2.5vl:7b) is fine at reading a frame and bad at planning. The 14B and 27B text models (qwen2.5:14b, qwen3.6:27b) plan reasonably and can't see pixels. So the vision model's job is to turn the frame into a short structured description: what kind of screen this is, any dialogue text, where the menu cursor is, HP, move PP. The text model reads that description and replies with a button and a sentence saying why.
Around those two there are a few helper agents that read the emulator's memory for things a screenshot doesn't show. One computes a route through the current map, because the barriers are drawn at load time and the vision model can't reliably tell a ledge from a path. One derives a level cap from the next major trainer's party data. One scores catches and party composition. One checks whether the wild Pokémon on screen is shiny, which is a 1 in 8192 event you would otherwise miss.
All of that is delivered as text in the observation, and the deciding model can ignore it. The helpers never press anything. I audited a run to see where presses came from and about 99% were the deciding model. The rest were bookkeeping the game forces on you, like advancing a text box that repeats.
The rule I kept wanting to break
When the model walks into the same wall twelve times in a row, the obvious fix is to let the pathfinder take the controller for a few steps. I did that more than once. Each time the run went better and the experiment got less interesting, because now I was measuring my pathfinder. I ended up with a firm line: pathfinding, battle state, and level caps are advice in the prompt. Move choice, switching, healing, and where to walk are the model's.
Keeping the harness dumb had a side benefit. Nothing in it knows which cartridge is loaded, so the same code runs FireRed and Emerald at the same time. I run three instances, and they relaunch themselves if a run gets stuck.
Logging
A run takes days. Nobody is going to sit and watch it. Every line of dialogue the game prints is logged, and every decision writes one record: what the model was shown, what it answered, which buttons fired, and the game state before and after. There is a browser panel that streams the same data live, but I mostly use the trace afterwards to figure out why a run stalled.
Results so far
It has beaten Brock from the start of the game, and it has beaten Lorelei of the Elite Four with all six party members still standing. Before that Lorelei win there was a loss where the 14B model stayed in a bad matchup too long and lost a Pokémon it should have switched out. Swapping to the 27B model fixed it. I didn't change the harness. That is the result I was hoping for: when it fails, it fails because the model isn't deep enough, and a better model fixes it.
Cost
Python, mGBA, and Ollama, all on one machine with my own GPU. No API key and no per-token bill; the running cost is electricity. This matters more than it sounds. A decision per frame over a multi-day run would be expensive against a hosted API, and I would have been tempted to add macros just to cut the bill.
Originally published at jackhomer.com/writing/pokemon-one-button/ by Jack Homer.
Top comments (0)