Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.
The first three combats in StS are drawn from a special pool of "easy encounters". Of those, the most straightforward is the Cultist: it starts with between 48 and 54 HP (meaning it takes 8 or 9 plays of "strike" on it to kill it). On its first turn, it performs a ritual, dealing no damage to the player. On every turn after, it attacks for an increasing amount of damage.
The cultist is there to teach human players the concept of a damage race: that it's sometimes better to take a little amount of damage now to save a large amount of damage later. The cultist's first attack is for 6 damage, just over the 5 block from a "defend" card. A player holding two defend cards and three strike cards would need to decide whether it's better to deal 18 and take 6, deal 12 and take 1, or deal 6 and take 0. The ground truth would come from calculating the expected value of remaining HP after the fight's end, taking into account draw order variance. That's the only unknown, since the cultist itself is perfectly predictable.
To test whether my agent can learn to do any spire-slaying well, I pitted it against the cultist, over and over again. Specifically, for 200,000 training steps, which is about 14,000 fights. The network would output a probability for every play as well as a prediction for how well it'll do. The agent sampled from the distribution of plays, and recorded how close the predictions were to the reality of how well it did. Rinse and repeat.
The agent did terribly: after this extensive training, it reached a 70% win-rate. That means 30% of the time it died to the cultist, showing it didn't have the wherewithal to learn a simple strategy like "always play attacks in hand".
That's because it had no idea what its hand contained.
In part 2, I briefly mentioned mean-pooling being performed on the hand encoding. The agent simply assigned the same probability to every card in hand, and what we learned was that if you randomly play 3 energy's worth of cards every turn, the cultist will die about 70% of the time.
Giving the agent the ability to know which card it's playing, by scoring every card in hand individually against the game state using a new head made the agent reach 100% win-rate after only 10,000 steps.
Extending the experiment to also include the other possible early encounters showed a similar result - the agent quickly learned to win its first combat with a starter deck.
(Technical note: the exact training regime was PPO with a learning rate 3e-4, 1,024 steps per batch, minibatches of 256, 10 epochs per batch, gamma 0.99, GAE lambda 0.95, clip range 0.2, entropy coefficient 0.15)
The next order of business was minimizing HP loss in the fight. On the lowest difficulty I expect to beat the cultist taking no damage and taking 0-4 for the other encounters. The closer the agent can get to that, the more confident I'd be in its ability to teach itself tactical principles with self-play.
The starting point agent concluded fights with about 47 HP left, meaning it took 23 damage on average. We set out to give it the tools it would need to make the right judgment calls about its level of aggression.
Peeking inside its log of plays, the agent was averse to blocking. This wasn't about trade-offs. Having played all possible attacks and with leftover defend cards and energy, the agent would end its turn rather than defending.
Why would an AI agent teach itself that blocking isn't a good idea when it's being penalized for losing HP in the fight? One good reason would be that defend cards didn't really do anything. Contrary to my idealized view of game design, the logic that made block cards apply block was sitting in a visual effect function, one that was silently skipped. We fixed that, added some safeguards to ensure there would be no regression, retrained the agent, and it refused to block, again.
With "a functional bug" no longer an explanation, we began by experimenting with rewards. The agent was rewarded for winning the fight and for preserving HP. That sounds fine on the face of it, but the rewards for blocking are delayed. You play a defend on turn 1, win on turn 4, and need to somehow figure out the 5 extra HP preserved are attributable to that and not the other 16 or so moves you made. We gave some immediate reward for blocking incoming damage, added a feature to avoid over-blocking, and decreased the "win" reward magnitude, all in an attempt to avoid drowning out the gradients showing "block is good".
Those experiments helped a bit, pushing average HP at fight end to 55, but there it stopped. So, I wrote some simple combat heuristics to bootstrap the agent. Just telling the agent to always block with leftover energy pushed average HP up to 63 while not hurting the win-rate. The problem was training from there actively degraded the agent's performance back to the 55 HP area (more interestingly, it collapsed to 34-ish, then climbed back up to 55 during further training). Trying to force the learned behavior to not stray too far from the heuristic (using something called KL regularization) either did nothing (when the constraint was tight) or showed the same degradation (when the constraint was looser).
Other experiments raised the "defend rate" but not the quality of blocking, meaning the agent actually blocked more when it wasn't attacked. Throwing in attention was not all we needed - it raised the HP preserved by about 4.4, but by blocking less and simply attacking more efficiently.
Paradoxically, the agent kept learning to block less when being attacked. The technical term is intent gap: the agent was slightly more likely (0.2%) to play a defend when the enemy wasn't attacking than when it was. Everything we did to improve its quality of play raised its propensity to attack, rather than block. So, we settled on the exploration trap: to learn that defending is good, the agent needs to try it out and see that it helps. And to do that, it needs to give a high probability to defending, meaning it needs to believe defending is good. The conclusion was that PPO (the learning method described) simply can't learn to play combat well, and so we pivoted to a search-based architecture.
The real reason, it turned out later, was that defend wasn't doing anything yet again. There was a second block bug: once the agent had died, blocking did nothing in further runs except waste energy. Why would the agent prefer preserving energy? Because due to another bug, energy didn't reset between turns (since that lived in another visual effect), so the 100% winrate was fallacious and probably relied on occasionally playing 4 strikes in a single turn.
The next post will deal with the bug hunting that followed, since after a couple of failures I adopted a principle: any surprising finding is a functional bug until proven otherwise. Claude and I went on a bug hunt that would conclude in our headless version being perfectly true to the real game, thus ensuring all future issues were really about data science and not simply a broken training environment...
...is what I thought until writing this post and trying to reproduce earlier results. Claude found a "latch" bug that corrupted PPO training runs. Once this bug triggered (about a third of the way through a training run), the agent would stop learning. Fixing the bug shows PPO can learn at least conditional blocking: the intent gap is 42% in the right direction now.
So, although search is still probably correct architecturally, the reasoning leading there was compelling, yet wrong.
Top comments (0)