Haven't played Slay the Spire? Here's the background - the rules, and why the game is hard to learn.
Editor's note: chronologically, we've caught up to where we were when I started to write this blog. Part 11 will still be published at the regular cadence, but we might slow down since future posts will require successes or interesting failures to talk about.
The Slime Boss fight involves several mechanics. The first is a damage check: on its third turn, the Slime Boss attacks for a lot of damage, often more than an act 1 deck can block easily. To negate the damage, the player can drop it to below half HP, which will cause it to split into two slimes.
The split mechanic itself is another interesting bit. Each of the two smaller slimes will again split in two when they reach half HP or below. That means in the best case, you deal 140 damage to the Slime Boss without it splitting and finish the fight. In the worst case, you deal 70 and have to deal with two 70-HP slimes. Deal 35 to each and now you have four slimes with 35 HP each, and end up having to deal 280 damage total to conclude the fight.
In addition to that, it shuffles useless Slimed cards into your deck, a challenge for the Silent, who already starts with more weak Defend and Strike cards in her deck than the other characters. Moreover, due to splitting, this is a multi-enemy fight, which means the most powerful poison cards are of limited value since they take out one enemy at a time (and poison is cleared when slimes split). Finally, the smaller slimes inflict Weak and Frail, limiting the effectiveness of attacks and blocks.
The strategy against the Slime Boss is to split it at low HP, to make the slimes easier to manage and require less damage overall to finish the fight. To that end, on turn 1 a good Silent player decides whether the plan is to tank the turn 3 hit or split by turn 3. Consequently, during the first few turns of the fight, the player gets the Slime Boss to 71 HP or close to it, then prepares a large damage burst.
The agent showed no sign of figuring any of this out. It either refused to split, keeping the Slime Boss at around 75 HP while it slowly died, or dealt damage mindlessly, got bad splits and died. Thus, we decided to train a special model just for the Slime Boss. After adding an ability to inject a state into the AlphaStS simulator, we extracted 442 "states" from the 9 Slime Boss fights we had from my games at the time. We wanted to use outcome labeling, but since I always prevailed, we would need some counterfactuals. Claude suggested I throw some runs, but I couldn't figure out how to do that intelligently, so instead we just went with random play.
Every turn, the agent would list all possible non-dominated plays, then pick one at random. The outcome would be labeled by whether it won or lost, how much HP it lost if it won, and how many potions it managed to preserve. It never beat the hardest fight. It won 94% of the time in the easiest one.
The agent ran each combat, from the starting position, 200 times, resulting in 1,800 fights or 11,816 positions with known outcomes. We combined them with the 442 positions from my games, weighted to about a fifth of the training, and ran into the first problem: the agent pruned my moves away. As mentioned in Part 9, our dominance pruning didn't account for potions. We added more dimensions to the vector, to account for potions, and now had five setups to compare: the baseline, with greedy potion use and without, the specialist with and without, and a "control" model that was trained only on my games, to see whether the random play adds anything at all.
The "regular" model, trained on many different fights, won 7 fights out of 9. The specialist that always drank all potions won 8 of the 9 fights it was trained on. However, the specialist that could consider preserving potions and the control both won only 4. Somehow enriching the specialist with the ability to preserve potions made it play considerably worse.
We tested several hypotheses as to why that was:
- That the model expects to lose anyway so the only reward it can reap is preserving the potions.
- That because I usually won without expending potions, the model learned this propensity without the associated tactical skills.
- That the absolute range of scores for positions was just smooshed together and the model essentially plays randomly.
- That the specialist only learned to ignore the evaluator and rely purely on search.
None held up with directed testing. A fifth hypothesis that did survive was that in Slime Boss fights, the agent never searched far enough to see a win. Adding potions exploded the number of possible lines (every line X every combination of potion usage), so the search budget ran out before search could find anything meaningful (whereas in the same fights, with the same search but potions forced, the agent did find wins). As such, the specialist fell back to its judgment of positions, which in practice was nearly identical among all non-pruned possibilities, making it essentially play at random even post-training.
At that point I became discouraged. It seemed that while causes changed, failure stayed constant. Intuitively, I felt there was a breakthrough waiting to happen, but in practice I found myself tinkering with search settings and making minuscule progress. It was time for a change of pace.
I took a break from the project and wrote this blog instead, as a way to organize my thinking. While doing that, I also used the time to record more games, since my corpus was very thin, forcing us to validate on the same fights we trained on.
As I'd hoped, writing these posts made me scrutinize my own prior work. I've identified some rushed conclusions, found a lot more functional bugs that impacted training, and learned to ask much better questions. Part 11 will talk about those and show some actual progress, although I haven't yet unlocked the self-play loop.
Top comments (0)