Part 11: Pick Your Battles

Haven't played Slay the Spire? Here's the background - the rules, and why the game is hard to learn.

Editor's note: We're all caught up. Future posts will be at a slower cadence, dictated by progress.

Back in Part 8, as part of estimating whether looking at more positions is likely to improve the agent's combat performance, I noticed a paradoxical case where the weaker agent did better against Lagavulin (Laga to its friends), taking 50 damage instead of dying outright. The conclusion was that draw order dominates the marginal increase in move quality when allowing more time to search. However, the real conclusion is that something is broken - nobody should take 50 damage to Laga.

To understand why, we'll need to discuss the fight. Laga has about 110 HP and starts the fight sleeping, giving the player up to 3 turns to prepare for the combat. Once the player deals any damage to it, or 3 full turns have passed, Laga begins trying to kill the player. It attacks for 18 damage for two turns, then on the third turn debuffs the player, making their block and attack cards less effective. Then it attacks for 18 twice again and so on.

To take 50 damage from Laga, a player would have to be unforgivably passive. You get one free damage turn before it wakes up, then you get four more full turns before you can even take so much damage, and this is assuming you barely blocked since even playing a single defend, on an attack turn, would put you on 49 damage. The immediate guess was that the agent is too short-sighted to deal damage in early turns, instead opting to block the scary 18-damage hit, which means later on it got ground down by the debuff. It's consistent with typical AI failures, where the bad thing lurks beyond the search horizon so the local optimum is overly defensive.

Of course, there's another possible explanation: you just can't kill it fast enough. Let's say you face Laga with your starter deck and no potions. The deck is 12 cards, of which 6 deal damage (five strikes and 1 neutralize), meaning every 12 cards you deal 33 damage, ignoring energy. At 110 HP, you need to redraw your attack cards a fourth time, which means the fight needs to go on for about 9 turns, more than enough time to take multiple Laga hits. The real way to avoid this grim fate is simple: do not fight an elite if your deck can't deal enough damage fast enough.

Inspecting that fight, both hypotheses were true. The agent foolishly challenged Laga with an almost-starter deck, showing the real problem in the fight was the strategy layer: it either passed up good damage cards, or opted to fight an elite when it shouldn't have. It added insult to injury by choosing a balanced play style rather than all-out attacking and tanking two hits for 36.

Judging combat quality from that is not very useful. It's true you can play this terrible position better than the agent had, but the more pressing question was: how well does the agent play winnable fights?

Thus, a new metric was born: pit the agent in combat with a starting state (deck, potions, HP) derived from my games. By now I had 150 games, which at about 20-30 combat encounters in a typical run, supplied a rich enough corpus to check on. To resolve the "use potion" confound, the agent was allowed to use only the potions I used in the fight, although it was free to choose the timing.

The agent did better than I thought:

	Fights	Agent lost	I lost	Avg HP lost (me)	Avg HP lost (agent)
Hallway	63	0	0	2.0	4.0
Elite	84	19	0	5.4	28.9
All	147	19	0	3.9	18.2

Broken down by acts:

	Agent lost	Avg HP lost (me / agent)
Hallway, act 1	0	1.8 / 1.5
Hallway, act 2	0	3.2 / 5.9
Hallway, act 3	0	0.9 / 4.7
Elite, act 1	4	11.3 / 19.8
Elite, act 2	8	6.3 / 38.5
Elite, act 3	2	0.0 / 18.7
Elite, act 4	5	4.0 / 38.7

What we see here is that the agent holds its own in simple combats (act 1 hallways), then its performance degrades rapidly as combats become longer. The next order of business was to peek inside fights to understand what pathologies lay dormant in its brain. If you've been with the series for any amount of time, you know what's coming.

Functional bugs.

A lot of them were boring: more hardcoded A20 values in the sim, or "todos" I never got around to. As an example, the sim had no intents for enemies beyond act 1 because at the time the agent never beat act 1 so we didn't want to invest the time in adding those.
These had an important impact: when the agent needs to pick a move it uses a heuristic we trained on combat outcomes, to evaluate which of the possible "end of search" states is the best for it to reach. The labels it was trained on contained some artifacts from faulty data - one striking finding was that they valued a full belt of three potions at about 3 HP. Just cleaning up the labels mattered: on choices that made a real difference, the agent's quick evaluation of its options picked the plan that played out best only 26% of the time, worse than picking at random (35%). With clean labels, 46%.

One bug, however, I consider a happy accident. When looking ahead multiple turns, the agent needs to assume something about what moves the enemies will make, and what cards it will draw. Because of the way we integrated to the simulator, it turned out random state was preserved. In other words, the agent could predict the future, including its own draws.

My first instinct was to fix this, but when I gave it more thought I realized this is actually a good way to approach the problem. Solve the game given perfect information, then complicate it. I'd actually taken a similar stance early in the project, when I disallowed the agent from picking Runic Dome (a relic which doesn't let you see enemy intents) and avoided it in my own games even when it was the best pick.

For curiosity's sake, we did benchmark the agent without foresight. It saw a possible future (or more than one) but not necessarily the one that would later occur. Its performance plummeted which I consider, perhaps paradoxically, to be a good thing: it means that it makes good use of the information it has. Recall that during earlier attempts it struggled to outperform random-non-silly play.

The "no foresight" experiment (run on 8 fights, some where the agent did well originally, some where it did poorly):

	Me	Agent, with foresight	Agent, without foresight
Elites, HP lost	144	357 (1 of 8 lost)	388 (3 of 8 lost)
Hallways, HP lost	82	12	90

We've come a long way from the Slime Boss specialist. Back then I could see no clear path to progress and abandoned development in favor of writing these posts. Now I had several threads to pull:

• Retrain the heuristic on clean data.
• Improve combat by adding a "planning" layer.
• Focus on the strategy layer so it reaches more winnable combats.
• Fix the foresight bug and use that as the new baseline for performance.
• Adaptive search budget (since the agent did consistently poorly in multi-enemy fights due to the resultant combinatorial explosion)

I chose to leave the foresight bug in for now. Turning it off is easy (that's how we ran the experiment above); playing well without it is the hard part, since we're dealing with two kinds of futures: enemy intents can be simulated, but future draws need to be aggregated.
The obvious approach, letting the agent plan against four possible futures and averaging the results, was a failure. It just reshuffled which fights went badly, and tended to favor the lines that looked safe in most futures. Against the Chosen, playing it safe let the deck fill up with Dazed, and the agent bled from 63 HP to 1. I suspect what we need is an attention-like mechanism since I usually care about specific cards and how fast I can get to them, but that's in the backlog for now.

We also simulated a placeholder "strategy" layer: one of several "pure" strategies (all-out offense, zealous defense, focus one enemy down, and a couple of others) dictates the first three turns of the combat, after which we rolled out normally using search. Replaying the "Laga with a near starter deck" fight ten times with different draws, the all-out attacker never died; the agent's balanced style died in half of them, and lost more HP on average (counting a death as losing everything). Balanced play only came out ahead once the fight was already under control.

The goal of this exercise was to see whether there's any merit to the approach. Initial testing showed that it had:

Strategy (first 3 turns)	Win rate	Avg HP lost	Fights where it beat the agent
Agent (no fixed strategy)	96%	9.4	—
Fortress (max HP after the enemy turn)	92%	16.9	53
Focus (kill, or hit the weakest enemy)	92%	22.1	16
Hold for burst (fortress until a big turn)	91%	21.4	31
Race (max damage)	90%	25.1	24

Strategies have an impact. Overall, in 124 out of the 507 elite fights we tested, a strategy outdid our current production agent. So, if we could reliably pick the right one, without even improving the strategies themselves, we could expect to do better in about a quarter of elite fights (at least with foresight). Of course, that just lends itself to an item on my backlog: adding this "strategy picking layer".

The other big thing was retraining the heuristic. Results are a bit strange here. Besides fixing the bugs (like missing enemy intents), we also tried to add some "future" features, similar to earlier work that added "turns to kill" to enemies. Claude came up with 10 features, and each was computed for each of 4 upcoming turns, for a total of 40 features. Besides "incoming damage", they tried to account for scaling: enemies getting stronger or making you weaker, and shuffling detrimental cards in your deck. To that we added a bunch of "turns to kill", "turns to die", "average damage" and so forth. The features were pretty flawed (since to properly construct them you need to assume some things about agent play) but we wanted to see whether they give us anything at all (since the agent could just as likely end up ignoring them).

The results, on the benchmark's 84 elite fights, all with foresight. Note the production agent now loses 3 instead of 19, thanks to the fixes above:

Version	Fights lost (of 84)	Total HP lost	HP per fight vs production [95% CI]
Production	3	899	—
Retrained on my games	2	943	+0.5 [-1.8, +3.1]
...with threat features	3	903	+0.1 [-1.5, +1.6]
...plus the rollouts	1	842	-0.7 [-2.9, +1.5]
...plus rollouts and threat features	1	830	-0.8 [-3.1, +1.3]
Me	0	454	—

What we see here is marginal improvement from the new things, small enough that it's within the noise margin. So why am I deeming it progress? Because the numbers show there's a lot of meat on those bones. Our problem is tuning but, at least with foresight, the strategies differ meaningfully, the new features carry a lot of weight (although currently they make the agent do worse on the hold-out set), and overall the agent isn't terrible at combat and looks like it can roughly match me at its current trajectory.

Most importantly, it looks like the unlock we still need to allow self-play outside the combat sim is the strategy layer: map pathing, card picks, potions, relics and so on. That's exciting because so far this aspect of the agent has been completely neglected, which also implies, due to the Pareto principle, we can expect a spike in agent performance on the real game once we tackle those. Once this lands, or lands on its face (as is more common in this project), you can expect the next post.