DEV Community: Doron

Part 4: Slay the Cultist

Doron — Tue, 01 Sep 2026 13:00:00 +0000

Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.

The first three combats in StS are drawn from a special pool of "easy encounters". Of those, the most straightforward is the Cultist: it starts with between 48 and 54 HP (meaning it takes 8 or 9 plays of "strike" on it to kill it). On its first turn, it performs a ritual, dealing no damage to the player. On every turn after, it attacks for an increasing amount of damage.
The cultist is there to teach human players the concept of a damage race: that it's sometimes better to take a little amount of damage now to save a large amount of damage later. The cultist's first attack is for 6 damage, just over the 5 block from a "defend" card. A player holding two defend cards and three strike cards would need to decide whether it's better to deal 18 and take 6, deal 12 and take 1, or deal 6 and take 0. The ground truth would come from calculating the expected value of remaining HP after the fight's end, taking into account draw order variance. That's the only unknown, since the cultist itself is perfectly predictable.

To test whether my agent can learn to do any spire-slaying well, I pitted it against the cultist, over and over again. Specifically, for 200,000 training steps, which is about 14,000 fights. The network would output a probability for every play as well as a prediction for how well it'll do. The agent sampled from the distribution of plays, and recorded how close the predictions were to the reality of how well it did. Rinse and repeat.

The agent did terribly: after this extensive training, it reached a 70% win-rate. That means 30% of the time it died to the cultist, showing it didn't have the wherewithal to learn a simple strategy like "always play attacks in hand".
That's because it had no idea what its hand contained.

In part 2, I briefly mentioned mean-pooling being performed on the hand encoding. The agent simply assigned the same probability to every card in hand, and what we learned was that if you randomly play 3 energy's worth of cards every turn, the cultist will die about 70% of the time.
Giving the agent the ability to know which card it's playing, by scoring every card in hand individually against the game state using a new head made the agent reach 100% win-rate after only 10,000 steps.
Extending the experiment to also include the other possible early encounters showed a similar result - the agent quickly learned to win its first combat with a starter deck.
(Technical note: the exact training regime was PPO with a learning rate 3e-4, 1,024 steps per batch, minibatches of 256, 10 epochs per batch, gamma 0.99, GAE lambda 0.95, clip range 0.2, entropy coefficient 0.15)

The next order of business was minimizing HP loss in the fight. On the lowest difficulty I expect to beat the cultist taking no damage and taking 0-4 for the other encounters. The closer the agent can get to that, the more confident I'd be in its ability to teach itself tactical principles with self-play.
The starting point agent concluded fights with about 47 HP left, meaning it took 23 damage on average. We set out to give it the tools it would need to make the right judgment calls about its level of aggression.

Peeking inside its log of plays, the agent was averse to blocking. This wasn't about trade-offs. Having played all possible attacks and with leftover defend cards and energy, the agent would end its turn rather than defending.
Why would an AI agent teach itself that blocking isn't a good idea when it's being penalized for losing HP in the fight? One good reason would be that defend cards didn't really do anything. Contrary to my idealized view of game design, the logic that made block cards apply block was sitting in a visual effect function, one that was silently skipped. We fixed that, added some safeguards to ensure there would be no regression, retrained the agent, and it refused to block, again.

With "a functional bug" no longer an explanation, we began by experimenting with rewards. The agent was rewarded for winning the fight and for preserving HP. That sounds fine on the face of it, but the rewards for blocking are delayed. You play a defend on turn 1, win on turn 4, and need to somehow figure out the 5 extra HP preserved are attributable to that and not the other 16 or so moves you made. We gave some immediate reward for blocking incoming damage, added a feature to avoid over-blocking, and decreased the "win" reward magnitude, all in an attempt to avoid drowning out the gradients showing "block is good".

Those experiments helped a bit, pushing average HP at fight end to 55, but there it stopped. So, I wrote some simple combat heuristics to bootstrap the agent. Just telling the agent to always block with leftover energy pushed average HP up to 63 while not hurting the win-rate. The problem was training from there actively degraded the agent's performance back to the 55 HP area (more interestingly, it collapsed to 34-ish, then climbed back up to 55 during further training). Trying to force the learned behavior to not stray too far from the heuristic (using something called KL regularization) either did nothing (when the constraint was tight) or showed the same degradation (when the constraint was looser).

Other experiments raised the "defend rate" but not the quality of blocking, meaning the agent actually blocked more when it wasn't attacked. Throwing in attention was not all we needed - it raised the HP preserved by about 4.4, but by blocking less and simply attacking more efficiently.

Paradoxically, the agent kept learning to block less when being attacked. The technical term is intent gap: the agent was slightly more likely (0.2%) to play a defend when the enemy wasn't attacking than when it was. Everything we did to improve its quality of play raised its propensity to attack, rather than block. So, we settled on the exploration trap: to learn that defending is good, the agent needs to try it out and see that it helps. And to do that, it needs to give a high probability to defending, meaning it needs to believe defending is good. The conclusion was that PPO (the learning method described) simply can't learn to play combat well, and so we pivoted to a search-based architecture.

The real reason, it turned out later, was that defend wasn't doing anything yet again. There was a second block bug: once the agent had died, blocking did nothing in further runs except waste energy. Why would the agent prefer preserving energy? Because due to another bug, energy didn't reset between turns (since that lived in another visual effect), so the 100% winrate was fallacious and probably relied on occasionally playing 4 strikes in a single turn.

The next post will deal with the bug hunting that followed, since after a couple of failures I adopted a principle: any surprising finding is a functional bug until proven otherwise. Claude and I went on a bug hunt that would conclude in our headless version being perfectly true to the real game, thus ensuring all future issues were really about data science and not simply a broken training environment...

...is what I thought until writing this post and trying to reproduce earlier results. Claude found a "latch" bug that corrupted PPO training runs. Once this bug triggered (about a third of the way through a training run), the agent would stop learning. Fixing the bug shows PPO can learn at least conditional blocking: the intent gap is 42% in the right direction now.
So, although search is still probably correct architecturally, the reasoning leading there was compelling, yet wrong.

Slay the Spire, for People Who Haven't Played It

Doron — Mon, 31 Aug 2026 17:06:32 +0000

What is Slay the Spire?

Slay the Spire (StS) is a roguelike deckbuilder. It's a single-player game where you assume the role of one of four characters attempting to "ascend" the titular spire. Every run (a single game) starts from the same point: you get a small deck of weak cards and little else. As you progress, you can add more cards to your deck, remove existing cards and accrue other resources, such as relics, gold and potions. Encounters become gradually harder too, with enemies being more dangerous and more resilient.

StS runs are split into three acts, each act comprises 16 floors. At the beginning of every act, the player is presented with a randomly-generated map with multiple starting points.
Nodes in the map can be combat rooms (where the player fights one of the act's potential encounters), elite rooms (where the player faces off more difficult foes in exchange for greater rewards), campfires (where the player may upgrade cards or rest, restoring some HP), shops (where the player can trade gold for relics, potions, cards or remove a card), treasure rooms (where the player may open a chest to receive a relic and sometimes gold) and unknown rooms (that may randomly be a combat, shop, or treasure room, or contain an "event" with many potential effects).
The act map always includes a layer of treasure rooms, a layer of campfires before the top layer, which is the boss room for that act.

A sample map. The player can start at any of the four normal fights on the bottom. Question marks are unknown rooms, horned enemies are elites, the bags of money are shops and the icon on the top informs the player the act's bosses are Deca and Donu.

Like many strategy games, StS requires players to balance short-term value against long-term value. An example is potions: potions can be used once to perform one of several beneficial effects. To that end, they typically have an immediate value. However, because they can be used only once, strong potions should be preserved to maximize impact - except you can only carry so many potions at once, so hoarding them risks trading immediate value for nothing.
The cards offered to players also exhibit a similar tension. Some cards are strong but have delayed effects, while others have a limited potential but work immediately. For example, the card Glass Knife deals 8 damage twice to a single enemy, then becomes weaker (so it would deal 16, then 12, then 8 and so forth). 1 energy for 16 damage is a good deal, but in longer-running fights the card becomes more and more of a liability. Conversely, Noxious Fumes applies 2 poison at the beginning of each player's turn. It does nothing the turn you play it, then 2 damage, then 3, 4 and so on. It ends up dealing an amount of damage quadratic in the number of turns, but ramps up very slowly.

Combat

One of the unique features of StS is the turn structure. Every turn starts with player actions, followed by enemy actions. This means if the player's actions take the enemies out of commission, the enemies have no chance to retaliate. Enemies indicate what move they intend to make on their turn in advance. Most moves' outcomes are completely predictable, while some are subject to limited randomness (for example, an enemy may indicate it plans to summon allies. The identity of allies will not be known ahead of time, but the pool is small).

At the beginning of each combat, the deck order is randomized. Every turn the player draws five cards from the cards not drawn yet (the draw pile). Played cards go to the discard pile (for the most part), as do cards in hand when the player ends their turn. When the player needs to draw a card but the draw pile is empty, the discard pile is shuffled and becomes the draw pile. This means that while draws are random, players have some ability to predict future draws.
For example, if I have one copy of a card and I draw it early on, I know I can either play it immediately, or will have to wait, at least however many turns it'll take me to go through my draw pile, before I can see it again.

At the beginning of every combat turn, a player's energy is set to some number (initially 3, but this value can change with some relics). Playing cards requires paying their energy cost, which ranges from 0 to 3. Generally speaking, the more cards cost, the more impactful they are.

Enemies often attack the player. The player may use block cards on their turn. Unblocked damage causes the player to lose HP. Excess block (more than needed to mitigate all incoming damage) disappears at the end of the turn. HP doesn't regenerate automatically between floors.

A fight versus The Champ. The player has 16 block accrued against The Champ's attack for 18 damage. The player has 3 energy left out of the 3 they started with. The draw pile contains 2 cards, the hand has 5 (costing 1, 2, 1, 0, 1 energy respectively) and the discard pile has 14.

Tactics

The first skill StS players learn is how to optimize their sequence of plays for a given turn. Given a particular draw, amount of energy and enemy intents, the question is what cards to play, and in what order, to maximize damage dealt and/or minimize HP loss.
Because some cards have unpredictable effects (either due to randomness or due to drawing cards), a human player would first decide whether to play those cards at all, and if so, would tend to play them first and re-evaluate once the unpredictability has resolved. For example, they might play a card that deals damage to a random enemy, then when seeing which enemy was hit, choose their next plays.

Some tactical decisions can only be made in the context of looking ahead several turns. There's often a choice between taking no HP loss and dealing less damage this turn, and taking some HP loss and dealing more damage. The correct answer would depend on factors such as enemy pattern (an enemy that becomes stronger over time would push the player to aggressively trade damage with it), the contents of the draw pile (if I know next turn my expected damage is low it might impact how I play this turn) and the exact values of enemy HP (pushing an extra point of damage is worth a lot more if it means I can kill this enemy next turn).

There are also combat-level considerations. For example, it's sometimes wise to defer dealing damage in order to not cross an enemy HP threshold that would change the enemy's move set for future turns, or to allow a relic that activates every N turns to progress its counter, or to wait to draw a card with an effect that persists after combat (such as wanting to play alchemize to generate a potion). These sorts of considerations aren't about looking ahead a few turns - rather they're about looking ahead to the next fight, or to the next phase of the current fight.

Balancing Risk versus Reward

After a map node is complete, the player chooses one of the connected nodes to travel to. Choosing the path through the map is a strategic concern requiring players to balance risk and reward. A player may lock themselves into a path that's high value (many elites and campfires) but no branches, meaning if a combat goes wrong and the player loses a lot of HP, the entire run is in jeopardy. The more gold you have, the more inclined you are to path towards a shop. The weaker a player is, the more inclined they are to go for normal combat encounters, where they can get potions and card rewards to improve their deck. The act boss is known in advance, and so players continuously evaluate their chances of beating it, adjusting their resource use and pathing accordingly.

Every route to a better deck runs through combat: card rewards come from winning fights, and the gold that buys cards in shops comes from the same place.
Elite fights contain stronger monsters. These are not regular monsters with more HP or higher damage - these are enemies designed to prey on specific strategies. Some enemies punish inconsistent decks by attacking every turn. Some punish over-reliance on frontloaded damage, some punish slow decks, or small decks, or a reliance on skills to deal damage. In exchange, defeating an elite yields a relic (an item bestowing a continuous positive effect to the player), more gold than a regular fight and a card reward containing rare cards, which typically have stronger effects one can build a deck around.

A card reward screen. The player may pick any one card, or skip to pick none of them.

Correctly evaluating when to fight elites is critical to success. When done well, it allows a player to accrue resources which in turn allow them to fight more elites, potentially snowballing the strength of their build and outpacing the ramping of difficulty in higher floors. Misjudging, when not fatal, can leave a player with low remaining health, which in turn would force further combat avoidance, resulting in loss of potential value such as card rewards, relics and card upgrades.
To judge accurately, a player weighs which elite they might face, how badly their draws could go, what the map's remaining branches allow, and what their relics and potions can add.

Deck building

Cards' value doesn't exist in a vacuum. A card can be considerably better or worse depending on the relics a player has and the other cards in the deck.
For example, Blade Dance generates three Shiv attack cards. Accuracy increases the damage of Shivs. Accuracy is worthless if you have no Blade Dances, but is quite strong if you have two or three of them, since it doubles the damage of each. StS has many such synergies, more and less direct.

Blade Dance and Accuracy work better together.

A card's value also shifts in accordance with the upcoming encounters. The classical example is the act 1 elite, Gremlin Nob. This enemy gains strength whenever the player plays a skill. So, early in act 1, if an elite is coming up, attack cards are higher value and skills are lower value. Conversely, right after fighting the nob, skills appreciate in value since you can't fight the same elite twice in a row. The same concepts apply to other encounters and the act boss - each may penalize some approaches to winning combats while rewarding others.

Finally, decks need to be balanced. Encounters, elites and bosses in StS are designed to punish one-dimensional decks. For example, the Time Eater limits the amount of cards a player can play each turn, inhibiting a set of strategies around infinite combos. The Nemesis becomes near-invulnerable every other turn, messing with decks whose damage-mitigation plan is to kill all the attackers. A good player considers "lose conditions" and builds their deck around those - or attempts to avoid encountering them.

Metagame knowledge

Randomness aside, StS always tells you what a certain card will do when played. However, there's a lot of game knowledge that needs to be discovered through playing a lot. For example, StS employs a pseudo-random distribution for randomness, meaning the more a random event doesn't happen, the more likely it becomes. The chance to get a potion from a combat is 40%. If you do, the chance goes down to 30%. If you don't, it goes up to 50%. The same goes for the chances of getting a rare card, the chance an unknown room will be a combat/shop/treasure/event and so on.
The other part is enemy patterns. Enemy intents tell you what they do this turn. However, only playing repeatedly teaches you that the Gremlin Leader is more likely to attack you if it has some gremlin sidekicks alive, or that Nemesis can't use its 45-damage attack two turns in a row.
This sort of knowledge helps players hedge against the inherent randomness in the game, meaning ultimately it's exceedingly rare to die to "bad luck".

In summary: StS requires players to balance between short-term value and long-term growth, constantly re-evaluate the value of their current resources and plan ahead over hundreds of "turns" in order to defeat the titular spire. It's a challenging game with a high skill ceiling, for both human and AI agents. Unlike chess, where your poor play (or the opponent's brilliant one) is immediately apparent when analyzing the game, in StS it's hard to tell which of a dozen inter-related decisions was a mistake, since outcomes alone don't carry this signal due to the inherent variance.

Part 3: Good Models Finish Last

Doron — Tue, 25 Aug 2026 13:00:00 +0000

The previous post glossed over teaching the agent to pick card rewards. Beginner StS players usually look online for "tier lists" - resources that give a letter grade to every card. Then, they pick the highest-tier card at every card reward and die horrible deaths due to having clunky decks.

Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.

Our cloned expert (spirecomm) chose cards exactly this way: it defined a strict ordering over all cards, with a cap so it doesn't end up running 6 copies of Footwork. "Skip" was also in there, which meant that at every card reward, the agent would pick the highest-rated card that's above "skip" for which it hadn't reached the quota. Interestingly, another StS AI project (bottled AI) had a similar approach.

When I saw my agent repeatedly crash against the act 1 boss and die with a poor deck, I naturally assumed the problem was quality of card picks. Even after fixing the card ID mismatch, the agent's decks were bad. I figured the problem was that heuristics were too crude, and decided to employ a different approach. Ironically, it ended up being mathematically equivalent to creating a tier list.

The card scores were derived using logistic regression. A "deck" is simply a list of numbers denoting how many copies it has of each of the 113 possible cards. The "model" multiplies each such number by a learned weight, adds them together with a constant and does sigmoid on the result. The number that comes out, between 0 and 1, is supposed to predict the win probability.

My thinking was that once we have that, we can test each card to see how it increases the deck quality - the chance the deck will win. It was a noble thought, but because of the mathematics of the model, it was exactly equivalent to just picking the card with the highest learned weight:
higher-weight cards contribute more to the output and thus result in increased win probability. We ended up creating a tier list... WITH SCIENCE.

Even in the "increase win chance" phrasing, this approach is still limited. Essentially, it attempts to learn causality from correlation. The longer a run goes on, the more rares you're expected to encounter, and thus the model is likely to over-appreciate rares, since they will be correlated highly with success.
Another problem is slightly more subtle: the model "wins" our training objective if it can tell apart winners from losers. Nothing in how we train it forces it to give the right magnitude to cards. If I were to ask it which of two similar decks is more likely to win, and it did well on that task, that would lend more credence to the emphasis it places on cards.

All of that notwithstanding, I set out to get data. The first data source I used was my own A20 runs. At the time I had 334 runs at 41% win-rate. To augment the data, the deck at every floor was used, meaning a collection of strikes and defends and a damage common got annotated "win" if 50 floors later I would beat the heart. The model did very well at predicting winning decks and not great at ranking cards, so I decided to get more data.

I augmented my data with 53 Silent A20H runs from Jorbs sporting a 60-something winrate. Adding those to mine confused the model. We value cards differently enough, apparently, that it made the data look bimodal and made the "guess if I won" game unfair to the model (and indeed, the game is inherently unfair, since winning is not just a function of the deck but also play quality, relics, potions and draw RNG).

So, I settled on just my runs and thus concluded my first attempt: a glorified tier list that had so few params it couldn't generalize beyond a single player's style. The model needed to take more into account.
The missing ingredient was synergy. Accuracy is terrible in a deck with no shiv cards, but strong in a deck that has three copies of blade dance. The value of cards is derived from the deck composition, as well as the current floor, act boss, elites we might encounter, relics and more. StS is a complex game.
To capture this complexity, we added features for relics (1 or 0 for each of 161) and 22 game parameters: floor, HP (as percentage of max), act, boss ID and more.
The other thing we changed was what the model did. The previous model said how strong a deck (or a card) was. This one would make card picks by getting access to critical information: the opportunity cost. The model had to play "guess what the human picked" when presented with a choice between three¹ card rewards and "skip", with access to the context specified above.

The model itself was considerably more elaborate. A "state encoder" turns 308 features, via a series of MLPs, into a 128-wide vector representing the current state. A "card encoder" enriches our old 25 features, again via a couple of MLPs, into a 32-wide vector. Finally, a scoring layer takes 128+32 numbers and derives a single scalar denoting which card should be picked. Overall, 202,000 learnable parameters, a far cry from our previous 114. More importantly, the extra MLP layers ensured it had the expressive power needed to go beyond a clever tier list.

Increasing the number of features and parameters necessitates an increase in data, as we saw when discussing card attributes. Fortunately, the other data source I found online was a 77-million run database containing 34,210 Silent runs and 358,870 card-pick decisions at various ascension (and player skill) levels, in which the win-rate for A20 is 1.5%.
Unleashed on this vast data set, the model converged to a reasonable (53.5%) accuracy, showing it could predict pretty well what a human would pick in a given situation. Encouragingly, it showed it learned something of synergy, evaluating Catalyst much higher when the deck had cards that applied poison than without, for example. The model did what we designed it to do.

Too bad we didn't design it to win at Slay the Spire. True to form, the bigger, more accurate model performed worse:

Model	Offline accuracy	Beat expert / lost / tied	In-game rank
+ card ID embeddings	55.8% (best)	12 / 23 / 15	4th (worst)
mechanical features only	53.5%	16 / 16 / 18	1st (best)
embeddings + propensity weighting	52.3%	15 / 22 / 13	3rd
propensity weighting	49.5%	14 / 21 / 15	2nd

This is the same agent with different card-pick engines, playing the same set of 50 seeds. Models that were better at predicting what humans would pick / what decks would win performed worse when guiding card picks. The simplistic "learned tier list" fared best of my attempts, and it still lost to just using the hand-crafted list from bottled AI directly. To quote the Claude instance in charge of designing experiments and interpreting results: "The embeddings learn to predict human picks more accurately, but human picks contain systematic biases. The mechanical features in v1 partially correct for these by being unable to distinguish popular-but-mediocre cards from unpopular-but-strong ones."

There's an interesting tension here: the more robust a model is, the more data it needs. The more data, the lower the quality of play. My hope was to bootstrap with mediocre-quality data and improve with self play. But before further attempts to reinforce ourselves into 100% win-rate against the spire, we would start with a more modest goal: consistently beating a cultist.

1 - sharp-eyed readers may notice Busted Crown, Binary and Question Card all impact the number of choices in card rewards. We padded with -1 for the former cases and silently dropped the fourth pick for the latter case.

Part 2: Looking Where the Light is

Doron — Tue, 18 Aug 2026 13:00:00 +0000

Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.

Setting out, when considering the agent I thought of two challenges: how many learnable parameters would it need, and what would be the reward function. Claude brought up another minor matter: what comes into the agent and what comes out.
This seemingly technical matter held surprising depth. The output is the simpler of the two: the agent can carry out "actions" which are playing a card or using a potion. Some will cause other action types, such as discarding a card (survivor) or choosing a card in the discard pile (liquid memories).

To make the network manageable, the "input" part would have to be some encoding of the game state derived from the game engine itself (versus, for example, making the agent read colored pixels from the screen or a zork-like textual representation along the lines of "you are fighting a single cultist. Your hand is two strikes..."). Choosing how to represent a game state as a series of numbers is not obvious, though.

The first choice is how to represent cards. We could give cards unique IDs, and let the network figure out over time that card #18 costs 2 energy, deals 15 damage and draws an extra 2 cards next turn. Or, we could decompose it into features like "cost", "deals damage", "draws next turn" and so on.
However, while some mechanics, like poison, appear on multiple cards, there are some unique mechanics.
Consider something like Phantasmal Killer. This is the only card that does what it does. So you add a feature for "deal double damage next turn", and another for "can only be played if your draw pile is empty" and one for "deal damage again if the enemy is poisoned". Each of these is sparse: one card out of 126¹ has a "1" in the field these denote, and 125 have a "0". The network will quickly learn to ignore this field over the course of optimizing itself, since it very rarely carries useful information.
So, we had to encode card effects, but prioritize the ones that are impactful and frequent, and hence their signal is meaningful.

The first encoding for cards had 25 such fields, listed below.
Generally speaking, numbers need to be normalized to values between 0 and 1, since otherwise binary features would look less important than numerical ones. This introduces another problem: you can theoretically have an unlimited amount of gold (using nightmare wish shenanigans), and more practically can easily have over 1000. However, typical values are between 100 and 300, which look small after normalization. We were hoping the network learns to compensate for that. There was another feature of this encoding, that due to clipping, some cards end up looking the same. For example, we are normalizing "damage dealt" by 50, meaning there's no difference in the encoding between Grand Finale and its upgraded version. Normalization values were chosen to minimize the loss of information due to clipping - applying 10 vul or 99 vul is practically the same.

#	Feature	Scaled by
0	energy cost	/5
1	damage dealt	/50
2	block given	/50
3	poison applied	/20
4	cards drawn	/5
5	energy gained	/3
6	weak applied	/5
7	vulnerable applied	/5
8	frail applied	/3
9	strength modifier	signed, x+5/10
10	dexterity modifier	signed, x+5/10
11	hits all enemies	0/1
12	multi-hit count	/10
13	exhausts	0/1
14	ethereal	0/1
15	innate	0/1
16	is a power	0/1
17	retain	0/1
18	self-damage	/20
19	unplayable	0/1
20	cards discarded on play	/5
21	HP healed	/20
22	Shivs generated	/5
23	other cards generated	/5
24	beneficial when discarded	0/1

You can see some cards cannot be represented in this way, such as the aforementioned Bane or something like Escape Plan. Moreover, looking at these features now, there are many mistakes: no silent card inflicts Frail, and indeed the column is zero for all. Energy goes up to 5, but Silent cards only cost 3 at maximum. This is because I wanted to get something up and running, and since I never thought the problem of encoding game state would be interesting, I wanted to get rid of it to get to the problems I expected to be interesting. So, I'd instructed Claude to read the sts wiki for all applicable cards, and didn't really check its work.

If I were to do it again, a simple test would be to encode all the cards, then check for features that have zero or near-zero standard deviation, meaning they carry very little information about the cards they apply to. This simple mechanical test would have saved a lot of parameters and compute power. In our concrete case, 2/25 features have zero standard deviation, and others (such as gain strength, which for The Silent is only J.A.X) have miniscule importance.

Similar to encoding cards, the state of player, enemies, deck, discard pile, exhaust pile and more all need to be represented. I won't list them all here, but another clear mistake I made was representing the hand and the draw pile the same way. Since the draw order isn't known, the draw pile is just a bunch of cards, which in turn are features. You can average them and get "expected damage per draw" which is a metric I actually use when evaluating fights against act 1 elites.

However, the hand encoding also went through a "mean-pooling" layer, which sounded smart and sciency to me, but in effect meant the model didn't know which hand slot had which card. The agent was effectively told: the cards in your hand deal an average of 4.5 damage, block an average of 3, apply an average of 1 poison... now, which specific card in hand do you want to play?

Having merrily butchered the representation of the game state, I finally got around to what I was interested in: the agent's reward function. When I was learning chess, I was told a pawn is worth one point, a bishop and a knight are three each, rooks are five, queens are nine. This crude heuristic was later augmented with the value of a pair of bishops, advanced pawns, open files and more. Chess engines have been working this way for years, until AlphaZero came along and just said "winning is 1 point, losing is zero" and taught itself everything else from basic principles.
I found that fascinating. I really wanted to have one of those too.

So, I'd planned to just reward the agent for advancing floors. Let it randomly play and stumble upon the fact that it's good to kill enemies, since you advance floors. Let the same mechanism teach it to preserve potions (if you don't, you'll die in a later floor you'd have passed if you held the potion), set up relics (you didn't stall until Incense Burner was on 4 so Nemesis 45ed you and you didn't advance past that floor) and more.

Being the people-pleaser he is, instead of pointing out that calling this approach "naive" is akin to saying the black plague was unpleasant, Claude just suggested a small tweak: instead of starting from a clean slate, we'll start with "behavioral cloning", which was a fancy way of saying we'll use spirecomm's AI's rules as a way to initialize our weights. Let our agent play, but rather than rewarding it for advancing floors, train it to do what spirecomm did over 500 recorded games, until it becomes good enough at predicting that, THEN we can give it +1 for every floor advanced at which point it'll solve A20.

I judged that to be honorable enough to not tarnish my noble objective of an agent learning everything from scratch, and off we went. The small network we defined quickly learned to emulate ForgottenArbiter's heuristics, no mean feat considering it had no idea what cards it held and thus could not predict the results of its own actions.

We unleashed it upon the spire. It reached the act 1 boss 51% of the time, reaching an average floor of 12.4.
Looking at the decks it built, the problem became clear: it barely picked cards. I again decided to forgo proper root-causing in favor of focusing on the most interesting potential issue: the magnitude of the reward from picking cards was too low compared to the constant signal from advancing floors. Picking a card barely moved the needle so the network optimized it away. The solution was to encourage better card picks, at which point performance improved to an average of 14.7 floors.

Looking at the agent again, I saw it didn't like to fight. I am a firm believer in setting proper incentives, and thus I gave it a bonus for picking fights in earlier floors. That increased the boss-reach-rate to 70% but the agent still hated fighting elites. So I gave it a bonus for relics, and rescaled some other bonuses, yet performance kept hovering.

Then I did what I should have done and set Claude to properly debug the issue. The expert we cloned from used a static list of card priority. E.g., if you see #1 pick it, otherwise pick #2, all the way to "skip" and the cards below it. That's a reasonable baseline, although one we've later revisited. However, it hinges on one key assumption: that the cards the agent sees, and the cards on the list, are the same cards. It turns out cards in Slay the Spire have several names. Sometimes it's something like "Strike_G" versus "Strike" since internally a silent strike and an ironclad strike are different but they're both called the same. Sometimes it's things like "Crippling Poison" that was renamed to "Crippling Cloud" during development but the internal name stayed the same. Sometimes it's as simple as "PoisonedStab" and "Poisoned Stab".
Bottom line, our agent was comparing game-internal strings to human-readable strings, with a silent fall-back saying if you can't find the card it's worthless.
It was not about the quality of the agent's card choices. Although that issue definitely existed (and would be improved upon), at that stage in the project it was masked by a simple string-matching bug that went unnoticed because I prefer a cool explanation to a boring one, and have sinned by reporting an error politely in a way that lets the rest of the code run instead of stopping and saying loudly "HERE BE A PROBLEM".

Fixing the card IDs got the agent to a 66% boss-reach-rate and an average floor of 15. It also ran into the first wall that would ultimately root-cause to something actually data-sciency (as well as two more functional bugs). More on that in the next post.

note: this table is pulled from internal project documentation which is spotty since I vastly underestimated how long and interesting this project would be. Hence the missing versions and confusing shorthand for results and issues.

Version	Floor reward	Deck mult	Avg floor	Boss reach	Behavior
v2	+1.0/floor	none	12.4	51%	Baseline
v3	+1.0/floor	×5	14.7	high	Routes through events, avoids combat
v4	+1.0/floor + combat bonus	×5	–	70%	More fights but thin decks
v5	+1.0/floor	×5 + potion/relic/maxHP	13.1	60%	Still unfocused decks
v8	+1.0/floor	×5 (fixed card IDs)	~15	~66%	Extreme combat avoidance

1 - 75 silent cards, 35 colorless, 16 status and curse cards. I'd deliberately disallowed Prismatic Shard and "A Note for Yourself" since otherwise the sparsity issue would be much worse.

Part 1: The First Fork in the Road

Doron — Thu, 13 Aug 2026 13:00:00 +0000

The first obstacle on the road to training an StS-playing agent is creating the training environment. Unlike humans, who can learn from instruction and very few samples, agents need to play tens of thousands of games in order to learn from the outcomes.

Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.

That meant the first order of business was to get to a point where you could reasonably run the game fast enough to accomplish that. Before me were two options:

Use a synthetic environment, such as sts_lightspeed or decapitate-the-spire, and add the missing functionality (in the former case, The Silent character I wanted to focus on).
Use ForgottenArbiter's work (spirecomm, CommunicationMod) to directly interface with the game engine, and speed up the game/gateways to allow playing fast enough.

An aside: coding agents can't estimate effort, or tell time

I asked Claude to estimate effort for the two options, finding out my first major lesson about coding agents: they have absolutely no clue how to estimate effort. In hindsight it's obvious: the training corpus contains many instances of breaking down a task into component parts, estimating said parts and aggregating. However, the estimates are for humans doing the work - there aren't enough documented instances of how long coding agents take to perform certain tasks. To make things worse, I found out Claude has no sense of time, e.g. it would work for ~20 minutes then conclude what it did took it five hours.

After several attempts to correct Claude ("no, that took twenty minutes"), I realized I'm barking up the wrong tree and simply added the following into SessionStart and UserPromptSubmit:

date '+[%Y-%m-%d %H:%M] Always start every text response with [YYYY-MM-DD HH:MM] using the time shown here. This includes after code blocks.'

This hook is somewhat wasteful (adds about 30 tokens to every turn in the conversation) but it meant Claude stopped gaslighting me about time, and it allowed me to try and extrapolate from how long things actually took.

Back to the decision

With estimates I could pretend to trust, I settled on the second option. My reasoning was that had it been simple to do, the authors of those projects would have fixed the bugs / added more characters already, since they're motivated to do so. Conversely, nobody really cared about the speed of the existing headless mode (lifted from SeedSearch) since it was designed to only play a few turns to discover seeds with some interesting properties. What I failed to consider is that this argument cuts both ways. Nobody cared about the fidelity of a seed-finding mod, either, beyond the first couple of floors. I ignored a core principle of software development: untested code is always wrong.

Not being a game developer, I envisioned a sort of model/view/controller design pattern, where the visual effects are a "view"-like layer decoupled from the business logic of the game engine and the model of the game state. And so, I set Claude on a happy pursuit of spawning the game in a headless mode and patching every time a visual effect tried to access a graphical entity and got a null pointer exception since we didn't instantiate those. After a day or two of that, the game stopped crashing and hanging, which could only mean it was running correctly.

The last order of business was speeding up the connection itself, since the agent ran in Python and the game ran in Java. Claude settled on using Py4J to pass the calls, and I pointed out where he should cache the state and batch calls to update it since there was a fixed overhead per-call of serializing and de-serializing objects.

And so, we were off to the races and could get started on actual data science work. At the time, I hadn't realized my wrong architectural decision would cost me several weeks (and not Claude-weeks; week-weeks), nor had I guessed I would end, as the Hebrew parable goes, "having to eat the stinking fish, get my lashes AND be exiled from the city". I would end up not only spending a lot longer than planned on headless mode, but also having to extend a synthetic environment anyway AND pay the cost of keeping the two aligned.

However, all that unpleasantness was in the future. Ahead of me lay my first big data science challenge: designing and training the agent. That is the topic of the next post, in which our hero discovers the problem to be slightly more complicated than giving an agent time and a reward signal.

Part 0: "Who are you?" and "Why are you doing this?"

Doron — Sat, 08 Aug 2026 07:29:23 +0000

A series about training an AI agent to play Slay the Spire. Technical content starts next week.

Haven't played Slay the Spire? Here's the background — the rules, and why the game is hard to learn.

My name is Doron, I'm a software engineer working for Intel. For the past 12 years, I've been building software for AI accelerators of various kinds, such as the much-maligned Intel Gaudi.

To build software for AI infrastructure, one needs at least a rudimentary understanding of how AI models work. Wake me up in the middle of the night and I can recite the series of computations comprising a transformer model.
At Intel this is considered "AI expertise".
However, I've never defined a network architecture, nor have I trained a net unless you count the time I had to debug why Resnet-50 training was converging to 0.5% accuracy less than the state of the art.

The other big gap was using AI. My day job lends itself to very narrow use of AI, mostly to perform research and write first drafts of documents. I wanted to use coding agents for, you know, coding, but it's been about three years since I got to write any meaningful amount of code.

The last piece of the puzzle is Slay the Spire. Since I was 5 or so, I've had at least one strategy game to pass the time with. I've played Chess, Poker, Bridge, Magic: the Gathering, Hearthstone and some less-known games to varying degrees of competitive success. StS is a single-player game with an absurdly high skill ceiling. It was released in late 2017, and until the release of its successor (creatively named "Slay the Spire 2") top players kept finding ways to do better (as measured by overall win-rate, and the crowd-pleasing "win-streak" metric).

I picked up StS in late '20, during COVID-19, and have not really put it down since, accruing about 3000 hours of play over the past six years. When not playing the game, I've often watched top players stream their games on Twitch, trying to learn from them. Interestingly, top players disagreed on some key decisions, which made me wonder who's right and how could we tell, since it's nigh-impossible to do so just by playing¹. As AI kept pushing boundaries in game-playing, from Go to Chess to Hold'Em and so on, I kept hoping a university or a frontier lab would tackle StS, but none have.

Looking on GitHub, I found ForgottenArbiter's spirecomm which included a POC simple AI with some heuristics, xaved88's bottled_ai that was a set of canned strategies claiming 20-52% winrate (depending on the character) at A20 (the highest level of difficulty) and two dedicated attempts to allow agents self-play: MANGO1234's AlphaStS, an AlphaZero-inspired implementation focusing on combat only and gamerpuppy's sts_lightspeed, a full reimplementation of the game engine in C++, unfortunately focused on the Ironclad character.

So, I decided to try and bridge the gap. Using Claude I will train an AI agent to play Slay the Spire well, thus teaching myself how to play well, as well as getting some hands-on data science experience and learn how to use coding agents. I've emailed Megacrit to ask for their blessing (since it was clear to do so would require modifications of their game and since an agent trained on their game could be considered their IP) and set out on my way in February '26. Six months later, I'm still not where I wanted to be. However, when Anthropic announced Fable 5, one of its headline results was that it "reached the game's final act three times more often" than Opus 4.8, so at least I'm in good company.

I've made many mistakes in this project, most are quite amusing. I plan to share them over a series of weekly posts. At some point posts will catch up to current progress, at which point I expect the publication rate will slow down to when I make actual breakthroughs.

1 - due to RNG elements in the game, as well as the small gap to begin with, the number of games players would have to play to make any claim with statistical significance is staggering. Moreover, it would be impossible to separate variables since players vary in other, subtler ways