Setting out, when considering the agent I thought of two challenges: how many learnable parameters would it need, and what would be the reward function. Claude brought up another minor matter: what comes into the agent and what comes out.
This seemingly technical matter held surprising depth. The output is the simpler of the two: the agent can carry out "actions" which are playing a card or using a potion. Some will cause other action types, such as discarding a card (survivor) or choosing a card in the discard pile (liquid memories).
To make the network manageable, the "input" part would have to be some encoding of the game state derived from the game engine itself (versus, for example, making the agent read colored pixels from the screen or a zork-like textual representation along the lines of "you are fighting a single cultist. Your hand is two strikes..."). Choosing how to represent a game state as a series of numbers is not obvious, though.
The first choice is how to represent cards. We could give cards unique IDs, and let the network figure out over time that card #18 costs 2 energy, deals 15 damage and draws an extra 2 cards next turn. Or, we could decompose it into features like "cost", "deals damage", "draws next turn" and so on.
However, while some mechanics, like poison, appear on multiple cards, there are some unique mechanics.
Consider something like Phantasmal Killer. This is the only card that does what it does. So you add a feature for "deal double damage next turn", and another for "can only be played if your draw pile is empty" and one for "deal damage again if the enemy is poisoned". Each of these is sparse: one card out of 1261 has a "1" in the field these denote, and 125 have a "0". The network will quickly learn to ignore this field over the course of optimizing itself, since it very rarely carries useful information.
So, we had to encode card effects, but prioritize the ones that are impactful and frequent, and hence their signal is meaningful.
The first encoding for cards had 25 such fields, listed below.
Generally speaking, numbers need to be normalized to values between 0 and 1, since otherwise binary features would look less important than numerical ones. This introduces another problem: you can theoretically have an unlimited amount of gold (using nightmare wish shenanigans), and more practically can easily have over 1000. However, typical values are between 100 and 300, which look small after normalization. We were hoping the network learns to compensate for that. There was another feature of this encoding, that due to clipping, some cards end up looking the same. For example, we are normalizing "damage dealt" by 50, meaning there's no difference in the encoding between Grand Finale and its upgraded version. Normalization values were chosen to minimize the loss of information due to clipping - applying 10 vul or 99 vul is practically the same.
| # | Feature | Scaled by |
|---|---|---|
| 0 | energy cost | /5 |
| 1 | damage dealt | /50 |
| 2 | block given | /50 |
| 3 | poison applied | /20 |
| 4 | cards drawn | /5 |
| 5 | energy gained | /3 |
| 6 | weak applied | /5 |
| 7 | vulnerable applied | /5 |
| 8 | frail applied | /3 |
| 9 | strength modifier | signed, x+5/10 |
| 10 | dexterity modifier | signed, x+5/10 |
| 11 | hits all enemies | 0/1 |
| 12 | multi-hit count | /10 |
| 13 | exhausts | 0/1 |
| 14 | ethereal | 0/1 |
| 15 | innate | 0/1 |
| 16 | is a power | 0/1 |
| 17 | retain | 0/1 |
| 18 | self-damage | /20 |
| 19 | unplayable | 0/1 |
| 20 | cards discarded on play | /5 |
| 21 | HP healed | /20 |
| 22 | Shivs generated | /5 |
| 23 | other cards generated | /5 |
| 24 | beneficial when discarded | 0/1 |
You can see some cards cannot be represented in this way, such as the aforementioned Bane or something like Escape Plan. Moreover, looking at these features now, there are many mistakes: no silent card inflicts Frail, and indeed the column is zero for all. Energy goes up to 5, but Silent cards only cost 3 at maximum. This is because I wanted to get something up and running, and since I never thought the problem of encoding game state would be interesting, I wanted to get rid of it to get to the problems I expected to be interesting. So, I'd instructed Claude to read the sts wiki for all applicable cards, and didn't really check its work.
If I were to do it again, a simple test would be to encode all the cards, then check for features that have zero or near-zero standard deviation, meaning they carry very little information about the cards they apply to. This simple mechanical test would have saved a lot of parameters and compute power. In our concrete case, 2/25 features have zero standard deviation, and others (such as gain strength, which for The Silent is only J.A.X) have miniscule importance.
Similar to encoding cards, the state of player, enemies, deck, discard pile, exhaust pile and more all need to be represented. I won't list them all here, but another clear mistake I made was representing the hand and the draw pile the same way. Since the draw order isn't known, the draw pile is just a bunch of cards, which in turn are features. You can average them and get "expected damage per draw" which is a metric I actually use when evaluating fights against act 1 elites.
However, the hand encoding also went through a "mean-pooling" layer, which sounded smart and sciency to me, but in effect meant the model didn't know which hand slot had which card. The agent was effectively told: the cards in your hand deal an average of 4.5 damage, block an average of 3, apply an average of 1 poison... now, which specific card in hand do you want to play?
Having merrily butchered the representation of the game state, I finally got around to what I was interested in: the agent's reward function. When I was learning chess, I was told a pawn is worth one point, a bishop and a knight are three each, rooks are five, queens are nine. This crude heuristic was later augmented with the value of a pair of bishops, advanced pawns, open files and more. Chess engines have been working this way for years, until AlphaZero came along and just said "winning is 1 point, losing is zero" and taught itself everything else from basic principles.
I found that fascinating. I really wanted to have one of those too.
So, I'd planned to just reward the agent for advancing floors. Let it randomly play and stumble upon the fact that it's good to kill enemies, since you advance floors. Let the same mechanism teach it to preserve potions (if you don't, you'll die in a later floor you'd have passed if you held the potion), set up relics (you didn't stall until Incense Burner was on 4 so Nemesis 45ed you and you didn't advance past that floor) and more.
Being the people-pleaser he is, instead of pointing out that calling this approach "naive" is akin to saying the black plague was unpleasant, Claude just suggested a small tweak: instead of starting from a clean slate, we'll start with "behavioral cloning", which was a fancy way of saying we'll use spirecomm's AI's rules as a way to initialize our weights. Let our agent play, but rather than rewarding it for advancing floors, train it to do what spirecomm did over 500 recorded games, until it becomes good enough at predicting that, THEN we can give it +1 for every floor advanced at which point it'll solve A20.
I judged that to be honorable enough to not tarnish my noble objective of an agent learning everything from scratch, and off we went. The small network we defined quickly learned to emulate ForgottenArbiter's heuristics, no mean feat considering it had no idea what cards it held and thus could not predict the results of its own actions.
We unleashed it upon the spire. It reached the act 1 boss 51% of the time, reaching an average floor of 12.4.
Looking at the decks it built, the problem became clear: it barely picked cards. I again decided to forgo proper root-causing in favor of focusing on the most interesting potential issue: the magnitude of the reward from picking cards was too low compared to the constant signal from advancing floors. Picking a card barely moved the needle so the network optimized it away. The solution was to encourage better card picks, at which point performance improved to an average of 14.7 floors.
Looking at the agent again, I saw it didn't like to fight. I am a firm believer in setting proper incentives, and thus I gave it a bonus for picking fights in earlier floors. That increased the boss-reach-rate to 70% but the agent still hated fighting elites. So I gave it a bonus for relics, and rescaled some other bonuses, yet performance kept hovering.
Then I did what I should have done and set Claude to properly debug the issue. The expert we cloned from used a static list of card priority. E.g., if you see #1 pick it, otherwise pick #2, all the way to "skip" and the cards below it. That's a reasonable baseline, although one we've later revisited. However, it hinges on one key assumption: that the cards the agent sees, and the cards on the list, are the same cards. It turns out cards in Slay the Spire have several names. Sometimes it's something like "Strike_G" versus "Strike" since internally a silent strike and an ironclad strike are different but they're both called the same. Sometimes it's things like "Crippling Poison" that was renamed to "Crippling Cloud" during development but the internal name stayed the same. Sometimes it's as simple as "PoisonedStab" and "Poisoned Stab".
Bottom line, our agent was comparing game-internal strings to human-readable strings, with a silent fall-back saying if you can't find the card it's worthless.
It was not about the quality of the agent's card choices. Although that issue definitely existed (and would be improved upon), at that stage in the project it was masked by a simple string-matching bug that went unnoticed because I prefer a cool explanation to a boring one, and have sinned by reporting an error politely in a way that lets the rest of the code run instead of stopping and saying loudly "HERE BE A PROBLEM".
Fixing the card IDs got the agent to a 66% boss-reach-rate and an average floor of 15. It also ran into the first wall that would ultimately root-cause to something actually data-sciency (as well as two more functional bugs). More on that in the next post.
note: this table is pulled from internal project documentation which is spotty since I vastly underestimated how long and interesting this project would be. Hence the missing versions and confusing shorthand for results and issues.
| Version | Floor reward | Deck mult | Avg floor | Boss reach | Wins | Behavior |
|---|---|---|---|---|---|---|
| v2 | +1.0/floor | none | 12.4 | 51% | 0 | Baseline |
| v3 | +1.0/floor | ×5 | 14.7 | high | 0 | Routes through events, avoids combat |
| v4 | +1.0/floor + combat bonus | ×5 | – | 70% | 0 | More fights but thin decks |
| v5 | +1.0/floor | ×5 + potion/relic/maxHP | 13.1 | 60% | 0 | Still unfocused decks |
| v8 | +1.0/floor | ×5 (fixed card IDs) | ~15 | ~66% | 0 | Extreme combat avoidance |
1 - 75 silent cards, 35 colorless, 16 status and curse cards. I'd deliberately disallowed Prismatic Shard and "A Note for Yourself" since otherwise the sparsity issue would be much worse.
Top comments (0)