I train Among Us bots by self-play: the crew and the impostors are both learners, and they play tens of thousands of games against each other and against saved copies of their earlier selves. I run the training in leagues, each one a roughly ten-hour run that starts from the bots the previous league ended with. The fifth league looked healthy for ten thousand games and then fell over. Between game 10,000 and game 12,500, the impostors' win rate in their own training games went from 50% to 75%, the crew's tasks went from 70% done to 41%, and the impostors almost stopped killing (6.52 kills a game down to 2.00). They had found something better than killing: sabotaging the reactor. The run went on for another 52,500 games and never recovered.
The game, briefly
Among Us puts a crew of crewmates and a few hidden impostors on a spaceship. The crew wins by finishing its tasks (each a short errand at a console, done in steps) or by voting every impostor out in meetings. The impostors win by killing until they match the crew in number. They can also travel through vents, shut a room's doors for 10 seconds, and sabotage the ship. Lights and comms sabotages are nuisances; the reactor and O2 are critical. When the reactor is sabotaged, two crewmates must reach it and hold it within 30 seconds, or the reactor melts down and the impostors win (O2 is the same idea with two panels in different rooms). My simulator follows the 2018 version of the game on its first map, the Skeld, with 15 players and 3 impostors.
A crewmate's walking is decided by a small network. At every step it is offered the rooms next to it, and staying put, and it scores each option as a weighted sum of a dozen inputs. Two of those inputs matter here: "this room is closer to my next task" and, during a sabotage, "this room is closer to the fix". Both are measured in the same unit, rooms along the ship's corridors, so a 1 means one room closer. The weights are not fixed: a recurrent network computes them each step from what the crewmate has seen, and training (PPO, a standard reinforcement learning method) adjusts that network. The option with the highest score is the most likely pick, so the weight on an input is a direct reading of how much the crew cares about it, and two weights on the same unit can be compared.
What the crew stopped doing
Sabotaging the reactor should not keep working against a crew that fixes it, so the obvious suspect was the crew. I wrote a probe that loads a league's saved crew and plays 40 games against the simulator's scripted impostors, so every crew faces the same opponents. Whenever a reactor or O2 sabotage is running, it records the crew network's weights on the two inputs above, and whether the crewmate picks the option that actually gets closer to the fix. It also counts how many of those sabotages get fixed. The probe uses a fixed random seed, so it gives the same numbers on every run.
| crew | weight on "closer to the fix" (10th to 90th percentile) | weight on "closer to my next task" | steps toward the fix | reactor/O2 sabotages fixed | games lost to a meltdown |
|---|---|---|---|---|---|
| league 4 (the run before) | 5.79 (4.47 to 7.57) | 1.15 | 90% of 494 | 15 of 18 | 0 of 40 |
| league 5 (collapsed) | 1.62 (-0.23 to 3.47) | 6.60 | 32% of 399 | 0 of 7 | 3 of 40 |
The weights are averages over every crewmate decision taken while a reactor or O2 sabotage was running, and the step counts are those decisions. League 4's crew, during a sabotage, valued getting one room closer to the fix five times as much as getting one room closer to its next task. League 5 started from league 4's crew and turned that around: it valued its next task 4.1 times as much as the fix. A crew that steps toward the fix one time in three does not get two people to the reactor in 30 seconds. League 5's crew met fewer sabotages (7) because its games went differently; of the seven, three melted down, and the other four were still running when the game ended some other way, mostly by the impostors outnumbering the crew.
Why it unlearned it
The crew's reward is the game result (1 for a win, 0 for a loss), plus 0.3 times the share of the crew's tasks that got done, plus 0.1 for being alive at the end, plus 0.15 for every task step a crewmate completes itself. Fixing a sabotage appears nowhere in that list. It pays only indirectly, by keeping the game going so that the result and the tasks can still pay later.
That is enough while fixes usually succeed. Once the impostors sabotage the reactor every game and some fixes fail, walking to the reactor starts to look like a bad trade. A crewmate who goes gives up task steps that pay 0.15 each, and if the fix fails anyway, the game is lost whatever it did. Learning then lowers the weight on the fix, fewer crewmates go, more fixes fail, and the weight drops further. The impostors' side of the same loop is simpler. Every failed fix is a win, so they sabotaged more (1.77 reactor sabotages a game at 12,500 games, against 0.28 at 10,000) and killed less, because killing had stopped being necessary.
League 4 had reached the same edge and come back. Its impostors used the reactor from early on (0.71 sabotages a game at 2,500 games) and peaked at 1.25 at game 10,000, then dropped to 0.04 by game 20,000, so its crew must have kept fixing often enough for the trade to stay worth it. League 5 differed from league 4 in two ways: a new input to the crew's walking network ("this room is closer to where I last saw the player I suspect most"), and a floor plan of the ship rebuilt to match the real game, with narrower rooms. I cannot tell which of them, if either, tipped it over. What I can say is that the reward never pointed at the fix directly. The evaluations show the result. Every 10,000 games each league plays 100 games against a fixed set of opponents, and at every checkpoint after the start, league 5's crew beat the league 4 impostor less often than league 4's own crew did (54% against 78% at game 60,000).
Paying for the fix
The sixth league started again from league 4's bots and added the missing term. Every crewmate standing at the fix when a reactor or O2 sabotage is fixed gets 0.3, about two task steps' worth, and lights and comms pay 0.1. I also put league 5's reactor-spamming impostor in the pool of opponents, so the new crew had to learn to beat it. Finally, I logged the crew's fix rate in every 2,500-game training window, so that a repeat would show up early.
The same probe on league 6's crew after 33,916 games:
| crew | weight on "closer to the fix" (10th to 90th percentile) | weight on "closer to my next task" | steps toward the fix | reactor/O2 sabotages fixed | games lost to a meltdown |
|---|---|---|---|---|---|
| league 6 | 22.68 (17.96 to 27.68) | 3.24 | 99% of 747 | 30 of 34 | 0 of 40 |
During a sabotage, league 6's crew values the fix seven times as much as its next task. In training the crew fixed between 85.6% and 93.9% of the reactor and O2 sabotages in every window. The impostors gave the reactor up almost entirely, at 0.02 reactor sabotages a game or fewer in every window. At the final evaluation the crew beat league 5's reactor-spamming impostor in 93 of 100 games.
What this does not show
League 6 changed more than the reward. It restarted from league 4 rather than continuing league 5, and between the two runs I fixed several simulator bugs that bear on sabotage. Meetings had been ending a running sabotage instead of pausing it. Impostors could see how many crewmates were in a room they were about to lock. Doors could be shut during a sabotage, which the 2018 game does not allow. So the fair statement is that the fix reward plus those changes produced a crew that fixes sabotage, not that the reward alone did. The 30 of 34 is also from one seeded probe of 40 games; a different seed gives a different count.
It also overshot. League 6's impostors never found a new strategy. They won 17% to 25% of their training games and beat their own crew in only 13 of 100 evaluation games. Human players are the other way round: impostors win 57.69% of the time, according to the game's developers in December 2020. Getting the bots there by training, rather than by handicapping either side, is the next run.
The simulator, the training leagues and the probe were written by Claude Code at my direction, and several of the bugs above were ones I found by watching replays. The code is not public. The probe is a single script that loads a saved league and prints the table rows above, and every other figure here comes from the leagues' own training logs.
The general lesson is about the reward rather than the game. A behaviour that only pays off through a delayed outcome is the first thing to go when an opponent starts punishing it, because the learner sees the cost every time and the benefit only sometimes. If there is an action your agents must keep doing under pressure, it needs a term of its own.
Top comments (0)