My Among Us bots can explain themselves. Every time a crewmate picks a room to walk to, the replay records why, and the exported videos show it as a thought bubble over the player: "→ Admin: next task: Swipe Card (Admin); avoiding Pink (I suspect them) in Weapons". For the crew, the arithmetic behind those captions is exact to about fourteen decimal places. The captions were still wrong in three different ways, and one of them made a collapse in training look, on video, like normal play.
Why the explanation can be exact
The bots are trained by self-play, in leagues: each league is a roughly ten-hour training run that starts from the bots the previous league ended with. A crewmate's walking network does not output a choice directly. At each step it is offered the neighbouring rooms, and staying put. Each option is described by the same dozen inputs, such as how much closer it gets to the next task, how many people are visible there, how suspected the most suspected person there is, and how much closer it gets to fixing a running sabotage. The network looks at what the crewmate has seen and outputs one weight per input. Each option's score is the plain weighted sum of its inputs, and the highest score is the most likely pick.
That structure makes "why A and not B" a question with an exact answer. If option A scores s_A and the runner-up B scores s_B, then:
s_A − s_B = Σ_i (input_A,i − input_B,i) × weight_i
So the lead A has over B splits into one part per input, and the parts add back up to the lead with nothing left over. The caption takes the two largest positive parts and turns each into a phrase. There is no approximation step and no separate model of the bot's behaviour, which most explanation methods for neural networks need. (The impostors' walking adds a learned map term that is not a sum over named inputs, so their captions lump that part together as "what it has learned about the map". Everything below about exactness is about the crew.)
To check that the split really is exact in the code and not just on paper, I wrote a probe. It plays 30 games with a league's saved bots, using a fixed random seed so the run repeats exactly, and for every crew move decision it compares the score gap with the sum of the parts:
| crew | decisions | largest error in the split, in score units |
|---|---|---|
| league 3 | 80,550 | 7.11e-15 |
| league 4 | 40,433 | 7.11e-15 |
| league 6 | 34,226 | 1.42e-14 |
The median gap between the chosen option and the runner-up was between 6.3 and 9.8 in the same units, depending on the league, so these errors are floating-point rounding. The arithmetic was never the problem.
Wrong way one: the sign of the weight
The first captions had one phrase per input, chosen when that input's part was large and positive. For the suspicion input the phrase was "keeping an eye on Red". That reads as "I am going where my suspect is", and it is only true if the weight on suspicion is positive.
For the league 3 crew it mostly was not. Over 19,085 decisions with a suspected player in view, its weight on suspicion averaged -3.94 (10th percentile -6.73, 90th -0.45), so it was negative in at least nine decisions in ten: a lean away from people it suspected. That lean was usually outweighed by other inputs, such as the next task. Where the chosen room and the runner-up differed in suspicion, the crew picked the more suspicious one 11,041 times and the less suspicious one 3,348 times. But whenever the weight was negative, a positive suspicion part meant the chosen room was less suspicious than the runner-up, so "keeping an eye on Red" was saying the opposite of what the number meant. The caption now checks the sign, and says "avoiding Red (I suspect them) in Weapons" when the weight is negative.
Wrong way two: relative is not a direction
Fixing the sign was not enough. The part for an input is a difference between two options, and a caption reads as an absolute statement about one.
My fifth league collapsed: its impostors learned to sabotage the reactor every game, and its crew stopped fixing it (written up separately). The replays from that league were full of crewmates "fixing the reactor" while walking the other way. Four replays survive from the end of that run, at games 57,500, 60,000, 62,500 and 65,000; one of them never had a reactor sabotage. In the other three I counted every crew move captioned "fixing the reactor" while a reactor sabotage was running. For each one I compared how many rooms the destination was from the reactor with how many the crewmate's starting room was:
| destination compared with where it stood | moves |
|---|---|
| closer to the reactor | 6 |
| the same distance | 10 |
| farther from the reactor | 45 |
Forty-five of 61 "fixing the reactor" moves went farther away. The explanation was right every time: once its weight is applied, the chosen room did beat the runner-up on the "closer to the fix" input. That can happen in two ways. Either the weight had turned negative, or the runner-up was even farther from the reactor than the chosen room, and I did not count which. Either way, "better than the runner-up" and "toward the reactor" are different claims, and a person watching the video reads the second. That is how a crew that had stopped fixing sabotage looked, in the replays, like a crew that was fixing it.
There are four "closer to X" inputs: the next task, a sabotage fix, the lights fix, and the emergency button. Their captions now say "away from the reactor fix" (and so on) whenever the weight is negative. That covers the first way and not the second: a crewmate that picks the less bad of two moves away from the reactor is still captioned "fixing the reactor". The collapse itself was confirmed by measuring the weight directly, not by reading captions.
Wrong way three: what the bot saw is not what was there
The impostors explain their door sabotage too, and these captions used to give a headcount: "shut the Cafeteria doors", followed by how many crewmates were inside. The morning I found the collapse, I had changed the simulator so that impostors choose doors using only the crewmates they can see, after noticing they had been choosing with knowledge of every room's true headcount. The decision was now right, and the caption still stated what the impostor saw as a fact about the room. It now says "I can see 5 crewmates in there".
Counting across the current replays from the sixth league: 39 door captions give a headcount. In 38 of them the number equals what the impostor could see from where it stood; in 13 the room actually held more crewmates than that. In the game at the end of the run, Yellow shuts the Cafeteria doors at tick 240 (a tick is about half a second) with "I can see 5 crewmates in there", and there are 7. The same rewording went into the crew's "nobody in engine hall", now "I can't see anyone in engine hall", and the impostors' "Lime is alone in Admin", now "I can see only Lime in Admin".
What I would check in any explanation like this
All three failures were in the last step, turning a number into words, and none of them would show up in a test of the arithmetic. An exact attribution still has three hidden qualifiers: it is relative to a specific alternative, it is signed, and it is conditioned on what the agent observed. A caption that drops any of them makes a different claim from the one the numbers support. The check that caught the second failure was the cheap one. Take the words literally, write down what would be true in the world if they were right (the crewmate gets closer to the reactor), and count how often it was.
The simulator, the explanation code and the probes were written by Claude Code at my direction. The reactor captions came to light because I was watching the replays, and the bots' explanations did not match where they were walking. The code is not public. The probes are single scripts over saved leagues and replays, and every figure above is their output.
Top comments (0)