DEV Community: Seth Wheeler

Giving an Open-Weight Language Model the Vote in Among Us Meetings

Seth Wheeler — Fri, 16 Oct 2026 03:00:00 +0000

Meetings are the part of Among Us that a language model looks made for. The crew is trying to find the few hidden impostors among them. When a body is found, everyone talks, states where they were, accuses someone and votes, and whoever gets the most votes is thrown off the ship. My simulator's bots normally vote with small recurrent networks trained on thousands of meetings, which read who saw whom and who accused whom and output a suspicion score for each player. I expected that handing the meeting to a real language model, one that could argue, ask questions and catch lies, would make the crew better at finding impostors. Across six setups it never did.

The setup

The model is gpt-oss:20b, an open-weight model from OpenAI, running locally through Ollama. In a meeting every surviving player, crewmate or impostor, is a separate prompt to the same model, and its reply is that player's speech and vote. A crewmate's prompt contains only what that player could know: where it walked, who it saw and when, any kill or vent it witnessed, and the table of where everyone claims to have been. An impostor's prompt also contains its real movements, marked secret, and permission to lie. Movement, tasks and kills between meetings stay with the trained bots; only the meetings change.

There were three meeting formats. In the first, players speak in rounds and may ask each other structured questions ("where were you 20 seconds ago?"). In the second, each crewmate's prompt also includes the voting network's guesses as an "instinct" (described below). In the third, players hold a free conversation in turns, again with the instinct.

The comparison is the same bots in the same lobby with the trained networks doing all the voting, crew and impostors alike, over 300 games. So the language model replaces both sides' votes at once, and a worse result for the crew could come from worse crew votes or from better impostor talk; the runs below cannot tell those apart. I measured the baselines after the language-model runs, on the current version of the simulator. Five of the six language-model runs used an earlier version, which I mark. The bots come from different leagues (training runs), and each league trained its own voting network.

Eight players, two impostors

These used the bots from my second league. The network baseline did not record wrongly ejected crewmates.

who votes	games	crew wins	impostors ejected per game	crewmates ejected per game
the trained networks (current version)	300	52.0%	0.99	not recorded
gpt-oss, rounds and questions (earlier version)	16	2 (12.5%)	0.06	0.94
gpt-oss, rounds and questions plus the instinct (earlier version)	16	8 (50%)	1.00	0.63
gpt-oss, conversation plus the instinct (earlier version)	16	6 (37.5%)	0.81	0.69

On its own the language model did much worse than the networks. The crew ejected an impostor once in sixteen games and an innocent crewmate almost every game, and won 2 games of 16 where the networks' crew wins about half. That gap is well beyond chance (one-sided Fisher exact test, p = 0.0017), with the caveat that the simulator version differed.

The "instinct" put the voting network's numbers into each crewmate's prompt as a line of text. It opens "Your instinct (a trained model reading everything YOU saw, plus the accusations so far; it picks the right impostor about 70% of the time, so weigh it with the facts):" and then gives each other player's percentage chance of being an impostor. The 70% in that line was my rough estimate when I wrote the prompt, not a measurement. With the instinct and the rounds format, the crew won 8 of 16, indistinguishable from the networks (p = 0.54). If the model mostly voted the way the instinct line pointed, that is the result you would expect; I did not measure how often it did.

Fifteen players, three impostors

In these lobbies the crew wins more often under both kinds of voting, and the ordering held. All three runs use the conversation format with the instinct.

bots	networks voting (300 games, current version)	gpt-oss, conversation plus the instinct
league 2	89.7%	10 of 12 (83%), earlier version
league 4	84.7%	11 of 20 (55%), earlier version
league 6	93.3%	16 of 20 (80%), current version

The league 6 row is the only like-for-like comparison: same bots, same simulator version, same settings, run on the same day. Sixteen of 20 against 280 of 300 is a one-sided Fisher exact p of 0.052, and the 95% interval on 16 of 20 is 58% to 92%. On its own, that row is suggestive rather than settled. The league 4 gap is far larger (p = 0.0025), but between that run and its baseline I changed how meetings treat a running sabotage and what impostors know when they shut doors, so I would not lean on it.

Counting across all six setups: in one the language model was clearly worse than the networks (eight players, no instinct). In the five with the instinct it matched the networks once and fell short four times, by margins that are individually too small or too confounded to be conclusive. I have no setup in which it did better.

What it costs

A 15-player game with gpt-oss meetings took 524 seconds of wall-clock time on average in the league 6 run. Nearly all of that was about 4 meetings' worth of turns, at one model call per player per turn. The networks' votes need no model calls at all. That would be a fair price for better votes, and the votes were not better.

Why I think it happens

This part is my reading rather than a measurement. The networks have an unfair advantage that is also the point: they were trained on the outcomes of thousands of meetings in this simulator, so their suspicion scores are calibrated against who actually turned out to be an impostor here. The language model brings general knowledge of how deception sounds. An impostor here does not sound like anything; it names a room, and the only way to catch the lie is to check that claim against who could have seen that room. The prompts contain what each player saw. Nothing in the results suggests the model made that check reliably, and with the network's numbers in the prompt it did no better than the network.

An earlier benchmark of mine, which ranked language models on spotting liars in Mafia, found first-day accusations at chance. There, on the first day, nothing about the liars is knowable yet. Here the evidence is in each crewmate's prompt, which makes falling short of a small network harder to excuse.

The simulator, the meeting prompts and the measurement scripts were written by Claude Code at my direction. The code is not public. Every figure above comes from the result files the runs wrote, and the significance tests are recomputed from those counts.

Rebuilding the Among Us Floor Plan From the Game's Own Collision Data

Seth Wheeler — Thu, 15 Oct 2026 03:00:00 +0000

I train Among Us bots in a simulator of the game. A crew walks the ship doing tasks and votes out suspects in meetings. Hidden impostors kill, travel through vents (tunnels only impostors can use) and sabotage the ship, including shutting a room's doors for 10 seconds. The bots learn by self-play in leagues, each league a roughly ten-hour training run that starts from the bots the previous league ended with.

The simulator decides where players can stand with a grid of 126 by 70 tiles. The grid comes from the map file of an open-source web clone of the game, MIT-licensed and written by a third party. I used it because it was there and it matched the ship's layout room for room. It is also wrong, in a way I only saw once the replays were drawn on the real game's art: a crewmate jogging along the top of a corridor with its feet on the corridor's back wall. Measured against the game's own collision data, the clone's floor is 1.75 times too big.

Where the real floor comes from

Among Us is a Unity game, and the 2018 Android build ships its levels as Unity asset files inside the APK (the Android app package). The ship is a single prefab (a saved Unity object tree) called ShipStatus. Alongside the artwork, it carries the colliders the physics engine uses to stop players walking through things. I read them out of the APK with UnityPy, a third-party Python library for Unity asset files, without running the game. There are 79 wall outlines, which are edge colliders along the edge of every room and corridor. There are also 33 solid obstacles: the Cafeteria tables, the MedBay beds, the engines, Navigation's chairs and a few task consoles.

The player prefab supplies the other number that matters. A player is a 0.7-scaled object with a circular collider of radius 0.2234 at its feet. So the centre of a player's feet can never get closer than 0.156 units (Unity's unit of distance) to a wall or an obstacle. One unit is 2.646 of my simulator's tiles (from aligning a render of the ship with the clone's map), which makes the clearance 0.41 tiles.

The rebuild draws every outline and obstacle into a grid eight times finer than the simulator's tiles. It flood-fills the inside of the ship from a known floor point, then removes everything within the foot clearance of a wall or obstacle. A tile is walkable if its centre survives. Two details mattered. The outlines had to be drawn three sub-cells thick, because two outlines meeting at a corner can leave a gap finer than one sub-cell; the first version's flood fill escaped through one and filled the outside of the ship. And 56 tiles of floor that were not connected to the rest were dropped. The build would also force each of the 14 vent grilles onto the floor, since the game snaps a player's feet onto a grille to enter it, but in the final build none needed it.

How wrong the old floor was

	walkable tiles
the clone's map	3,314
the game's colliders	1,894
on both	1,867
on the clone's map only	1,447
on the game's colliders only	27

Every room shrank, unevenly. Eight of the 21 rooms and corridors, the worst and best by the ratio of old to new walkable tiles:

room	old	real	old / real
O2	98	32	3.06
Weapons	180	78	2.31
Shields	133	58	2.29
Upper Engine	110	48	2.29
Cafeteria	658	351	1.87
MedBay	142	105	1.35
central hall	253	196	1.29
Storage	162	147	1.10

The pattern is what the replays suggested. In a corridor, the clone's map counts the whole drawn height as floor. That includes the band of wall panels along the top, which the game draws as the back wall, so a corridor loses that band plus the foot clearance on each side. Rooms full of furniture lose the furniture; on the clone's map you could walk straight across the Cafeteria tables. Storage, an open room with crates mostly along its edges, barely changed.

Before switching, I checked that nothing the simulator depends on fell off the new floor. All 14 vents are on walkable tiles. Every task station moved by at most 3 tiles, to the nearest real floor in its own room. Each of the 7 rooms with doors still cannot be walked into from outside while its doors (13 in all) are shut. Impostors can still vent into some of them, as in the game.

How much it changed the game

I do not know precisely, which is the honest answer. The bots were trained on the old floor and played on the new one without retraining. I played 40 games on each floor with the bots from my fourth league, 15 players and 3 impostors, using a fixed random seed so the run repeats exactly:

floor	crew wins	tasks done	kills per game	vents per game
clone's map	36 of 40	69%	5.8	3.3
game's colliders	34 of 40	65%	6.0	3.6

Two games in 40 is within what a different seed could produce. But an earlier unseeded test, also 40 games each, with the same league's bots taken at an earlier checkpoint and an earlier version of the simulator, measured the crew at 78% on the old floor and 68% on the new one. Forty games cannot separate a 2-point effect from a 10-point one, so the effect is somewhere in that range. Either way it is far smaller than a factor of 1.75 in walkable floor suggests. My guess at why is that the bots decide between rooms rather than between tiles. A room's neighbours, its vents and the walking distance between rooms are close to unchanged, so the floor mostly changes where a player stands inside a room rather than where it goes.

I switched anyway. The simulator is meant to model the real game, and the old floor was visibly wrong in every replay, whatever it did to the win rate. Training since then has used the real floor, and the old one remains available for comparison with older runs. My fifth league, the first to train on the new floor, later collapsed for reasons I could not separate from the floor change, which is a separate post.

The simulator, the extraction and the rebuild were written by Claude Code at my direction, and the corridor wall-walking was something I spotted in the replays. The code is not public, and the extracted game data and art stay on my machine. The figures above come from the rebuild script's output and a seeded comparison script; anyone with the 2018 APK and UnityPy can read the same colliders.

What I take from it is that a simulator can be wrong by a factor of 1.75 in a quantity that sounds fundamental and still produce similar outcomes. Outcome metrics are a poor check on whether a simulator is faithful. The check that found this was looking at the picture.

Asking Trained Among Us Bots Why They Moved

Seth Wheeler — Wed, 14 Oct 2026 03:00:00 +0000

My Among Us bots can explain themselves. Every time a crewmate picks a room to walk to, the replay records why, and the exported videos show it as a thought bubble over the player: "→ Admin: next task: Swipe Card (Admin); avoiding Pink (I suspect them) in Weapons". For the crew, the arithmetic behind those captions is exact to about fourteen decimal places. The captions were still wrong in three different ways, and one of them made a collapse in training look, on video, like normal play.

Why the explanation can be exact

The bots are trained by self-play, in leagues: each league is a roughly ten-hour training run that starts from the bots the previous league ended with. A crewmate's walking network does not output a choice directly. At each step it is offered the neighbouring rooms, and staying put. Each option is described by the same dozen inputs, such as how much closer it gets to the next task, how many people are visible there, how suspected the most suspected person there is, and how much closer it gets to fixing a running sabotage. The network looks at what the crewmate has seen and outputs one weight per input. Each option's score is the plain weighted sum of its inputs, and the highest score is the most likely pick.

That structure makes "why A and not B" a question with an exact answer. If option A scores s_A and the runner-up B scores s_B, then:

s_A − s_B = Σ_i (input_A,i − input_B,i) × weight_i

So the lead A has over B splits into one part per input, and the parts add back up to the lead with nothing left over. The caption takes the two largest positive parts and turns each into a phrase. There is no approximation step and no separate model of the bot's behaviour, which most explanation methods for neural networks need. (The impostors' walking adds a learned map term that is not a sum over named inputs, so their captions lump that part together as "what it has learned about the map". Everything below about exactness is about the crew.)

To check that the split really is exact in the code and not just on paper, I wrote a probe. It plays 30 games with a league's saved bots, using a fixed random seed so the run repeats exactly, and for every crew move decision it compares the score gap with the sum of the parts:

crew	decisions	largest error in the split, in score units
league 3	80,550	7.11e-15
league 4	40,433	7.11e-15
league 6	34,226	1.42e-14

The median gap between the chosen option and the runner-up was between 6.3 and 9.8 in the same units, depending on the league, so these errors are floating-point rounding. The arithmetic was never the problem.

Wrong way one: the sign of the weight

The first captions had one phrase per input, chosen when that input's part was large and positive. For the suspicion input the phrase was "keeping an eye on Red". That reads as "I am going where my suspect is", and it is only true if the weight on suspicion is positive.

For the league 3 crew it mostly was not. Over 19,085 decisions with a suspected player in view, its weight on suspicion averaged -3.94 (10th percentile -6.73, 90th -0.45), so it was negative in at least nine decisions in ten: a lean away from people it suspected. That lean was usually outweighed by other inputs, such as the next task. Where the chosen room and the runner-up differed in suspicion, the crew picked the more suspicious one 11,041 times and the less suspicious one 3,348 times. But whenever the weight was negative, a positive suspicion part meant the chosen room was less suspicious than the runner-up, so "keeping an eye on Red" was saying the opposite of what the number meant. The caption now checks the sign, and says "avoiding Red (I suspect them) in Weapons" when the weight is negative.

Wrong way two: relative is not a direction

Fixing the sign was not enough. The part for an input is a difference between two options, and a caption reads as an absolute statement about one.

My fifth league collapsed: its impostors learned to sabotage the reactor every game, and its crew stopped fixing it (written up separately). The replays from that league were full of crewmates "fixing the reactor" while walking the other way. Four replays survive from the end of that run, at games 57,500, 60,000, 62,500 and 65,000; one of them never had a reactor sabotage. In the other three I counted every crew move captioned "fixing the reactor" while a reactor sabotage was running. For each one I compared how many rooms the destination was from the reactor with how many the crewmate's starting room was:

destination compared with where it stood	moves
closer to the reactor	6
the same distance	10
farther from the reactor	45

Forty-five of 61 "fixing the reactor" moves went farther away. The explanation was right every time: once its weight is applied, the chosen room did beat the runner-up on the "closer to the fix" input. That can happen in two ways. Either the weight had turned negative, or the runner-up was even farther from the reactor than the chosen room, and I did not count which. Either way, "better than the runner-up" and "toward the reactor" are different claims, and a person watching the video reads the second. That is how a crew that had stopped fixing sabotage looked, in the replays, like a crew that was fixing it.

There are four "closer to X" inputs: the next task, a sabotage fix, the lights fix, and the emergency button. Their captions now say "away from the reactor fix" (and so on) whenever the weight is negative. That covers the first way and not the second: a crewmate that picks the less bad of two moves away from the reactor is still captioned "fixing the reactor". The collapse itself was confirmed by measuring the weight directly, not by reading captions.

Wrong way three: what the bot saw is not what was there

The impostors explain their door sabotage too, and these captions used to give a headcount: "shut the Cafeteria doors", followed by how many crewmates were inside. The morning I found the collapse, I had changed the simulator so that impostors choose doors using only the crewmates they can see, after noticing they had been choosing with knowledge of every room's true headcount. The decision was now right, and the caption still stated what the impostor saw as a fact about the room. It now says "I can see 5 crewmates in there".

Counting across the current replays from the sixth league: 39 door captions give a headcount. In 38 of them the number equals what the impostor could see from where it stood; in 13 the room actually held more crewmates than that. In the game at the end of the run, Yellow shuts the Cafeteria doors at tick 240 (a tick is about half a second) with "I can see 5 crewmates in there", and there are 7. The same rewording went into the crew's "nobody in engine hall", now "I can't see anyone in engine hall", and the impostors' "Lime is alone in Admin", now "I can see only Lime in Admin".

What I would check in any explanation like this

All three failures were in the last step, turning a number into words, and none of them would show up in a test of the arithmetic. An exact attribution still has three hidden qualifiers: it is relative to a specific alternative, it is signed, and it is conditioned on what the agent observed. A caption that drops any of them makes a different claim from the one the numbers support. The check that caught the second failure was the cheap one. Take the words literally, write down what would be true in the world if they were right (the crewmate gets closer to the reactor), and count how often it was.

The simulator, the explanation code and the probes were written by Claude Code at my direction. The reactor captions came to light because I was watching the replays, and the bots' explanations did not match where they were walking. The code is not public. The probes are single scripts over saved leagues and replays, and every figure above is their output.

Training Among Us Bots: The Crew That Stopped Fixing the Reactor

Seth Wheeler — Tue, 13 Oct 2026 10:09:48 +0000

I train Among Us bots by self-play: the crew and the impostors are both learners, and they play tens of thousands of games against each other and against saved copies of their earlier selves. I run the training in leagues, each one a roughly ten-hour run that starts from the bots the previous league ended with. The fifth league looked healthy for ten thousand games and then fell over. Between game 10,000 and game 12,500, the impostors' win rate in their own training games went from 50% to 75%, the crew's tasks went from 70% done to 41%, and the impostors almost stopped killing (6.52 kills a game down to 2.00). They had found something better than killing: sabotaging the reactor. The run went on for another 52,500 games and never recovered.

The game, briefly

Among Us puts a crew of crewmates and a few hidden impostors on a spaceship. The crew wins by finishing its tasks (each a short errand at a console, done in steps) or by voting every impostor out in meetings. The impostors win by killing until they match the crew in number. They can also travel through vents, shut a room's doors for 10 seconds, and sabotage the ship. Lights and comms sabotages are nuisances; the reactor and O2 are critical. When the reactor is sabotaged, two crewmates must reach it and hold it within 30 seconds, or the reactor melts down and the impostors win (O2 is the same idea with two panels in different rooms). My simulator follows the 2018 version of the game on its first map, the Skeld, with 15 players and 3 impostors.

A crewmate's walking is decided by a small network. At every step it is offered the rooms next to it, and staying put, and it scores each option as a weighted sum of a dozen inputs. Two of those inputs matter here: "this room is closer to my next task" and, during a sabotage, "this room is closer to the fix". Both are measured in the same unit, rooms along the ship's corridors, so a 1 means one room closer. The weights are not fixed: a recurrent network computes them each step from what the crewmate has seen, and training (PPO, a standard reinforcement learning method) adjusts that network. The option with the highest score is the most likely pick, so the weight on an input is a direct reading of how much the crew cares about it, and two weights on the same unit can be compared.

What the crew stopped doing

Sabotaging the reactor should not keep working against a crew that fixes it, so the obvious suspect was the crew. I wrote a probe that loads a league's saved crew and plays 40 games against the simulator's scripted impostors, so every crew faces the same opponents. Whenever a reactor or O2 sabotage is running, it records the crew network's weights on the two inputs above, and whether the crewmate picks the option that actually gets closer to the fix. It also counts how many of those sabotages get fixed. The probe uses a fixed random seed, so it gives the same numbers on every run.

crew	weight on "closer to the fix" (10th to 90th percentile)	weight on "closer to my next task"	steps toward the fix	reactor/O2 sabotages fixed	games lost to a meltdown
league 4 (the run before)	5.79 (4.47 to 7.57)	1.15	90% of 494	15 of 18	0 of 40
league 5 (collapsed)	1.62 (-0.23 to 3.47)	6.60	32% of 399	0 of 7	3 of 40

The weights are averages over every crewmate decision taken while a reactor or O2 sabotage was running, and the step counts are those decisions. League 4's crew, during a sabotage, valued getting one room closer to the fix five times as much as getting one room closer to its next task. League 5 started from league 4's crew and turned that around: it valued its next task 4.1 times as much as the fix. A crew that steps toward the fix one time in three does not get two people to the reactor in 30 seconds. League 5's crew met fewer sabotages (7) because its games went differently; of the seven, three melted down, and the other four were still running when the game ended some other way, mostly by the impostors outnumbering the crew.

Why it unlearned it

The crew's reward is the game result (1 for a win, 0 for a loss), plus 0.3 times the share of the crew's tasks that got done, plus 0.1 for being alive at the end, plus 0.15 for every task step a crewmate completes itself. Fixing a sabotage appears nowhere in that list. It pays only indirectly, by keeping the game going so that the result and the tasks can still pay later.

That is enough while fixes usually succeed. Once the impostors sabotage the reactor every game and some fixes fail, walking to the reactor starts to look like a bad trade. A crewmate who goes gives up task steps that pay 0.15 each, and if the fix fails anyway, the game is lost whatever it did. Learning then lowers the weight on the fix, fewer crewmates go, more fixes fail, and the weight drops further. The impostors' side of the same loop is simpler. Every failed fix is a win, so they sabotaged more (1.77 reactor sabotages a game at 12,500 games, against 0.28 at 10,000) and killed less, because killing had stopped being necessary.

League 4 had reached the same edge and come back. Its impostors used the reactor from early on (0.71 sabotages a game at 2,500 games) and peaked at 1.25 at game 10,000, then dropped to 0.04 by game 20,000, so its crew must have kept fixing often enough for the trade to stay worth it. League 5 differed from league 4 in two ways: a new input to the crew's walking network ("this room is closer to where I last saw the player I suspect most"), and a floor plan of the ship rebuilt to match the real game, with narrower rooms. I cannot tell which of them, if either, tipped it over. What I can say is that the reward never pointed at the fix directly. The evaluations show the result. Every 10,000 games each league plays 100 games against a fixed set of opponents, and at every checkpoint after the start, league 5's crew beat the league 4 impostor less often than league 4's own crew did (54% against 78% at game 60,000).

Paying for the fix

The sixth league started again from league 4's bots and added the missing term. Every crewmate standing at the fix when a reactor or O2 sabotage is fixed gets 0.3, about two task steps' worth, and lights and comms pay 0.1. I also put league 5's reactor-spamming impostor in the pool of opponents, so the new crew had to learn to beat it. Finally, I logged the crew's fix rate in every 2,500-game training window, so that a repeat would show up early.

The same probe on league 6's crew after 33,916 games:

crew	weight on "closer to the fix" (10th to 90th percentile)	weight on "closer to my next task"	steps toward the fix	reactor/O2 sabotages fixed	games lost to a meltdown
league 6	22.68 (17.96 to 27.68)	3.24	99% of 747	30 of 34	0 of 40

During a sabotage, league 6's crew values the fix seven times as much as its next task. In training the crew fixed between 85.6% and 93.9% of the reactor and O2 sabotages in every window. The impostors gave the reactor up almost entirely, at 0.02 reactor sabotages a game or fewer in every window. At the final evaluation the crew beat league 5's reactor-spamming impostor in 93 of 100 games.

What this does not show

League 6 changed more than the reward. It restarted from league 4 rather than continuing league 5, and between the two runs I fixed several simulator bugs that bear on sabotage. Meetings had been ending a running sabotage instead of pausing it. Impostors could see how many crewmates were in a room they were about to lock. Doors could be shut during a sabotage, which the 2018 game does not allow. So the fair statement is that the fix reward plus those changes produced a crew that fixes sabotage, not that the reward alone did. The 30 of 34 is also from one seeded probe of 40 games; a different seed gives a different count.

It also overshot. League 6's impostors never found a new strategy. They won 17% to 25% of their training games and beat their own crew in only 13 of 100 evaluation games. Human players are the other way round: impostors win 57.69% of the time, according to the game's developers in December 2020. Getting the bots there by training, rather than by handicapping either side, is the next run.

The simulator, the training leagues and the probe were written by Claude Code at my direction, and several of the bugs above were ones I found by watching replays. The code is not public. The probe is a single script that loads a saved league and prints the table rows above, and every other figure here comes from the leagues' own training logs.

The general lesson is about the reward rather than the game. A behaviour that only pays off through a delayed outcome is the first thing to go when an opponent starts punishing it, because the learner sees the cost every time and the benefit only sometimes. If there is an action your agents must keep doing under pressure, it needs a term of its own.

Giving Evolved Monopoly Players a Memory of Past Trades

Seth Wheeler — Mon, 12 Oct 2026 03:00:00 +0000

My theory was that a Monopoly player who remembers trades should beat one who does not. If a player tried to sell me the brown properties for $60 two turns ago, I can offer $45 now and expect a yes. I evolved two populations of players with identical training budgets, one with a memory of trades and mortgages and one without. The memory players won 56.1% of the games where two of each sat at the table. That looked like the answer until a third population, which barely used its one new input, beat the no-memory players by the same margin in the same two-against-two games.

The setup

The players are neural networks evolved with NEAT (NeuroEvolution of Augmenting Topologies). NEAT keeps a population of networks, and each generation the better ones breed and mutate, changing weights and adding connections and nodes. It groups similar networks into species so that new structure is not wiped out before it is tuned; these populations used a version of that grouping adjusted for large starting networks, described in the previous post in this series.

The game is a simplified four-player Monopoly written for the experiment. It has houses and hotels with the real limit of 32 and 12, mortgages, and all 32 Chance and Community Chest cards. Players can offer to buy or sell a property for cash at 0.5 to 2.5 times its printed price, or swap one property for another. There are no doubles and no auctions, and a bankrupt player's properties go back to the bank. A game still going after 200 rounds goes to the richest player; in the final generation of training, 84% of the memory population's games and 93% of the other's ended in bankruptcy instead.

Each network sees 44 inputs describing the board and the decision in front of it. It answers every question the game asks (buy this? build? accept this offer? offer how much?) with a yes or no and an amount.

With memory: 13 of the inputs summarise history. They include how often this opponent has accepted my offers, whether they rejected an offer for this property and at what price, whether they tried to sell it in the last 20 rounds and for how much, and recent mortgages. Two more inputs read back two memory slots, which the network writes with two extra outputs at every decision.
Without memory: the identical network with those 15 inputs frozen at "nothing happened" and the slots never written.

Both populations had 100 networks and trained for 30 generations by self-play, meaning the networks' opponents were other networks rather than scripted players. Each network played 200 games per generation, and its score was the fraction it won. Most games were against its own population; the rest included members of a hall of fame, the best network from each earlier generation, so that the population kept beating old strategies and not just its current neighbours. Both populations started from the same random seed, so the first generation was identical.

The champion of each population is its best-scoring network in the final generation. Each champion then played 2,000 games in each of several line-ups: against three copies of a scripted player, against three copies of the other population's champion, and two against two. One scripted player buys and builds everything it can. The "sensible" one buys and builds only while it keeps $300 in cash, and proposes a trade only to complete a color group of its own. With four players, 25% is an even share of wins, and a 2,000-game figure carries a standard error of about one percentage point.

The first result

line-up	champion's share of wins
with memory, against 3 "sensible" players	27.8%
without memory, against 3 "sensible" players	29.1%
with memory, against 3 players that buy everything	25.4%
without memory, against 3 players that buy everything	24.6%
1 with memory against 3 without	33.0%
1 without memory against 3 with	21.6%
2 with memory against 2 without, games won by the memory pair	56.1% (50% is even)
with memory, all 15 memory inputs blanked, against 3 "sensible" players	20.8%

Against the scripted players memory made no difference. I expected that: they rarely trade and never try to sell, so there is little about them to remember. Head to head, the memory player won a third of its games and the memory pair won 56.1%. Blanking the memory champion's memory inputs dropped it from 27.8% to 20.8% against the scripted players, so it depended on them. A network fed inputs it never saw in training can fail for reasons unrelated to memory, though.

The lowball offers

I checked the motivating example directly. Four copies of each champion played 1,000 games at one table, and I split their offers to buy by whether the owner had tried to sell that property in the last 20 rounds. The price is the average offer as a multiple of the printed price.

champion	offers after the owner tried to sell	all other offers
with memory	49 offers at 0.52 times the price, 40 accepted (82%)	69,155 at 0.51 times, 8,483 accepted (12%)
without memory	74 offers at 1.33 times the price, 0 accepted	49,965 at 1.05 times, 1,073 accepted (2%)

The no-memory player cannot see sale attempts, so its split only reflects which properties tend to be put up for sale. The memory player did not offer less after a sale attempt. It already offered close to the 0.5 floor on everything, so the floor hides any price effect. What changed was acceptance: 82% against 12%. That fits the theory, but it rests on 49 offers. It is also partly a statement about the sellers, who were copies of the same champion and had recently tried to sell.

Over training, both populations learned to refuse nearly every offer to buy their property. The acceptance rate fell from 44.7% in the first generation to 5.0% in the thirtieth with memory, and from 34.3% to 5.3% without.

A memory that predicts

Hand-picked memory features are a guess about what matters. The next test was whether a model reading the raw history could predict an opponent's answer better. I recorded every trade offer from 1,000 games between the two champions and split them by game: 700 games for training the predictors, 300 held back for testing them on games they had never seen. The training games held 137,561 offers for the first seed and 149,307 for the second, with 4.0% and 3.5% accepted. Each predictor used only information the player making the offer could have seen, and was trained and tested on samples balanced to half accepted and half rejected.

The four predictors differ only in what they are given. The first three are ordinary feed-forward networks. The fourth is an LSTM, a recurrent network that reads events one at a time and carries a state between them. The score is AUC: the chance that the predictor ranks a random accepted offer above a random rejected one, so 0.5 is guessing and 1 is perfect.

predictor	test AUC, seed 1	test AUC, seed 2
the offer alone, no history	0.873	0.875
the offer plus the opponent's last 20 events, as 300 separate inputs	0.963	0.964
the offer plus 12 of the hand-made memory features	0.965	0.961
the LSTM reading the last 20 events in order, then the offer	0.981	0.982

History matters a lot: without it the predictor managed 0.87, and with it every model was above 0.96. The flattened events and the hand-made features came out level, and the LSTM beat both on both seeds, so reading the events as a sequence added something that having them all at once did not. On the balanced test sets the LSTM was right 93.6% of the time on both seeds. (The hand-made set is 12 features rather than 13 because the prediction script left out one of them, the proposer's own recent mortgages.)

This table is a rerun. A review of this post found that in the first version of the prediction script, two of the 15 numbers describing each event, "is this the same property?" and "how long ago was it?", sat after a code comment on the same line and were always 0. The first run gave the LSTM 0.978 and 0.979 and the flattened events 0.953 and 0.959, which had made the flattened control look worse than the hand-made features. With the fix it is not.

The population that barely used its input

Next I trained a third population, C, with the same budget and seed as the first two. Its hand-made memory inputs were frozen and its memory slots switched off, as in the no-memory population. One input, "how often this player accepts my offers", was replaced by the LSTM's predicted probability that the current offer would be accepted. That input was live only when the network was deciding whether to propose a trade.

line-up	C	memory population, for reference
against 3 "sensible" players	25.0%	27.8%
1 against 3 of the no-memory population	28.2%	33.0%
2 against 2 of the no-memory population, pair's share	55.8%	56.1%
2 C against 2 of the memory population, C pair's share	51.4%

In two-against-two games C beat the no-memory population by the same margin as the memory population, and it held even with the memory pair. One against three it did less well, 28.2% against 33.0%. Then I froze its LSTM input at 0.5 to see how much it depended on it. Against the scripted players it went from 25.0% to 24.9%, and one live C against three frozen copies won 25.9%. C was barely using its one new input.

There are at least two reasons it might have ignored a good predictor. The network chooses its offer price in the same step that reads the prediction, so the prediction was always for an offer at the printed price, not the one about to be made. And C used the LSTM from the first, buggy run: in play it received the two event features that had been 0 in its training, so its predictions during C's games were not the ones measured in the table. I have not retrained C.

That changes what the first result means. In practice C is one more population with next to no memory. It differs from the no-memory population in one input it barely uses and in the path its evolution happened to take. Yet in the two-against-two games it beat the no-memory champion 55.8% to the memory champion's 56.1%. One reading is that the no-memory run simply produced a weaker champion than the other two runs did. With one training run per setup, I cannot separate that from a genuine benefit of memory. The 2,000-game line-ups measure these particular champions precisely. They say nothing about how much champions vary from one training run to the next, and that variation is what needed measuring.

What would settle it

Several seeds per population. Three or more independent training runs for each setup, compared by the spread of their head-to-head results rather than one champion against one champion.
Ask the predictor about the price being offered. Giving the network the predicted acceptance at several prices would make the prediction usable for the decision that matters.
Retrain C with the fixed predictor, and retrain the predictor on C's own games. The LSTM learned from the first two champions, whose habits C's opponents did not share.

The code is a learning project and is not public. I wrote the NEAT implementation. Claude Code wrote the Monopoly simulator, the training and comparison scripts and the prediction experiment, and the LSTM code came from another Claude Code session. Every figure here comes from those scripts' saved output.

A predictor that was right 93.6% of the time did not make a better player, and a two-against-two margin that looked like the effect of memory showed up in a population with next to none. The experiment's direct evidence for the theory is 49 lowball offers, and I would want many more before believing it.

Why NEAT Stopped Forming Species When the Starting Genome Grew

Seth Wheeler — Sun, 11 Oct 2026 03:00:00 +0000

NEAT, short for NeuroEvolution of Augmenting Topologies, evolves a population of neural networks, adding connections and nodes as it goes. A newly added node usually makes a network worse until its weights are tuned, so NEAT sorts the population into species. A network with a new node then competes with its relatives rather than with the whole population, which gives the new structure time to pay off. I had it forming species on XOR and assumed the same code would do the same on a larger problem. On a Monopoly player with 180 starting connections it formed exactly one species for ten generations straight. Nothing about the run looked wrong. It ran, mutated and bred as usual, and the species count simply stayed at 1.

How NEAT decides two networks are different

A genome is the list of connections that describes one network, and each connection in it is a gene. Every gene carries an innovation number recording when that connection first appeared anywhere in the population, which lets two genomes be lined up gene by gene. Every genome starts as a copy of the same minimal network, with each input and a bias wired straight to each output. For XOR that is 2 inputs plus a bias to 1 output, 3 genes. For my Monopoly player it is 44 inputs plus a bias to 4 outputs, 180 genes, and every member of the population shares all of them. The original paper measures the difference between two genomes as a compatibility distance:

δ = c1·E/N + c2·D/N + c3·W̄

E counts excess genes and D counts disjoint genes. If one genome has innovations 1 to 5 and the other has 1 to 3 and 7, then 4 and 5 are disjoint (gaps in the middle) and 7 is excess (past the end of the other genome). W̄ is the average weight difference over the genes both share, and N is the gene count of the larger genome. Each generation, a network joins the first species whose representative, a random member of that species from the previous generation, is closer than a threshold; if none is, it founds a new species. Because it is compared with a representative rather than with every other member, a species can hold pairs that are further apart than the threshold. I used the paper's XOR coefficients of 1, 1 and 0.4 and a threshold of 3. The paper also allows N to be 1 when both genomes have fewer than 20 genes, and my implementation does that.

The division by N is the part that matters here. It makes two large genomes that differ by a few genes count as similar. That makes sense when every gene in a genome was added by evolution, as it nearly is on XOR. It makes much less sense when most of the genes were in the starting genome and every member of the population has them.

The measurement

I ran the project's NEAT for ten generations on two problems, twice each, from one script with a single fixed random seed. Then I sampled 800 random pairs from each final population and took the median distance. The last column lists the species count after each of the ten generations.

problem	starting genes	N counts	median distance	species per generation
XOR	3	all genes	7.203	1 1 3 4 6 7 8 8 10 13
XOR	3	genes added since the start	8.266	1 1 1 4 4 5 7 11 13 14
Monopoly	180	all genes	0.368	1 1 1 1 1 1 1 1 1 1
Monopoly	180	genes added since the start	6.379	1 1 1 3 7 6 6 6 6 6

The XOR population had 150 networks and was scored on the four XOR cases. The Monopoly population had 100 networks, each with 44 inputs and 4 outputs. Each was scored on eight games against three scripted players that buy everything they land on.

After ten generations the average Monopoly genome had 181.6 genes, only 1.6 more than it started with. Two genomes typically differed by a handful of genes out of about 180. Even five differing genes, divided by 180, add under 0.03 to the distance, so the median of 0.368 is almost entirely the weight term. For the weight term alone to reach 3, the shared weights would have to differ by 7.5 on average, and they never did. The 90th percentile distance was 0.403, so almost no pair came anywhere near 3.

The change

The question was whether distance should account for the starting genome's size, and the change I made was small. N now counts the genes added since the start rather than all of them:

let added = Math.max(this.connections.length, other.connections.length) - baseGenes;
let n = (added < 20) ? 1 : added;

baseGenes is the starting genome's gene count. The NEAT object records it when it is created and passes it to every distance call during speciation, and with a default of 0 the function behaves exactly as before. The Monopoly genomes had added about three genes on average, far short of 20, so N fell to 1 and each added gene counted fully. The median distance went from 0.368 to 6.379, and the population held six species from the sixth generation on.

On XOR the change does nothing, because the genomes have fewer than 20 genes and N is 1 either way. A test in the project checks that on 100 random pairs of small genomes. The two XOR rows still differ, 7.203 against 8.266 and 13 species against 14, because the seed was set once for the whole script, so the second run drew different random numbers from the first. That difference is what ordinary run-to-run variation looks like. The Monopoly gap is 17 times the distance and five extra species, which variation of that size does not explain.

A unit test shows the same effect on a smaller network without a full run. It builds a 60-network population from a 54-gene starting genome, grows half of it, and speciates: 31 species with the change and 1 without.

What this does and does not show

It shows that the species mechanism was switched off, not that turning it back on helps. I have not compared Monopoly play from a population with 6 species against one with 1. The paper argues that speciation protects new structure long enough to tune it. Whether that matters here is a separate experiment.
The species count can run away the other way. With N at 1, every added gene adds a full point to the distance, so a population that keeps adding structure keeps splitting. XOR reached 14 species out of 150 networks by generation 10. Spread across dozens of species, 150 networks leave each one too small to evolve much. Some NEAT implementations adjust the threshold as they go to hold the species count near a target; mine does not yet.
One seed per configuration. The Monopoly result is large enough that I do not expect another seed to reverse it, but I have not run one.

The code is a learning project and is not public. I wrote the NEAT implementation; Claude Code wrote the tests I built it against and the script that produced the table above. The fix is two lines. What I would check first in any other NEAT port is whether its distance formula assumes a small starting genome. This failure produces a run that looks like it is working, with the one counter nobody watches stuck on 1.

Hill Climbing, Backpropagation and NEAT on the Same XOR Problem

Seth Wheeler — Sat, 10 Oct 2026 03:00:00 +0000

I expected backpropagation to learn XOR with the least work of the three methods I had built. It computes the direction every weight should move instead of guessing, so it seemed obvious. On a small network with 2 inputs, 3 hidden neurons and 1 output (a 2-3-1 network) it needed 9,268 to 11,124 network runs at learning rate 0.5. A hill climber that nudges every weight at random, and keeps the nudge only if the error drops, needed 1,512 to 1,648. Raising the learning rate to 2 brought backpropagation down to 3,028 to 3,372, still about twice the hill climber's work. The order only flipped once the problem grew to a full adder with two outputs.

The setup

XOR as a learning problem is four examples: the inputs 00, 01, 10 and 11 should produce 0, 1, 1 and 0. It is the classic first test because no single neuron can learn it, so the network needs a hidden layer. Each neuron multiplies its inputs by weights, adds a bias, and squashes the sum through a sigmoid, an S-shaped curve that flattens out near 0 and 1. Learning means finding weights and biases that make the outputs match.

All three learners live in one Node.js project, which is a learning exercise and is not public. I wrote the network, the hill climber, the backpropagation and the NEAT implementation myself. Claude Code wrote the test suite I implemented against and the comparison script whose output is quoted below, so the numbers come from a harness I did not write but can rerun.

Hill climbing: every round, add a random amount between -0.5 and +0.5 to every weight and bias. Run the four XOR cases, and restore the previous weights if the total error did not go down.
Backpropagation: calculus tells each weight which way to move to reduce the error, and the learning rate sets how far it moves. The weights are updated after every example, on the same 2-3-1 network. One pass through the four examples is an epoch.
NEAT: NeuroEvolution of Augmenting Topologies. A population of 150 networks, each described by a genome listing its connections, starts with no hidden nodes. Each generation, the better networks breed and mutate, adding weights, connections and nodes. Networks are grouped into species so that one with a freshly added node, which usually makes it worse at first, only competes with similar networks while its new weights get tuned.

Every method stops at the same target: total squared error below 0.04 over the four cases. (The two-output problem later on uses 0.08.) Work is counted in one unit that means the same thing for all three. A network run is one forward pass on one input. The hill climber spends one evaluation, four runs, per round. Backpropagation spends an evaluation plus one training pass per example, eight runs, per epoch. NEAT spends four runs per genome per generation.

Each configuration ran ten independent learners, and each figure is the median over the ones that reached the target. I ran each configuration three times, so a range is the spread of three medians.

XOR

method	median network runs, three repeats	learners that reached the target
hill climbing, batch A	1,512 to 1,648	9 or 10 of 10
backpropagation, learning rate 0.5, batch A	9,268 to 11,124	10 of 10
NEAT, batch A	64,800 to 67,800	10 of 10
hill climbing, batch B	1,340 to 1,544	10 of 10
backpropagation, learning rate 2, batch B	3,028 to 3,372	9 or 10 of 10
NEAT, batch B	65,400 to 69,000	10 of 10

Each batch is three repeats of all three methods, and the only setting that differs between them is backpropagation's learning rate. The hill climber and NEAT rows in the two batches are the same method measured twice, which shows how much the figures move on their own.

The explanation I have is that the network is tiny. The 2-3-1 network has 13 parameters: three hidden neurons with two weights and a bias each, plus an output neuron with three weights and a bias. In 13 dimensions a random step has a fair chance of pointing downhill. A learner that tries a direction and throws it away when it is wrong gets there in surprisingly few evaluations.

Backpropagation never takes a step in the wrong direction; its steps were just small. At learning rate 0.5 it crawled across the flat regions of the sigmoids near 0 and 1, and took 1,158 to 1,390 epochs. Quadrupling the rate cut that to 378 to 421 epochs. The method was fine, and my setting of it was the problem.

The full adder

A full adder adds three bits and outputs a sum and a carry. That is eight cases and two outputs instead of four and one. Both fixed-shape learners used a 3-4-2 network, which has 26 parameters, and backpropagation kept learning rate 2.

method	median network runs, three repeats	learners that reached the target
hill climbing	10,064 to 12,120	9 or 10 of 10
backpropagation	3,640 to 3,928	10 of 10
NEAT	205,200 to 218,400	8 or 10 of 10

The order reversed. Backpropagation now needs about a third of the hill climber's work. My explanation is the same one run backwards: a random step in 26 dimensions is far less likely to help than one in 13, while the gradient points downhill however many weights there are. I have not isolated that, though, since the full adder also doubled the cases and the outputs, and I tried only one learning rate on it. The dimension argument is the standard reason large networks are trained with gradients, and it is also why XOR is a poor place to see it.

What NEAT spends its extra work on

NEAT needed 64,800 to 69,000 network runs on XOR, 39 to 51 times the hill climber. It took 108 to 115 generations, and every generation evaluates all 150 networks. What that buys is the one thing the other two cannot do: it chooses the network's shape.

The other two learners needed me to decide on three hidden neurons, or four for the full adder, before they started. The example NEAT solutions from the six XOR repeats had between 4 and 7 hidden nodes and 18 to 21 connections. On the full adder they ranged from 4 hidden nodes and 23 connections to 12 hidden nodes and 36. It did not choose a smaller shape than mine; its solutions were bigger. The cost is not worth paying when a person can choose a shape that works in advance. The rest of this series uses NEAT on Monopoly, where I had no idea what shape a good player's network should have.

What the counts leave out

A network run is not a unit of time. Backpropagation's backward pass is real work that the count ignores, and so is the hill climber's copying of weights. Wall-clock time is too noisy at this size to replace it: backpropagation at learning rate 2 took 4.4 to 18.3 ms for the same configuration across three runs.
Medians exclude learners that failed. A run that got stuck and never reached the target counts only in the "reached the target" column, which is why that column is there. With an even number of successes, the script reports the upper of the two middle values.
Three repeats is a small sample. The ranges are the spread I saw, not a confidence interval.

The part I expect to generalise is not which method won. A method's reputation comes from the problems it is famous for, and on a small enough problem the naive method can win honestly. I only found out by counting the same unit of work for all three, at more than one setting of the knob the losing method depends on.

A Published Exponent That Was a Digit Count

Seth Wheeler — Fri, 09 Oct 2026 03:00:00 +0000

In a private research repo of mine there are twenty-one small programs, each with a frozen oracle: a reference implementation plus a set of probes, which are inputs paired with the answer or the refusal the program is required to produce. A generator reads those oracles and writes a contract document, so somebody reimplementing one of these programs can read what it must accept, what it must reject, and the largest magnitude anybody probes it with. That last figure exists so a reader knows where the edges are.

It published the number of digits, not the order of magnitude. A probe of exactly 10**12 came out as ~10^13.

Three of the twenty-one operations publish that figure. All three were wrong.

The repo is private, so there is nothing to link to. The figures come out of the generated documents themselves, and out of commands I re-ran while writing.

The three, and the dangerous one

A count of changed rows cannot show this defect, because the whole defect is a number being one off. So here is every figure that moved, against the literal in the probe it came from:

operation	the literal probed	published before	after	what the probe expects
`number_names`	`10**12`	`~10^13`	`~10^12`	a refusal: a ceiling to reject at
`binary_convert`	`1000003`	`~10^7`	`>10^6`	an answer: a floor to clear
`luhn`	`6011000000000004`	`~10^16`	`>10^15`	an answer, and it is a card number rather than a magnitude

The first row is the one that matters, and it is worth being precise about why. That probe requires the program to refuse the value, so the published figure is a ceiling: reject at or above this. Publishing ~10^13 told a reader to reject an order of magnitude too high, so a program built faithfully to the contract accepts `1012` and fails the very check the figure exists to protect.**

It misreported in the direction of the defect it exists to prevent. A figure wrong toward safety is a nuisance; this one was wrong toward the failure.

The fix also separates two things the old format ran together. When the probe value is an exact power of ten the contract now says ~10^12; when it merely sits above one it says >10^6. 6011000000000004 is a credit card number, and calling it ~10^16 invited a reader to think a round magnitude had been chosen deliberately.

The test was watching the wrong property

There was a check over this figure. It requires every published magnitude to start with ~10^.

That is a test of the format, not of the value. ~10^13 passes it as readily as ~10^12, so it could not see this defect at all, and it was green while the figure was wrong.

Worse, it made the cheap fix invisible. Subtracting one and leaving the format alone would have kept it green, and would have kept the question closed. The repair was only visible as a repair because it also changed the format to >10^6 for the two floors, which broke the format check and forced the question into the open.

So this round edits a test that exists to stop exactly that kind of edit, which needs a justification. The justification is that the widening was written down in advance by whoever wrote the test, and they left the instruction in the codebase: expect this check to fail beside the new wording, that is the regular expression reading the new text rather than a leaked value, so widen it in the same change. They were right that it would fail and wrong about how, having expected a polarity word where the round shipped a symbol. Worth noting that the instruction is not a comment beside the assertion it governs; it is the failure message of a different check sixteen lines further down, which is a fragile place to leave something load-bearing, and it is the only reason the edit is defensible rather than convenient.

Then a check was written that can actually see the defect. It re-derives the expected string from each operation's own literal and compares it against what is published. Planting the digit-count rule back in the generator, which means reintroducing the defect on purpose to confirm the check notices, turns it red, naming binary_convert ('~10^7', '>10^6'). Restoring turns it green.

That new check also carries a denominator, and the reason is a previous version of exactly this control. A check over an empty set passes, so a control has to assert that something was compared. The earlier one asserted only that the published list was non-empty, and stayed green while two of its three entries were being silently discarded before the comparison. The shipped version requires all three, which is the difference between "it ran" and "it ran on everything".

The two findings underneath

The generator could not run at all. This is a document that is supposed to be regenerated from its sources, and regenerating it was the first thing the round tried. A document whose generator has quietly stopped working has become a hand-maintained file without anybody deciding that, and every claim in it about being derived is now a claim about history.

The rule lived in two files, so the first fix reached neither artifact a reader is handed. Two separate pieces of code computed the same published figure, and repairing one left both documents unchanged. That is the failure mode where the fix is real, the diff is correct, the commit message is accurate, and nothing a reader can see has moved.

What generalises

A test that pins a format does not pin a value, and the two are easy to confuse when the format encodes the value. ~10^N looks like it is about the number. It is about the string. Anything asserting on a rendered figure is worth reading twice with one question: would this be green if the number were wrong?

The direction of an error matters more than its size. One off is nothing until you ask which way. This figure is consumed as a bound, and being one order high on a ceiling turns a protective constraint into permission.

A cheap fix that keeps every test green is the worst outcome available. Subtracting one would have been correct, quiet, and would have left a format check standing that cannot see the class of defect it appears to guard. The repair that breaks a test is the repair that gets the test fixed.

The Table Named After the Programming Language

Seth Wheeler — Thu, 08 Oct 2026 03:00:00 +0000

appgen is a tool of mine that builds a small working application from one sentence, with no language model in the generating path. Before it can build anything it has to find the entity: the noun the app is about, which becomes its table name, its form fields and its URL paths. Given "Build a recipe manager with search" the answer should be recipe.

Given "Build your own Blockchain in JavaScript" the answer was javascript, and the tool built a table of javascripts.

It was not alone. "Let's implement a cryptocurrency in Kotlin" produced kotlin. "Create a Twitter Bot in Python Using Tweepy" produced tweepy. "Build a Microblog with Flask" produced flask.

One more rule matters for everything below, because it decides whether a wrong answer is cosmetic or fatal. The tool's checker rejects any artifact whose entity is not a word of the request. So an entity taken from the wrong part of the sentence still builds; an entity invented from nowhere does not.

Private repo, so no code link. What is checkable is the provenance of each figure, which I give as I go.

Why the language wins

The rule scans the request for candidate nouns. On an imperative sentence it prefers the rightmost content word, because English noun phrases put the head last: in "a recipe manager" the head is manager, and the informative word sits just to its left, so reading rightwards gets you to the right neighbourhood before scanning back.

A trailing in JavaScript, using Tweepy or with Flask is an adjunct. It says what the thing is built with, not what it is about. But it is also, almost always, where the rightmost content word lives. So the part of the sentence that is explicitly not the subject sits exactly where the rule looks for the subject.

The fix names no languages

Strip the tail, try the head, and keep today's answer if the head names nothing. Fourteen lines, and it shipped.

Build your own Blockchain in JavaScript            javascript -> blockchain
Let's implement a cryptocurrency in Kotlin         kotlin     -> cryptocurrency
Create a Twitter Bot in Python Using Tweepy        tweepy     -> bot
Code a 2D Game Engine using Java - Full Course     beginner   -> game
Build a Microblog with Flask                       flask      -> microblog

The thing I like about this one is what is not in it. There is no list of programming languages. javascript, kotlin and tweepy are not named anywhere in the change, and the rule would work the same way on a framework invented next year. A hand-written list of languages would have scored the same on this corpus and would have started rotting the day it shipped.

And the title of this post is from the half with no evidence behind it

Here is the part that cuts against everything above.

The scored corpus is 75 project titles with hand-written gold answers. They come from three third-party lists, taken complete, none written with any knowledge of this tool: 861 linked items, of which 87 have gold answers, of which 75 reach the entity reader at all. On those 75 the fix moved the score from 31 to 37.

Break the six gained rows down by which preposition fired:

preposition	rows changed	rows gained
`in`	1	0
`using`	2	0
`with`	12	6

Every one of the six gains is a with row. The in branch, which is the one that produced javascripts and the one this post is named after, changes a single scored row and gains nothing. It changes 57 rows on a larger corpus that has no gold answers, so I can see it doing something and cannot show that the something is right.

So javascripts is a real defect, really fixed, and "31 to 37" is not evidence that fixing it helped. The reach is carried by in and using; the measured accuracy is carried entirely by with. Those are two different claims and only one of them has gold labels underneath it.

Both arms in that measurement are the real functions rather than reimplementations. The before arm is the shipped rule with this change lifted back off; the after arm is the shipped entry point itself. An earlier round reimplemented an arm, watched it drift from the thing it claimed to measure, and took a whole round to notice.

One word was worth more than the grammar

X Clone means a clone of X, and the tool was answering clone.

The rule already knows about generic heads: nouns that say nothing about the domain, so that when it finds one it scans leftwards for the real subject. That set already held engine, box, book, collection, catalog, app, platform, system, service and program, which is why "Trello Platform" already answered trello. clone was in neither that set nor the stop list of words that can never be an answer.

Adding it is a one-word change, no new code path runs, and it shipped too. On the same 75 rows it moved the score from 37 to 49: fourteen rows moved, twelve gained and none lost. With clone in place the fourteen-line grammar fix is worth +8 rather than +6, since both arms move together, so the word is worth about one and a half times the grammar.

One of the twelve deserves a discount. Google Docs Clone answers docs where the gold set wants document or doc, and it scores as a gain on a technicality; a reader who thinks both answers are wrong in the same way should read eleven.

The mechanism was already written down in the tool's own README, one paragraph from the problem the grammar fix solved: on noun phrases the leftmost scan takes a leading modifier, naming clone as four rows of it. The fix was sitting in the documentation waiting for me to read my own writing.

Where those twelve rows came from, and why that is a problem

27 of the 75 rows contain clone. The corpus is a list of project ideas, and "build a clone of X" is one of the two or three things such a list is mostly made of.

And the word was not chosen independently of these rows. The README paragraph that named it was itself written from an error analysis of this corpus, which lists four X Clone Application titles by name. So clone is a word selected from the rule's own mistakes on the corpus it is then scored against, which is the setup an earlier post in this series measured directly: words picked that way gain in sample by construction, and the question is whether they travel.

Here they do not. The controls say so by being inert, which is the more useful way to read them:

On a second corpus of 47 rows, nothing moves: 36 before, 36 after, no row gained or lost. One of those 47 does contain the word, and a different rule fires on it first, so the channel is blocked rather than empty.
On the 1,440 requests the repository's own generator can produce, nothing moves either: 772 before, 772 after. Those 1,440 requests contain the word zero times.

That is exactly the pattern the earlier post measured with a random-draw control: a real gain on the corpus the word came from, and precisely nothing anywhere else. Two rounds, two methods, same answer.

Where it does regress

It regresses, and the round went looking for where. Five hand-written requests a user could plausibly type come out worse:

a clone tracker                          clone -> record
a clone log with search                  clone -> record
a VM clone tracker                       clone -> record
a plant clone log for my nursery         clone -> plant
a clone inventory with creating…         clone -> inventory

If you genuinely want to track clones, clone is now a generic head and the tool scans past it. Two of those five land on a reasonable alternative. Three land on the tool's fallback noun, record, which is not a word of the request, so the checker rejects the artifact outright.

None of these appears in any corpus this tool is scored on. They were written by hand precisely because no corpus contains them, which makes the regression real and unmeasured rather than real and small.

One more number, so two posts do not look like they disagree

49 of 75 is the entity reader scored on its own. Elsewhere in this series the same rule appears at 47 of 75, and that is the whole shipped path with a routing layer in front of the reader. Same corpus, different functions, and the two should not be subtracted. The rounds here measure the reader because the reader is what the change touches, and the source write-up is explicit that the delta is +6 either way and that you have to name the harness when quoting one.

What generalises

A rule that scans for the most important word will find whatever is where the important words usually are. The language name did not fool a heuristic about languages; it occupied a position. Fixing the position generalises to frameworks that do not exist yet. Fixing the names would have been a list, and a list is a promise to keep updating it.

The vivid example and the measured gain were in different halves of the corpus. I fixed javascripts, wrote the post about javascripts, and the gold labels only ever confirmed the with rows. The breakdown that shows this is one extra column of the same table, and I did not produce it until the round was almost written up.

Say which of your controls could have fired. Two controls here report no change. One of them covers a population where the word under test never appears, and the other has a single row holding it, blocked by a different rule. Listed as green checks, those read as two independent confirmations; they are one measurement and two abstentions.

Reading a Reinforcement Learning Taxonomy Against My Own Tools

Seth Wheeler — Wed, 07 Oct 2026 03:00:00 +0000

Somebody handed me a list of reinforcement learning algorithms and asked whether any of them could help my tools. It is a ChatGPT session printed to PDF: roughly ninety algorithm names across fifteen sections, with no descriptions, no citations and no results. I want to be precise about that, not to dismiss it. It is a menu of mechanism names, it was a good thing to be handed, and it is the reason I went and looked at one of my own tools properly. Where a claim below is about how an algorithm behaves, the authority is the third-party documentation I read alongside it, mostly OpenAI's Spinning Up and a couple of tutorial pages, and never the list.

The tools are mine and none of them is a policy, which is what reinforcement learning optimises: a rule for choosing an action, tuned by whether past actions turned out well. appgen turns a sentence into a running application with no language model in the generating path. pycomplete is a code completer mixing an n-gram model, a buffer cache and a small transformer. growone is an online learner that adds hidden units as it goes.

Four rounds later, nothing shipped from any of it. The four reasons are different, and that is the interesting part.

The tools are in a private research repo, so none of them is linkable. The numbers are, in the sense that each one is attributed to where it came from.

Most of the list cannot apply, and that is the first result

Sections two through nine and eleven cover every value-based, policy-gradient, actor-critic, distributional, model-based, offline, multi-agent, hierarchical and large-scale family on the list. All of them need the same input: a number saying how well that went. Policy-gradient methods need to know whether an action beat the average; Q-learning needs a reward plus its own estimate of what the next state is worth. Without that number there is nothing to descend.

appgen has not got one, and the reason is not that nobody has built it yet. The obvious candidate is the tool's own verification verdict, ALL CHECKS PASSED, and it is not a reward, because of what it means: the program matches its own published contract. A correct implementation of a misread request passes it. An earlier round drove 138 externally-written requests through the tool and got 34 wrong builds; all 34 printed ALL CHECKS PASSED. Optimising that signal trains the tool toward wrong programs that verify.

That is a structural fact about the tool rather than a gap in the literature, and it removes nine of the list's fifteen sections in one paragraph. The remaining sections were not all assessed against that requirement, so this is nine of fifteen and not "all but one".

The one branch that does apply

Section ten is exploration bonuses: pseudo-counts and Random Network Distillation. Both are reward-free. In reinforcement learning they manufacture a reward out of novelty, to push an agent toward states it has not seen; here you can keep the novelty estimator and throw the use away. The offline family motivates the same thing from the other side, which is that off the data's support your estimates are untrustworthy, so stay on it.

Both reduce to one mechanism: estimate whether a request is inside the distribution the rules were built for. What the tool would do with that is refuse earlier and for a better reason. It already refuses most requests it cannot handle, by failing to find a grammar rule that matches; a novelty score would let it decline a request that parses fine and is simply unlike anything the rules were written for, which is the case that currently produces a confidently wrong app.

The population is 52 requests the tool actually built. 18 of them are apt, meaning the program it produced is the one the request asked for, judged against the request rather than against the tool's own contract; 34 are not. The in-distribution reference is not curated: it is the repository's own corpus generator, so the generator is the definition of a request the tool was built for, and no judgment call picks the fit set.

The scoring is AUC: take one apt request and one inapt one at random, and AUC is the probability the score ranks them the right way round. 0.5 is a coin flip and 1.0 is perfect. A permutation test at this sample size puts the floor for a resolvable result at 0.641, written down before any arm ran.

C1_length        AUC 0.717   p=0.0035
C2_coverage      AUC 0.739   p=0.0020
A1_pseudocount   AUC 0.784   p=0.0003
A2_rnd           AUC 0.676   p=0.0194

All four clear the floor, and the two that are not the imported machinery are the reason nothing shipped.

The control that ate the result

C1_length is token count. I put it in as the confound control, and wrote down in advance that it clearing the floor would matter more than the arms clearing it. It clears at 0.717: short requests are apt about 47% of the time, long ones about 11%.

So the arms have to survive length, and the round strips it two ways, both specified before either was run. Residual regresses the arm on length and scores what is left, which assumes the relationship is linear. Stratified takes the AUC separately inside each half of a median split, which assumes nothing but works on small groups.

arm	raw	residual	stratified
`C2_coverage`	0.739	0.682 (p=0.016)	0.720	survives both
`A1_pseudocount`	0.784	0.567 (p=0.22)	0.716	the two controls disagree
`A2_rnd`	0.676	0.655 (p=0.033)	0.644	survives both, barely

The best raw arm is the only one the controls disagree about. Its 0.784 collapses to a null once the linear part of length is removed, and holds at 0.716 under the assumption-free read. I do not know which is right, and reporting the 0.716 alone would be choosing the favourable control after seeing both.

Then the part that decided it. The corpus was scraped from two websites, and they are not interchangeable. One contributed 40 requests at a 45% apt rate. The other contributed 12, all of them tutorial titles about reimplementing a technology (Chess Engine In C, mal - Make a Lisp, Write your own Operating System), and none of the twelve is apt. So knowing which website a request came from scores AUC 0.676 on this population, identical to the Random Network Distillation arm to three decimals.

Hold the corpus fixed and read inside the larger source only, with the floor recomputed for that smaller sample, and the RND arm drops to 0.644 and stops clearing. It was reading provenance. The pseudo-count arm survives that read; it is the one that does not survive the length control.

What actually decided it, which was not any of that

The confound is a reason to distrust an arm. It is not what stopped the tool shipping, and I want to be careful about that, because deciding afterwards is exactly how a round like this goes wrong.

The pre-registration said: a tool ships only if the separation holds and the operating point is usable. At a threshold keeping 90% or more of the apt builds, the best arm rejects 26% of the inapt ones, against the 40% named in advance. So the second condition failed and nothing ships, and that was settled before the arms ran.

A round that ran four arms, found p = 0.0003 on one of them, and then asked what counts as success would have shipped a gate on a 26% rejection rate.

The second tool has a return, and a plateau

pycomplete does have a usable number, because you can ask whether its top suggestion was right, so the machinery has the signal it needs. That turned into its own round with its own answer, and a third reason for shipping nothing.

One framing point is worth carrying here. Offline, over logged positions, every option can be scored at every setting, because the answer is recorded: the feedback is full, which makes the offline problem ordinary supervised learning and not reinforcement learning at all. The bandit framing only bites in deployment, where you see whether the completion you showed was accepted and nothing about the ones you did not. The offline version was measured first because if the offline ceiling is empty the online one cannot be better.

The third tool maximises a proxy, which is the one real hit

growone has a decision, a candidate set and a signal. When it adds a hidden unit it trains four candidates from random starting points and installs the one whose activation has the largest absolute covariance with what the model is currently getting wrong.

That is a proxy. The thing wanted is lower error going forward; the thing maximised is correlation with the current error right now. This is the exact split the literature names between directly optimising what you want and optimising a surrogate, and the list is what sent me to look.

Measured, the proxy is directionally right and already takes most of what is there. Perfect selection among its four candidates is worth 2.6% of loss, and the shipped rule captures 71.8% of that. No change ships, and that is the correct outcome.

The finding worth keeping is that the alignment is stratified. The measure is regret: how much of the available gap between the best and worst candidate the rule gives away, where 0 is perfect selection and 0.5 is choosing at random. On a smooth nonlinear stream regret is 0.228. On the open-key stream, where feature names arrive mid-run, it is 0.453, which is nearly blind. That stream is the one the tool is named for and is its distinguishing claim.

Two readings of that make opposite predictions. Either a unit installed for a feature that is still arriving looks worthless over a short window and pays later, so regret falls as the window lengthens; or the score simply does not rank candidates on that stream, so regret stays. Running each candidate set forward 600 points once and cutting the same run at three lengths, so the windows share the same installs:

window	regret on the open-key stream
150 points	0.440
300 points	0.401
600 points	0.442

Paired change from four times the window: +0.002, CI [−0.052, +0.057]. The horizon explanation is rejected. The proxy is near-blind there rather than short-sighted.

(That 0.440 and the 0.453 above are the same measurement on different populations. The window study can only use installs with at least 600 points of stream left after them, which is 149 of the 224 the earlier round scored on that stream, so the two numbers should not be read as a change.)

And there is almost nothing to be blind about: the spread between the best and worst candidate on that stream is 2.5% to 3.1% of mean loss at any window. Ranking them perfectly is worth a rounding error, which is the real reason this closes rather than continues.

What I could not check

Seven of the tools the question was asked about were uninitialised submodules in the worktree that round ran in, so they were not examined. That was a fact about my environment rather than a finding about them, and two of them were checked out and audited in a later round, which found nothing across nineteen runners. The rest are still unexamined.

What generalises

A taxonomy is a shopping list, and the first question is which aisle you are allowed in. Nine of fifteen sections went away on one property of my tool: there is no reward, and its nearest candidate is a verdict saying this program matches its own contract, which is exactly what is true of a correct implementation of the wrong request. Working that out cost an afternoon and saved building any of them.

Put the dumb control in before the clever arm, and write down in advance that it matters more. Token count at 0.717 is the whole result here. Reporting the pseudo-count arm's 0.784 next to a null, and not next to the length control, would have read as a success.

The corpus can be the classifier. When an arm ties with the name of the website a row was scraped from, and then falls below the bar as soon as you hold the website fixed, the arm was measuring provenance. That was visible in a two-row table of source against outcome, and I did not look at that table until after the arms had run.

A Measured Headroom That Nothing Could Reach

Seth Wheeler — Tue, 06 Oct 2026 03:00:00 +0000

pycomplete is a code completer of mine that runs on a laptop CPU. It predicts the next token by mixing three things: an n-gram count model over your repository, a cache of the buffer you are currently editing, and a small transformer. One number, α, decides how much of the mixture is transformer, and it is a constant. The shipped value is 0.45, and a comment in the source says it is a constant on purpose.

The obvious idea is to stop it being a constant. Some positions in a file surely want more transformer and some want less; pick α per position and win. Three rounds later α is still 0.45, nothing shipped, and the reason is not the one I expected.

Everything below is scored in top-1: the fraction of positions where the token you were about to type is the single highest-ranked suggestion. And headroom means the gap between what a perfect chooser would score and what the best single fixed choice scores, which is the most any router could ever win.

A per-position oracle, allowed to pick the best α for every position after seeing the answer, scores +0.0705 top-1 over the best fixed α, with a paired 95% confidence interval of [+0.0623, +0.0786]. Three families of features captured 0.0%, −3.1% and −0.9% of that.

The completer lives in a private research repo, so there is no code to link. Every number below is traced to a results file or a re-run command.

The ceiling, and why it is not an artifact

6,000 positions sampled at a fixed stride through 42 held-out files, against a 116-file index of the Python standard library. The split is by file, fixed before any code existed, because positions inside one file share a buffer and an identifier vocabulary, and splitting by position leaks the answer across the boundary in exactly the direction being tested.

	test top-1
shipped, α = 0.45	0.4948
best fixed α chosen on the training half (also 0.45)	0.4948
per-position oracle	0.5653

An oracle sweeping a grid can only look better as the grid gets denser; +0.0705 could therefore have been an artifact of trying seven values. On a 2,500-position subset, doubling the grid to thirteen values moves the oracle by +0.0012, which is 1.7% of the headroom. The headroom is structure, not resolution.

One piece of care matters more than it sounds. The mixture is replicated inside the measuring harness so that every α is scored on identical arm outputs and the transformer's forward pass is paid once instead of seven times. That replication is a second implementation of the thing being measured, which is how harnesses quietly drift from tools; so the probe re-scores a sample of positions through the shipped completer and refuses to write its output if any disagree. The results file exists because 400 of 400 agreed.

Family one and two: nothing, and less than nothing

Every policy here is tabular. Sort each position into a cell, work out the best α for each cell on the training half, and use it. With no state transition and no discounting that is exactly a greedy contextual-bandit rule, which is the most legible thing to fail.

policy	keyed on	test	gain over best fixed	paired 95% CI
P-BUFFER	how long the buffer is	0.4948	+0.0000	[−0.0033, +0.0033]
P-CONF	the three arms' confidence and agreement	0.4927	−0.0022	[−0.0065, +0.0022]
oracle	the answer	0.5653	+0.0705	[+0.0623, +0.0786]

Buffer length capturing exactly nothing reproduces what the tool's own source comments already claimed, on a different corpus from the one that settled it; confidence does slightly worse than doing nothing.

A flat result has two very different causes, and the round separated them in advance: a policy that overfits (big training gain, no test gain) is a different animal from one whose features cannot express anything at all. Permute the hit vectors across positions, which destroys any relation to the features while keeping every margin identical, and refit:

train-fit gain from pure noise : mean +0.0054   95th pct +0.0095
P-CONF's real train-fit gain   :      +0.0039   -> p = 0.740

Fitting noise beats fitting the features. With eight cells and seven values of α, picking the best-scoring α per cell is upward-biased by construction, and the confidence policy does not even reach that bias. So the finding is these features carry no routing signal, which is a narrower claim than routing does not work, and it is the only one that was measured.

Family three: the optimum really does move, and it is still worth nothing

The first round closed by naming the untried family: syntactic context, which is a different measurement rather than a better learner. The next round built it, importing the mixture, the α grid, the verification contract, the splitting rule and the placebo so that the features would be the substantive change.

The corpus is not held fixed, and that matters for reading the two rounds together. This one runs 12,000 positions over 72 held-out files rather than 6,000 over 42, and the best fixed α moves from 0.45 to 0.6 with it. So the tables below are internally consistent and are not comparable cell by cell with the tables above. The ship bar is the same in both: a router ships only if it clears +0.020 with a confidence interval excluding zero.

I predicted the payoff would come after a ., because the next token is an attribute and the repository index holds the project's attribute names, so α should fall there. Measured, . wants α = 0.6, which is this round's global best, with a lift of exactly 0.0000. The classes that move are , and =, which want less transformer at the start of a fresh expression. The direction of the mechanism I argued for was wrong, and I am leaving that on the page.

The gate held, though, in the sense of the pre-registered criterion: best α per previous-token class spans 0.30 across nine classes, where buffer length was flat and confidence was below the noise floor. Syntax genuinely moves the optimum, and it is the first family in three to do so.

Then the question the criterion does not ask: does moving pay? For each class, how much does its own best α beat the global best α on its own rows?

class	n	best α	lift over the global best
`,`	261	0.3	+0.0421
`:`	202	0.45	+0.0198
`=`	204	0.3	+0.0098
`id` (identifier)	2137	0.45	+0.0066
`kw` (keyword)	616	0.45	+0.0016
`(` `)` `.` `op`	1639	0.6	+0.0000

Size-weighted, the ceiling is +0.0063. Against this round's own test-half oracle of +0.0662 that is 9.5%, and it is the most a per-class router could win, before anyone asks it to generalise. Four of nine classes want the global optimum exactly, and the two classes that actually pull away from it, , and =, hold 465 of 5,325 training rows between them.

The tool's own source said why before any of this existed: the values from 0.45 to 0.6 are a plateau, and two of them differ on eighteen positions. Picking the best α on a plateau moves for free and is worth nothing.

Fitted, it behaves exactly as that predicts:

policy	test	gain	paired 95% CI	cells
best fixed α	0.4900	n/a	n/a	n/a
P-SYN, previous-token class by depth	0.4894	−0.0006	[−0.0049, +0.0036]	22, smallest 22 rows
P-SYN+, plus the token before that	0.4800	−0.0100	[−0.0153, −0.0048]	182, smallest 1 row

Finer cells bought variance, not reach; and the placebo gives the round in one line:

train-fit gain from pure noise : mean +0.0051   95th pct +0.0075
P-SYN's real train-fit gain    :      +0.0073   -> p = 0.065
the per-class ceiling          :      +0.0063

Syntax carries a little real signal, sitting above the noise mean where the confidence features sat below it. But the ceiling (+0.0063) and the noise floor (+0.0051) are the same size. The entire signal available to a per-class router is no larger than the bias from fitting one. That is a statement about per-class routing on this tool, not about every router anyone could build.

The same shape, one switch over

There is a second thing in this tool applied unconditionally. A suffix layer re-ranks candidates by how the token being typed ends, which helps when you are halfway through a word, and it is weighted the same everywhere. A per-position oracle over whether to apply it is worth +0.0235, CI [+0.0208, +0.0262], which clears the +0.020 bar. A perfect switch would be worth shipping, which is why it was worth trying.

The best fitted switch, given free choice of threshold and direction over the 5,325 training positions, returns the identity. It vetoes nothing.

The reason is arithmetic. The layer changes the top-ranked candidate on 2,670 positions: it breaks 282 of them and fixes 1,015. So a veto that fires blindly would trade away three and a half fixes for every break it prevents, and one that fires selectively has to find breaks at better than one per fix to come out ahead. Since breaks are only 0.28 of fixes to begin with, a useful veto needs to be about 3.6 times as selective as chance. The best ratio reachable by any threshold on any of the four available features is 0.48 breaks per fix, which is 1.7 times the base rate. Not enough, by a factor of two, so the optimiser correctly declines to fire at all.

My stated mechanism was wrong there too, in every feature. I had reasoned that the layer overrides a confident transformer, so a veto should fire on high confidence. Having the right answer ranked first and being confident are different quantities, and the breaks sit on the low-confidence side for all four features.

What this leaves

The ceiling is the contribution. +0.0705 with a confidence interval, stable under grid density, is a number this project did not have, and it bounds every future attempt. The next person does not have to find out whether there is anything there, only whether their features can reach it.

44.6% of positions are wrong at every α, which is what caps the oracle in the first round. Routing cannot touch those. Only a better arm can, and that is a different project.

What is untried is features, not method. The tabular policy was never the limitation: it beat nothing because its inputs carried nothing, and when its inputs finally carried something, the objective was flat.

The online version is now known to be unpromising rather than unmeasured. Measured offline over logged positions, every arm can be scored at every α, so the answer is fully available; a deployed system sees only whether the completion it showed was accepted. With full information these features found nothing, and less information cannot help.

What generalises

Measure the ceiling before you build the router. It costs one pass over logged data, it is the same measurement whatever policy you would have built, and it tells you the largest number any of them could return. Mine came back large, which is why I kept going, and three feature families later the interesting discovery is that a large ceiling and a flat objective are perfectly compatible.

A best choice that moves is not a best choice that pays. I wrote a criterion asking whether the optimum shifts across classes, it held at a span of 0.30, and I nearly reported that as the good news. The lift table is the one that matters and it was one extra column of the same data.

Print the bias from fitting noise next to every fitted gain. Two of the three families here produced a positive training-set gain. In both cases permuting the labels produced a larger one. That comparison is cheap, it fits in three lines of output, and without it both rounds would have had a number that looked like a small win.

A Learner That Scored Below Its Own Random Control

Seth Wheeler — Mon, 05 Oct 2026 03:00:00 +0000

appgen is a tool of mine that turns a sentence into a running application with no language model in the generating path. Something has to read the sentence first, and one field of that reading does more work than the rest: the entity, the noun the generated app calls a row. It becomes a table name, a form field and a URL path, so getting it wrong means the right program gets built about the wrong thing.

Today that field is read by a hand-written keyword rule. A local language model reads it better, and I am not simply using the model, for two reasons: it costs a 30-billion-parameter forward pass per request against a rule that costs nothing, and when I measured the whole stack end to end the model's plan won no requests the rule's did not. So the interesting question is whether something cheap and learned could sit between them. I put a learner there, and it scored below random units of the same shape.

The learner is growone, a small online model of mine that starts with no hidden units and adds them as it goes. Its selling point is an open key universe: it is built for features that arrive mid-stream, which is exactly what happens when each token of a request becomes a feature key and the corpus keeps reaching words it has not seen. It produces one number per candidate rather than a sequence, so the entity is never generated: every token of the request is scored, the highest wins, and a this request names nothing to track sentinel is carried as one more candidate rather than as a special case. No part-of-speech filter and no stop list, because a hand-written stop list is a rule, and importing a rule into a learner's candidate set is how a learner takes credit for somebody else's work.

The corpus is 75 project titles with frozen hand-written gold answers, drawn from three third-party lists: build-your-own-x, project-based-learning and Project-Ideas-And-Resources. The gold set holds 87 rows; twelve of them name a domain the tool already has a table for, so the reader is never consulted on those, and 75 reach it. The code is in a private research repo, so there is no link to that. Every figure comes from a results file or a command I ran, and I say which.

The comparable arm, and the one that looks respectable

There are two protocols here and only one of them may be set beside the baselines. The cross-corpus arm trains on 47 gold rows from a different corpus, freezes, and evaluates. That is cold on these 75 rows, exactly as the rule and the model are. The prequential arm predicts each row before learning from it, which is growone's native protocol and is not comparable, because by the time a late row is scored the net has already seen fifty earlier rows of the same corpus.

Every learner score below is a mean over seeds, which is why some of them are not integers.

arm	of 75	what it is
the model	55	qwen3-coder:30b, one run, transcribed from an earlier round rather than re-run here
the shipped rule	47	the hand-written incumbent, re-measured on today's tree
prequential	33.5	not comparable to the two above
`control`: random units, same shape and schedule	17.0
`floor`: growth switched off, so a plain linear model on the same features	17.0
the grown net	15.5	the comparable arm
always answer `post`, the most common gold noun	13

That 47 deserves a footnote, because this post is partly about stale numbers. The write-up that first scored this rule on these 75 rows published 29; the same scorer returns 47 today, because the rule has been worked on since. Every figure here is the re-measurement.

The 33.5 is the number that looks respectable and it is the one that should not be set against the rule. The comparable figure is 15.5.

And 15.5 is the interesting number for a reason that has nothing to do with the rule. The search is worth −1.5 rows against random units of the same shape and the same schedule. That is growone's own question, did the hidden units do anything, asked on a real task rather than on a generator.

Two seeds each, so be careful with the sign: the grown arm scored 17 and 14, the random-unit control 18 and 16, and the linear floor 17 twice. The round's pre-set band for a real difference on this arm was 2.9 rows, and 1.5 is inside it. So the finding is not that searching for units hurts; it is that it buys nothing measurable, against a control drawn to have exactly the same shape and schedule. The tool's own README reports the same thing on its home ground, where a linear floor beats the grown net at every size and randomly drawn units match the searched ones.

The only arm it clears is the constant. I should say what that constant nearly was. The first version answered with the last token of the request, on the reasoning that English noun phrases are head-final. It scored 1 of 75, because these titles end in the framework: …with React Native. An arm that cannot win makes the comparison against it pass by construction, which tests nothing.

The candidate set is not the excuse

The obvious objection is that picking the highest-scoring token of the request is a bad design: a row whose gold noun is not in the sentence cannot be won by it, however good the learner. That is true and it is worth 11 rows.

	of 75
ceiling: some candidate is accepted by the grader	64
unwinnable, because no token of the request is the answer	11
the same ceiling with the rule's own answer added as a candidate	64, no change
rows the rule gets right that are not reachable as a raw token	0

The eleven are not hard rows, they are rows where the noun is simply absent: Fitness App wants workout, A COVID-19 Tracker wants case, Tik Tok Clone - MERN STACK wants video. Supplying those is world knowledge, not extraction, and nothing in this round touches it.

Read against 64 instead of 75, every rate multiplies by the same 1.172 and no ranking moves. What the third line settles is the excuse: adding the rule's own answer to the candidate set raises the ceiling by zero rows, and not one row the rule gets right is unreachable as a raw token. The learner was picking from a set that contained the rule's answer every time it mattered, and picking something else.

Handing it the answer

So I handed it the answer directly. One extra feature key marks whichever candidate the shipped rule chose, and everything else stays. All arms in this section are prequential, so they are comparable to each other and all of them carry the same advantage over the cold baselines: 74 rows of practice the rule and the model never get.

arm	mean of 75, five seeds	what it is
ceiling	64	oracle over the candidate set
the model	55	transcribed, not re-run
`rule_only`	48.0	a net given the rule's pick and nothing else
the rule	47	the incumbent
`hybrid_floor`	41.0	hybrid features, growth switched off
`hybrid`	40.4	all the lexical features plus the rule's pick
`base`	32.6	the lexical features alone

The rule's read is worth a real and reliable +7.8 rows to the learner, from 32.6 to 40.4. Every seed gains between six and ten rows and not one seed loses a row, at p between 0.0020 and 0.0312. It is still not enough to reach the rule that supplied it, and that comparison does not resolve: the rule leads the discordant count (the rows where exactly one of the two is right) in all five seeds, and no seed gets below p = 0.096. At n = 75 the ranking is simply unresolved, which is weaker than calling it a loss and is what the numbers support.

Then there is the row I did not expect. rule_only, a net handed the rule's pick and nothing else, scores 48.0. hybrid, which is that same key plus thirteen families of lexical features, scores 40.4. Adding features to a working signal costs about eight rows, and rule_only leads the discordant count in every seed.

The mechanism is not mysterious. The token, suffix and neighbouring-word keys are dense and arrive on every candidate; the rule's pick is a single key arriving on one candidate; and the net has 75 rows in which to learn that the rare key outranks the common ones. It is the same shape as something one layer up in the same stack, where a planner filling a field suppressed the reading the layer below was already doing. Here it is a token vocabulary drowning the rule's.

rule_only beating the rule itself, 48 against 47, is not a result. It splits 2–1 discordant on every seed at p = 1.0. It is a pinned arm landing where a pinned arm should, and its value is precisely that it did: a harness that could not reproduce the rule from the rule's own answer would have invalidated every other number on this page.

Three arms separated by one key will report a margin whether or not the key does anything

So the key was blinded. The rule's accessor is rebound in memory to return nothing, and each arm is required to move the way its name demands.

arm	with the key blinded	unblinded	what it must do
`base`	33.3	32.6	must not move; it never had the key
`hybrid`	33.3	40.4	must collapse onto `base`
`rule_only`	21.0	48.0	must collapse to no signal

All three did. The load-bearing one is base: a check that every arm responds to shows only that the harness is live, and the arm that must not respond is what scopes it.

Growth earns nothing, and the totals were order-dependent

	mean	per seed
`hybrid`, growing	40.4	41, 41, 39, 39, 42
`hybrid_floor`, growth switched off	41.0	41, 41, 41, 41, 41

The floor is higher and it is identical on every seed. On this task, growth buys seed variance and nothing else.

A follow-up round found something else in those totals. The corpus arrives in source order, and the three lists are fully contiguous: 4 rows from one, then 38, then 33. So a prequential learner trains on one register, meaning one house style of writing project titles, and is then tested inside it. Shuffle the order and hold the seed count fixed, and the learner drops 3.7 rows:

	rule	learner	hybrid
shipped order, 5 seeds	47	32.6	40.4
shipped order, 20 seeds	47	32.9	40.6
shuffled, 20 seeds	47	29.2	37.9

The middle row is why the bottom one means anything. Seed count moves 0.3 rows; the order moves 3.7. Comparing five unshuffled seeds against twenty shuffled ones would have confounded the two.

Another label does help, and it never changes the ordering

The prequential protocol scores row i with a net that has seen rows 0 to i−1, so accuracy by position is the label curve. Over twenty shuffles, by thirds:

third	rule	hybrid	learner
first	0.648	0.384	0.302
middle	0.606	0.544	0.404
last	0.626	0.586	0.462

The rule is the control here, not a baseline. It does not learn, so its per-third rate is row difficulty alone, and it is flat to slightly down. Every row of the learner's rise is therefore learning rather than an easier tail, and it comes to +4.55 rows over that control, rising at every step.

That is a real learning signal, and it changes nothing about the ordering: rule, then hybrid, then learner, in every third, with the gap in the last third still about four rows in twenty-five. The stream runs out of labels before anything crosses.

A straight line through gained 0.16 accuracy over roughly fifty labels, still 0.164 short says another fifty would close it. That line is not evidence and I am not drawing it. Learning curves flatten, the reachable maximum for this design is 64 of 75 rather than 75, and nothing here measures the shape past label 75. What is established is narrower and still useful: the curve had not flattened when the labels ran out, so more labels would not help is refuted, and the label budget is the live lever.

What generalises

Three things, in the order I would want to be told them.

A learner's own control is the comparison that matters, and it is the one that gets left out. Against the incumbent rule this arm loses by 31.5 rows and the honest summary is "not close". Against random units of the same shape and schedule it is level, and that is the number saying the search bought nothing, which is a claim about the method rather than about the task.

Adding a feature to a working signal can subtract. I would have said more features is weakly monotone, and it cost eight rows here, for the ordinary reason that dense keys outvote a rare one when there are only 75 rows to sort it out in.

A pooled rate hides whether a learner is failing or merely starving. The same 32.6 comes out of a method that does not work and a method that has not finished; only the per-position curve separates them, and here it says the second one, while also saying the ordering never moved.