This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
I was tuning an agent for a simulation competition. Thirty in-game days, two players, one shared commodity market. Plant crops, hire hands, buy livestock, sell into a market whose prices move with supply. Whoever ends with more coins wins.
The agent has about forty knobs. How much cash to keep before buying an animal, how many hands to hire on each day, when to buy more land. Tuning it means picking a knob, moving it, then finding out whether the agent got better.
Which needs an opponent. So I generated one from my own agent: same file, frozen configuration, saved as bench/rivalsim.py. Sixteen seeds, sixteen games, count the wins. Standard practice.
The result that should have scared me
I lowered animal_reserve from 400 to 150, so the agent would keep buying livestock instead of hoarding cash, bought eight cows instead of five and pushed the crew to twelve hands.
16 out of 16. Plus 40,000 coins of margin.
Every previous experiment had moved the number by two or three thousand. This one moved it by forty. I had been fiddling with the same parameters for days and suddenly one change swept the board.
I shipped it to the live ladder. Then I shipped a second variation on the same idea.
Both lost.
The knob was wired to both players
My agent reads its overrides from an environment variable. One line near the top:
_ov = _os.environ.get("KAGG_P") # tuning overrides, parsed into the config
The sparring partner was generated by copying the agent source. It copied that line too.
So when I set animal_reserve=150 and ran the match, the environment variable was set for the whole process. Both agents read it. My opponent also stopped keeping cash, also bought livestock it could not feed, also starved its herd.
I was not measuring a better strategy. I was measuring an opponent I had quietly sabotaged, using the same lever I thought I was pulling on myself.
Here is the same configuration re-measured with the override hook stripped out of the partner:
| config | wins | margin |
|---|---|---|
| baseline | 11/16 | +1,953 |
| the "breakthrough" | 0/16 | -15,314 |
| the variation I shipped after it | 1/16 | -13,044 |
A 16-0 win became 0-16. The margin swung by 55,000 coins. The change was one of the worst regressions I had produced and it presented as the best result of the project.
Why it read as a triumph
This is the part worth generalising, because the bug is not specific to farming simulations.
Sabotaging your opponent and improving yourself move the metric in the same direction. Win rate cannot tell them apart. A benchmark exists to hold everything constant except the thing under test. A shared configuration channel breaks that invariant without leaving a trace. No exception, no warning, no failing assertion. Just a number that got better.
The magnitude was the tell and I read it as the reward. Forty thousand against a usual two or three thousand should have been the moment I stopped and asked what else changed. Instead the size of the jump is what convinced me it was real. That is exactly backwards. I would not have said so out loud before it happened to me.
What I changed
The generator strips the hook. The script that produces a sparring partner now cuts the override block out of the copy, so the partner is a fixed strategy rather than a mirror of whatever I am currently testing.
A partner has to prove it is inert. Run it against an unrelated third agent, once with the environment variable set to something deliberately destructive and once without. Byte-identical scores or the partner is not frozen. Mine now returns 155,875 either way. That check takes about a minute and it is the cheapest thing in the pipeline.
One opponent is not a benchmark. Even with an inert partner, a single opponent rewards whatever exploits that opponent. A later build went 16 out of 16 against one partner and then sat around 27th of the ~30 opponents it met on the real ladder. So the gate became a seven-agent round robin with the seating permuted, 144 games each. Two identical agents score about 42% there with 16% ties, which gives me a floor: anything not clearly above that is noise.
Treat a great result as a measurement bug until proven otherwise. Not as false, just as unverified in a specific direction. The question is no longer "did this help" but "what else did this touch".
The line I keep coming back to
A sparring partner that reads the same configuration as the candidate is not a sparring partner. It is a mirror. A mirror always tells you that you are winning.
If you benchmark anything against a copy of itself (common in agent work: self-play, A/B harnesses, model comparisons) go and check what your two sides share. Environment variables, config files, a singleton, a cache, a random seed. Anything the candidate can reach that the opponent can also reach is a channel where your experiment leaks into your control.
Two bad submissions went out before I found mine.
Written with AI assistance (Claude, Anthropic). The bug, the diagnosis and the numbers above are from my own work and were re-measured before publishing: every figure in the tables comes from a captured 16 seed run, plus the inert-partner check is reproducible with one command.
Top comments (0)