Last week I wrote about an experiment where I asked an AI to author tests for the same policy 50 times.
Across 49 valid runs, every run covered all six boundary classes we had defined.
What bothered me afterward was not the coverage number.
It was where the generated values were landing.
The policy had two important thresholds: 40 and 70. The generated tests crowded around them:
39.99
39.999
40
69.99
70
Out of 784 accepted test records, 410 sat exactly on an edge or within 0.01 of one.
At first I had a simple explanation:
Maybe the model wasn't really discovering the test surface. Maybe it was seeing the numbers in the policy and decorating them with boundary cases.
That explanation sounded plausible.
So I designed an experiment that could prove it wrong.
It did.
And then, while checking the write-up, I found that my explanation of the new experiment was wrong in several places too.
That second failure may be the more useful result.
First, an important correction to the original experiment
The authoring prompt explicitly asked for borderline cases.
It told the model to exercise every clause and include the borderline cases a careful clerk would have noticed.
So the fact that the tests concentrated near thresholds was never evidence, by itself, that the model had independently discovered something clever.
The more interesting question was:
Which boundaries can it find, and what happens when I change how those boundaries are represented?
The original distribution was strikingly concentrated.
Values were not smoothly spread around 40 and 70. They were welded to them.
Across the whole corpus, only four records landed between 0.01 and 1.0 away from an edge.
I also looked at another reference point: 39.
The policy never prints 39, but one of the six registered classes begins there. Records did land exactly on 39, which is not surprising given the class definition.
What was almost missing was the space immediately below it.
That made the original explanation worth testing.
Copying a visible threshold and reasoning about the semantics around that threshold can produce very similar-looking tests.
To separate them, I needed to change the representation.
The comments that turned this into another experiment
After the first post, people pushed on the interpretation rather than the headline.
One suggestion was to rewrite and reorder the policy while preserving its meaning, then see whether the generated test surface stayed the same.
Another observation was that repeated generation may profile the policy text as much as it profiles the model.
A third line of questioning was about measurement itself: if there were an off-by-one error at 70, would a coverage metric like "6 of 6 classes covered" even expose it?
Those comments changed the experiment.
Instead of asking:
Can the model generate all the boundary classes we expect?
Study 012 asks something closer to:
Does that boundary coverage survive when the representation of the same policy changes?
Five versions of the policy
I created five variants.
| Version | What changed | Threshold representation |
|---|---|---|
| A | Baseline | 40 and 70 |
| B | Reworded | 40 and 70 |
| C | Clauses reordered | 40 and 70 |
| D | Thresholds moved | 45 and 72 |
| E | Threshold literals removed | "four tenths" and "seven tenths" of a 0-100 scale |
Versions A, B, C, and E have the same decision semantics.
Version D deliberately does not. Its thresholds move from 40 and 70 to 45 and 72.
I ran 30 authoring attempts per version - 150 total - against one pinned model snapshot, gpt-5.6-sol, through a digest-pinned CLI, all within one day.
That matters because this is one model, one policy family, and one experimental window.
It is not a statement about language models in general.
More importantly, I wrote down the prediction before running the experiment:
Coverage should collapse in Version E when the literal threshold numbers disappear.
I also wrote down what I would conclude if it didn't.
Without that, it would have been far too easy to inspect the results afterward and redefine what I had meant all along.
Result 1: move the thresholds, and the tests move with them
Version D changed:
40 and 70
to:
45 and 72
The generated tests followed.
Across Version D's 480 accepted records:
- 61 landed exactly on 45
- 50 landed exactly on 72
- zero landed on the old thresholds 40 or 70
The registered scorer calls this:
COVERAGE-FOLLOWS-THE-NUMBERS
I think this result matters more than I initially gave it credit for.
Forty-five and seventy-two are not especially attractive round numbers, yet the generated tests moved directly to them.
So a simple explanation like:
The model just prefers round values.
does not explain the result very well.
Nothing pulled the tests back toward 40 or 70 either.
Then came the result that broke my own prediction.
Result 2: hide 40 and 70, and the model still finds them
Version E never prints the literals 40 or 70.
Instead, the thresholds are expressed indirectly as four tenths and seven tenths of a scale defined from zero to one hundred.
My preregistered prediction was that boundary coverage would collapse.
It didn't.
All five versions covered all six registered boundary classes in every valid run.
The valid-run counts were:
A: 29 / 30
B: 28 / 30
C: 28 / 30
D: 30 / 30
E: 27 / 30
Version E reconstructed 40 and 70 exactly.
It also produced 107 records exactly on one of its three registered edges, the same total as the baseline.
That tie is prettier than the result deserves, so I don't want to oversell it.
The 107 count includes the third edge at 39. Looking only at the two main thresholds, Version E produced 103 exact-edge records versus 105 in the baseline.
Still, the result that matters is straightforward:
Removing the literal 40 and 70 did not produce the collapse I predicted.
My prediction was wrong.
The explanation I had published - that the strong boundary behavior was primarily caused by anchoring on printed threshold numerals - no longer survived its own test.
But a failed prediction does not prove the opposite
This is where I made the next mistake.
If my anchoring hypothesis fails, I do not automatically get to say:
The model understood the policy.
I also do not get to say:
The printed numerals had no causal effect.
The study does not establish either claim.
The defensible conclusion is much narrower:
For this policy family, this model snapshot, and this particular way of hiding the literal thresholds, removing 40 and 70 from the text did not reduce registered boundary coverage.
There are several reasons to stop there.
The policy may not have been novel to the model
This policy family had already been public in the repository for several days before the experiment.
That means contamination cannot be ruled out.
If the model had encountered the policy previously, recovering 40 and 70 would not necessarily demonstrate that it derived them fresh from the transformed policy.
This experiment cannot distinguish:
derived it
from:
recognized it
Version E hides the numerals, not the arithmetic
Version E still contains the threshold information.
"Seven tenths of a 0-100 scale" deterministically evaluates to 70.
"Four tenths" evaluates to 40.
So Version E tests the effect of indirection, not the effect of making the boundary genuinely absent.
That is a much narrower experiment.
Each perturbation is represented by one policy instance
Version B tells us that this particular paraphrase did not substantially change the measured boundary coverage.
It does not tell us that paraphrasing never changes boundary discovery.
The same limitation applies to clause ordering and threshold denaming.
One instance per perturbation type is not enough to generalize across all possible rewrites.
Then I discovered that my retraction was wrong too
This part is less flattering, but probably more useful.
After writing the first retraction, I checked the prose mechanically against the published result files.
I found eight material errors.
The worst involved a claim I had described as the part of the original interpretation that still survived.
I had effectively said that none of the versions explored the immediate space below the unstated reference point.
That was false.
Version C contains a record at 38.
I had also treated 39 as though it were the equivalent reference point for Version D, even though Version D moved its lower threshold to 45.
Its comparable reference point is 44.
Another sentence said there were "0 of 464 records" below 39 in Version A.
That wording made it sound like there were no values below 39.
In reality:
117 of Version A's 464 records are below 39.
The zero referred only to a narrow interval immediately below the edge.
Those are very different claims.
Then I checked the correction.
And found another problem.
The corrected prose still said, in several places, that:
The numerals were not the cause.
The experiment does not establish that.
It establishes that this particular removal of the literal numerals did not change the measured coverage.
A failed prediction does not prove its opposite.
I had written that rule into the study.
Then violated it while summarizing the study.
What actually survives
After correcting those mistakes, there is still an interesting pattern.
Across all five versions, the one-unit interval immediately below the unstated reference point is almost empty.
For Versions A, B, C, and E, that interval is:
[38, 39)
For Version D, because the threshold moved, the equivalent interval is:
[43, 44)
Across the entire experiment, there is only one record in those intervals:
Version C's value at 38.
Four of the five policy variants have nothing there at all.
But this needs another qualification.
Values below 39 are not generally rare.
Version A has 117 records below 39.
Version E has 101.
The observation is specifically about the immediate approach to the boundary.
That distinction matters because I had previously conflated:
nothing immediately below 39
with:
nothing below 39
They are not remotely the same claim.
And importantly, the nearly empty approach exists in the baseline just as it does in the version where the threshold numerals were hidden.
So the clean explanation I started with no longer works.
If the gap had appeared only after I removed the printed thresholds, "the model copies visible numbers" would have been an appealing explanation.
It didn't.
What I can say is:
Removing the literal thresholds did not remove the pattern.
What I cannot currently say is why the pattern exists.
I thought I knew.
The experiment told me I didn't.
What developers can actually use from this
The practical lesson is not that generated tests are bad.
It is that boundary coverage is a much weaker metric than it looks.
If a generated test suite says:
6 / 6 boundary classes covered
that tells you the expected categories appeared.
It does not tell you whether the generator understood the decision space, explored nearby alternatives, or simply concentrated values around obvious rule boundaries.
A very cheap diagnostic is to plot the generated numeric values relative to the policy thresholds.
If nearly everything is welded to exact thresholds and their nearest decimal neighbors while the surrounding space is mostly empty, that tells you something the coverage percentage cannot.
Not necessarily that the model failed.
But certainly that:
100% boundary coverage
should not automatically be read as:
the policy was understood
Those are different claims.
The lesson I didn't expect to get from the study
The experiment went through extensive adversarial review.
The prose explaining it did not.
That turned out to be a mistake.
The experimental apparatus had:
- preregistered predictions
- frozen artifacts
- deterministic scorers
- raw records
- review rounds designed to find overclaims
Then I sat down afterward and summarized all of that in ordinary prose.
That is where the errors appeared.
The number errors were in the explanation.
The causal overclaim was in the explanation.
The experiment itself had a rule saying:
A failed prediction does not license the opposite conclusion.
The write-up broke the rule.
So the meta-result I am taking away is simple:
The write-up is part of the research artifact.
If an experiment is important enough to adversarially review, the final article probably deserves the same treatment.
In this case, every material error found so far has been in the layer where evidence gets turned into a story.
That is uncomfortable.
It is also useful evidence about where another verification boundary belongs.
This study started in the comments
One thing I don't want to lose in the write-up: Study 012 came directly out of the discussion on the previous experiment. @deanlee suggested the core falsifier - keep the boundary classes fixed, perturb the wording and ordering of the policy, and see whether the model still finds the same test surface. @alexshev pointed out that repeated generation may be telling us as much about the shape and fragility of the policy text as about the model itself. And @kartik-nvjk pushed on the rejected run and whether something as small as an off-by-one error at the threshold would even be visible behind the clean coverage number.
Those were not just good comments to acknowledge. Together they changed what I tested next. Thanks Dean, Alex, and Kartik for pushing on the result instead of simply accepting the headline.
Everything is public:
Study 012 - Policy Perturbation
The repository includes the preregistration written before the data existed, review records, raw generated tests, result files, the retraction, and the corrections.
The claim I would carry forward is deliberately narrow:
This experiment falsified my prediction that hiding the literal threshold numbers would collapse boundary coverage. It did not establish why the model still found those boundaries.
For now, that is enough.
Top comments (0)