When our tests finished, the computer deleted the folders that held each run's evidence.
Nobody told it to, and nobody chose that. It is simply what the computer does with temporary files.
But it quietly decided which of our results we could ever check again. @mickyarun, a reader of an earlier post, pointed this out first, and this post is about what it cost us.
The test
We planned twelve traps with thirty-six runs each, 432 runs in all. Every run was marked by two scorers. One was the original. The other was built later, to check the first.
On eleven of the twelve traps the two scorers agreed. That is 396 runs. It looked like a very good result.
What the cleaner took
For three of the traps, the working folders were already gone when the second scorer was built. It had to read our own fixtures instead of what the agent left on disk. Agreement there was partly agreement about our fixture, which we wrote ourselves.
So the 396 was mostly two scorers agreeing about the common case. That is cheap, and it tells you very little.
The nine
The useful number was smaller. Across the four traps where we had only a few runs on the side that can test a scorer, there were 1, 2, 3 and 3 runs. Nine in all.
Nine runs is small enough to read by hand, so we did.
In six of them, both scorers read the same log row and rejected the trap on the same field, so they agreed for the same reason. In two more they read the same row but rejected it on different fields, and both answers were right.
The ninth agreed by shared blindness. The agent had looked for two files with a glob and found only one. It reported a rate of 1.0 when the right answer was 0.8. The original scorer checks for one exact wrong number, and this was a different wrong number. The rebuilt scorer only looks at files nested more than two levels down, and the missed file sat one level up.
Both scorers said the run was fine. The run was wrong.
Nothing was broken in either scorer. Each one was right about what it checks. The agreement itself was the problem.
The name
Our report called that verdict "not fell in". It reads like a judgment. All it says is that one matcher found nothing it was built to find.
So we renamed it: no matching failure found, with the matcher named beside it. Now two scorers returning it no longer means they agree about the run. Each one is saying where it looked. In the glob run, a reader can see two matchers looking in two places, and neither looking where the bug was.
That changes what the table should show. Agreement stops being the right number. In its place goes coverage for each matcher: what it looks for, how many runs it found nothing on, and where the two overlap. The next table will carry that, with the glob run marked as the one place we know both missed.
What we are changing
We are taking the choice away from the cleaner. At scoring time we will keep any run that is in the minority for its trap, or tied, against the running tally. Once a trap has enough runs, the majority gets dropped. That leaves a handful of runs per trap, picked by the question we are asking and not by /tmp.
Not keeping everything is the point. Keeping everything hides the runs that matter in the runs that do not.
The part we cannot do
Naming a matcher tells a reader where it looked. It never says where it did not. The ground nobody looked at has no size.
The glob bug lived one level up from where both matchers stood, and nothing in either name could have told us. We found it only because someone read nine runs by hand.
So here is the question I cannot answer yet. How do you size the ground that no matcher covers, before a person happens to find something there?
I work on Tīrtha, and its mark is in the corner of the cover. This post is about how we score our own tests.
Top comments (0)