Earlier this year I built a test-quality harness called killcheck and wrote about it here. The short version of what it found: tests written by AI coding agents often pass while catching almost nothing.
I have spent the months since on a follow-on, entered into a Kaggle research-paper competition about AI coding agents built on Google's Gemma 4 models. The broad question is this. When an AI agent fixes a bug, how do you know the fix is actually right, and do the tools it is handed show it the right parts of the codebase?
The paper submits in November. Results and code come out after the competition closes, so there are no numbers in this post. What I can talk about now is the thing I did before any of it ran.
I pre-registered it.
What that actually means
Before running a single experiment, I wrote down three things.
What result would count as success. What result would count as failure. What I would do in each case.
Then I committed that file to git with a timestamp, so I could not quietly move the goalposts later.
That last clause is the whole point. A plan that lives in your head is not a plan. It is a mood. It updates itself as the data arrives, and it always updates in the direction of the answer you were hoping for, and you never notice it happening because at no moment did you decide to cheat.
A timestamped commit removes that option. The file says what I expected. Git says when I said it. If I later want to argue the goalposts were always somewhere else, the repository disagrees with me in public.
The part that was harder than it sounds
I also fixed minimum sample sizes in advance.
If the data came in under the minimum, the answer would be "inconclusive". Not "close enough". Not "suggestive, pending further work".
Writing that sentence cost me nothing in March. It cost me quite a lot later.
Because when you are two months in and the numbers are nearly there, "close enough" is sitting right next to you being extremely reasonable. It points out that you have done the work. It points out that the direction is clear. It is not even lying. It is just choosing a threshold after seeing the data, which is the same move as not having a threshold at all.
The pre-registered minimum is what makes that conversation short. You already decided. You decided when you had no stake in the answer, which is the only time anyone can decide it honestly.
And then the idea I liked best did not work
I had a favourite hypothesis going in. The kind you want to be true because it would be neat, and because you would get to be the person who showed it.
It did not pan out the way I hoped.
I am reporting that, plainly, in the paper. Partly because the pre-registration gave me no choice, which is exactly what it is for. But also because a clean negative result is still a result. Somebody else is going to have the same neat idea. If the only published work is from people whose neat ideas happened to work, that person burns a few months rediscovering my dead end.
This is the part of the method I would recommend to anyone building something they intend to make claims about. Not because it makes you virtuous. Because it makes the negative publishable. Without the pre-registration, a result that disappoints you just quietly becomes a project you stopped working on, and nobody learns anything, including you.
Why I bother with this on a side project
Nobody is auditing my side projects. There is no reviewer, no ethics board, no grant. I could have skipped all of it and nobody would ever have known.
But the last project taught me something uncomfortable. When I went back through the bugs in my own measurement code, every single one of them had pushed the result in the direction I wanted. Not because I was being dishonest. Because a disappointing number gets investigated and a pleasing number gets written up, and that asymmetry does the work on its own.
Pre-registration is the version of that lesson you apply before you start, instead of discovering afterwards that you have been standing on a scale that was wrong in your favour the whole time.
The lesson, stated plainly
Decide what "it worked" means before you look.
That sentence is easy to agree with and genuinely uncomfortable to carry out, because the moment where it binds is always a moment where you have already invested real time and the honest answer costs you something.
That is also the only moment where it is worth anything.
The paper goes in next month. The results and the code follow once the competition closes. I will write up what actually happened then, including the idea that did not work, because that is the one I pre-registered myself into telling you about.
Top comments (0)