DEV Community: Marvin Okafor

The Antivirus Ate 30GB. Notes From a 50-Hour Experiment on a Laptop.

Marvin Okafor — Thu, 15 Oct 2026 13:30:18 +0000

I ran roughly fifty hours of compute for a research paper on my own MacBook. Not a cluster, not a rented GPU box. The laptop I also use for everything else.

The work is a follow-on to killcheck, the test-quality harness I wrote about here earlier this year. This time it is a paper for a Kaggle competition about AI coding agents built on Google's Gemma 4 models, working through 80-odd real bug fixes taken from popular open-source Python libraries. It submits in November, and the results and code come out once the competition closes, so there is nothing here about what I found.

This is the other story. The one where the idea was never the problem.

The lid

I closed the laptop.

That is it. That is the whole first failure. I closed the lid in the middle of a run, the machine went to sleep, and the job died somewhere in the middle of a batch I then had to work out how to identify and redo.

The fix is embarrassing in its simplicity. Lid open. Charger in. A keep-awake command wrapped around the job so the machine cannot make that decision on my behalf.

I lost real hours to this, and I would like to pretend I lost them once.

The antivirus

Partway through, the machine started swapping heavily. Everything slowed down, including the run, and nothing in my code explained it.

The antivirus scanner had quietly consumed more than 30GB of disk. It was indexing, in real time, every temporary file a long-running experiment produces, which is a lot of them, and it kept the results.

It is uninstalled now.

What makes this one worth telling is that it had nothing to do with my work and no visible connection to it. The symptom appeared in my job. The cause was a piece of software I had not thought about in a year, doing exactly what it was designed to do, to a workload nobody designed it for.

The two copies

I restarted the job while the first copy was still running.

Both copies then wrote to the same files. For a while neither of them was obviously wrong, which is the worst version of this, because the output kept arriving and looked fine.

Every result from the overlapping window went in the bin and got rerun. That was the only honest option. I could not tell, after the fact, which copy had produced which row, and a result you cannot attribute is not a result.

The slow leak

The long-lived worker processes leaked. Memory first, then temp files, gradually, over hours. Nothing dramatic enough to crash. Just a machine that got worse the longer it ran, which meant late results were being produced under conditions that did not match early results.

Three fixes, all of which I should have had from the start.

A fresh process per task, so nothing accumulates across a run. Temp files written to an external SSD instead of the system disk. And a disk watchdog that pauses the run before the disk fills, rather than discovering the floor by hitting it.

What I would tell myself at hour zero

The infrastructure fails before the idea does.

I did not lose a single hour to the research question being wrong. I lost all of them to sleep settings, a scanner, my own fat fingers, and processes that aged badly. None of that is interesting, none of it is in the paper, and all of it is where the time went.

So, the one thing that would have saved the most time, stated as plainly as I can:

Make every run resumable before you start.

Not after the first crash. Before. Assume the machine will stop at an arbitrary moment, because it will, and the only question is whether that costs you five minutes or a day. Checkpoint per task, not per batch. Record what finished, so a restart skips it. Make the job idempotent, so running it twice is safe rather than silently corrupting.

Everything else on this list is a consequence of that one not being true at the start.

A note on the laptop

I could have rented a machine. For some jobs you should.

But fifty hours spread across a few weeks, on hardware I already own, with no meter running, let me stop and restart and change my mind as often as I needed to. That turned out to matter more than raw speed, because I changed my mind a lot. The cost of a bad hour on a rented box is a bad hour plus the bill. On your own laptop, it is just a bad hour.

It is a worse machine and a better arrangement. I would do it this way again. I would just do it with the watchdog in place from the first run, instead of adding it the day after I needed it.

The paper goes in next month. I will write up what it actually found once the competition closes.

A Change That Does Nothing Should Score Zero. Mine Didn't.

Marvin Okafor — Sat, 10 Oct 2026 17:54:39 +0000

I ran a small pilot before the real experiment. It turned up more than ten bugs.

Every one of them was in my own measurement code. Not one of them taught me anything about the question I was actually trying to answer.

This is a follow-on to killcheck, the test-quality harness I wrote about here earlier this year. The new work is a research paper entered into a Kaggle competition about AI coding agents built on Google's Gemma 4 models, looking at how you tell whether an agent's bug fix is actually correct. It submits in November and the results come out after the competition closes, so this post has no findings in it. It has the part that came before the findings, which I think is the more useful half anyway.

The bugs

They were boring. That is what makes them dangerous.

Ordering that was not stable between runs, so the same input produced a different sequence depending on how the filesystem felt that morning. Randomness I had not seeded and did not know was there. Timeouts that fired under load and were recorded as a result rather than as a missing result. Output that simply differed between two identical runs for reasons I could not immediately explain.

None of those announce themselves. None of them crash. They all produce a number, and the number looks exactly like a real number, and you can put it in a table and nobody will blink.

I had written about this pattern after the last project. Then I went and did it again on the new one. Knowing the failure mode exists turns out to be almost no protection at all.

The one trick that worked

Here is the thing that found most of them.

Build controls whose answer you already know.

The clearest example: a change that does nothing should always score zero. If you feed your pipeline something inert and the pipeline reports a result, the result came from your pipeline. There is nothing else it could have come from. You have just caught your instrument manufacturing a number out of nothing.

Mine did not score zero. More than once.

That is the whole technique and it is almost embarrassingly simple. You are not testing the thing you are curious about. You are testing whether the machine that measures it can correctly measure something you already know the answer to.

If it cannot, every number it has ever given you is unverified. Not wrong necessarily. Unverified, which in practice means the same thing when you are about to publish.

Why this is different from just testing your code

I had tests. The tests passed.

Tests check that your code does what you told it to do. A control checks whether the thing it produces corresponds to reality. Those are different questions, and on measurement code the second one is the one that matters, because the failure mode is not an exception. The failure mode is a plausible number.

A plausible wrong number has no symptom. It does not throw. It does not look odd in a log. It sits in your results table being quietly incorrect for as long as you let it, and the only thing that ever catches it is a case where you knew the answer in advance and can see that the machine disagreed with you.

This also explains why the bugs clustered where they did. Everything in my pipeline that had an obvious right answer got checked early, because checking it was easy and the answer was unambiguous. Everything that produced a judgement call went unchecked for much longer, because there was nothing to check it against. So I built something to check it against.

The order this forces on you

What I would do differently, and what I did do this time once I had learned it the hard way, is to build the controls before the experiment rather than after the first confusing result.

It feels like a delay. You want to get to the interesting part. The controls are not the interesting part, they produce no insight, and nobody reads a paper for the section where you confirm your tooling can count to zero.

But every hour spent on them up front is an hour you do not spend later trying to work out whether a surprising result is a discovery or a defect. That second question is much more expensive, because by then you have an opinion, and opinions make you lenient with evidence that agrees with you.

The lesson

Before you trust a number about someone else's system, make your instrument prove it can produce a number it already knows.

A no-op scores zero. An empty input gives an empty result. An identical rerun gives an identical answer. These are not deep insights. They are the floor. But the floor is where the whole table stands, and I have now twice found out the hard way what happens when you skip it.

More than ten bugs before I learned a single thing about the actual research question. I think that ratio is normal. I think most people just do not count.

I Pre-Registered a Side Project. My Favourite Idea Did Not Survive.

Marvin Okafor — Wed, 07 Oct 2026 10:03:29 +0000

Earlier this year I built a test-quality harness called killcheck and wrote about it here. The short version of what it found: tests written by AI coding agents often pass while catching almost nothing.

I have spent the months since on a follow-on, entered into a Kaggle research-paper competition about AI coding agents built on Google's Gemma 4 models. The broad question is this. When an AI agent fixes a bug, how do you know the fix is actually right, and do the tools it is handed show it the right parts of the codebase?

The paper submits in November. Results and code come out after the competition closes, so there are no numbers in this post. What I can talk about now is the thing I did before any of it ran.

I pre-registered it.

What that actually means

Before running a single experiment, I wrote down three things.

What result would count as success. What result would count as failure. What I would do in each case.

Then I committed that file to git with a timestamp, so I could not quietly move the goalposts later.

That last clause is the whole point. A plan that lives in your head is not a plan. It is a mood. It updates itself as the data arrives, and it always updates in the direction of the answer you were hoping for, and you never notice it happening because at no moment did you decide to cheat.

A timestamped commit removes that option. The file says what I expected. Git says when I said it. If I later want to argue the goalposts were always somewhere else, the repository disagrees with me in public.

The part that was harder than it sounds

I also fixed minimum sample sizes in advance.

If the data came in under the minimum, the answer would be "inconclusive". Not "close enough". Not "suggestive, pending further work".

Writing that sentence cost me nothing in March. It cost me quite a lot later.

Because when you are two months in and the numbers are nearly there, "close enough" is sitting right next to you being extremely reasonable. It points out that you have done the work. It points out that the direction is clear. It is not even lying. It is just choosing a threshold after seeing the data, which is the same move as not having a threshold at all.

The pre-registered minimum is what makes that conversation short. You already decided. You decided when you had no stake in the answer, which is the only time anyone can decide it honestly.

And then the idea I liked best did not work

I had a favourite hypothesis going in. The kind you want to be true because it would be neat, and because you would get to be the person who showed it.

It did not pan out the way I hoped.

I am reporting that, plainly, in the paper. Partly because the pre-registration gave me no choice, which is exactly what it is for. But also because a clean negative result is still a result. Somebody else is going to have the same neat idea. If the only published work is from people whose neat ideas happened to work, that person burns a few months rediscovering my dead end.

This is the part of the method I would recommend to anyone building something they intend to make claims about. Not because it makes you virtuous. Because it makes the negative publishable. Without the pre-registration, a result that disappoints you just quietly becomes a project you stopped working on, and nobody learns anything, including you.

Why I bother with this on a side project

Nobody is auditing my side projects. There is no reviewer, no ethics board, no grant. I could have skipped all of it and nobody would ever have known.

But the last project taught me something uncomfortable. When I went back through the bugs in my own measurement code, every single one of them had pushed the result in the direction I wanted. Not because I was being dishonest. Because a disappointing number gets investigated and a pleasing number gets written up, and that asymmetry does the work on its own.

Pre-registration is the version of that lesson you apply before you start, instead of discovering afterwards that you have been standing on a scale that was wrong in your favour the whole time.

The lesson, stated plainly

Decide what "it worked" means before you look.

That sentence is easy to agree with and genuinely uncomfortable to carry out, because the moment where it binds is always a moment where you have already invested real time and the honest answer costs you something.

That is also the only moment where it is worth anything.

The paper goes in next month. The results and the code follow once the competition closes. I will write up what actually happened then, including the idea that did not work, because that is the one I pre-registered myself into telling you about.

I Had Incredible Bosses. They Still Only Used 30% of Me.

Marvin Okafor — Fri, 02 Oct 2026 16:16:44 +0000

I Had Incredible Bosses. They Still Only Used 30% of Me.

A quick note first. My last few posts have stepped away from the project write-ups. The crop rotation one was that. This one is too. A slice of life, not a ruling on how anyone should work.

The technical stuff is not gone. I have been in the middle of work I actually want to write up. Grading AI-written code properly. Putting AI inside a real business system instead of beside it. A couple of builds that are bigger than a feature ticket. Those are coming. This week I just wanted to say the other thing out loud.

I have had incredible bosses.

That includes the job I was let go from. The employer was still wonderful about it. If anything, that was the place where my competence grew fastest. I am not writing this to laugh at old companies. I learnt a lot in those rooms. I was lucky with the people.

The limit was never the people. It was the shape of the role.

The box

When I was a junior, and later when the title changed but the box did not, the work came in clean and narrow.

Build this feature on this site. Run this analysis on that dataset. Make this page faster. Match this design. Ship by Thursday.

The stack was already chosen. The patterns were already agreed. My job was to be good inside the lines, and I was good inside the lines.

I was also not using about 70% of what I knew. Being honest, 30% is me being generous about what I did use.

It was not cruelty. It was design.

Junior roles are supposed to be defined. A company cannot hand a new engineer the whole system and hope for the best. Some of the best things I learnt came from that discipline. How to finish. How to take a review without sulking. How to work inside someone else's decisions.

It still felt like a lid.

What changed

Now I work mostly for myself, on contract.

The businesses I work with say something close to this: here is our tech, we trust you, deliver it how you see fit.

That one sentence changes the whole job.

I have built enterprise applications. I have built mobile apps. I have put AI into a company's actual systems, not as a demo sitting beside the real thing. The stack is no longer a corridor. If the right answer is a different tool, I can pick it. If the right answer is small and boring, I can do that too.

The point is that the choice is mine.

I have been running a bit wild. In a good way. The competence those old jobs built is finally getting spent.

The part I do not want to flatten

Here is the thing I keep coming back to.

That 30% got very good because it was only 30%.

When your whole world for two years is making one system faster, you learn things about speed that you simply do not learn when you are making forty decisions a week about forty different things. The depth came from the years in a lane.

If I had started with the freedom I have now, I would have wasted it. I would have built ten half-things. I would have picked the wrong tool at least twice and learnt the same lessons much more expensively, in front of clients.

The narrow years were not the opposite of growth. They are what made the growth cheap.

And the freedom has a bill attached.

Nobody is above me now. There is no senior engineer quietly fixing my mistake in review before it ships. The extra room I get to use is also extra room to be wrong in, and it is all mine when I am.

So which is better

I do not think that is the question.

There are real perks on the employed side. A good boss. A salary that shows up. People who already know the domain. A role that tells you what finished looks like. Some engineers do their best work there. They want the box, and the box is not an insult. It is a container, and some of the finest work I have seen was made by people who loved the container.

Freelance has its own tax. You own the blank page. You also own the doubt, the scope arguments, and the weeks where nobody tells you whether you are aiming at the right thing. Freedom is not the same as ease.

So this is not "freelance is better." I am not an authority on your career. I am one engineer who did well inside good companies, then did a different kind of well once the whole system was handed over.

Where I landed

If you are the sort of person who wants a defined lane, stay in the lane. A good boss inside a clear role is a gift. I had those. I would not trade them.

If you keep hitting the walls of the ticket, the itch might not be the company. It might be the box.

It took me years to work out which one I am, and I am not sure I could have worked it out faster. You cannot really tell from the inside of only one of them.

I only know which one I am. You will know which one you are.

I Use AI Every Day. I'm Also Googling Crop Rotation.

Marvin Okafor — Sat, 26 Sep 2026 09:48:36 +0000

I have been a software engineer for a little over eight years.

For most of those years I did not really use AI. I wrote the code. I read the docs. I sat in the bugs until they made sense. That was the job.

Then the last year and a half happened.

I started using AI the way some people start drinking coffee again after years of "I'm fine." Quietly, then constantly. Drafts. Refactors. Tests. That awkward first version of an idea I didn't want to stare at for forty minutes. I use it every day now. I like it. I'm not pretending otherwise.

It made me faster. It made some boring work less boring. It made me better at the part of the job that was never just typing: deciding what is worth building, what is sloppy, what will break at 2am.

I am still young. I am still a senior engineer. I am still very much in this AI season of the industry. I am not writing this from a cabin after I "quit tech."

I am writing this because I caught myself planning a farm.

Not as a joke. Well, first as a joke. I actually laughed at myself. "Building an agricultural farm." Who talks like that. I meant land. I meant growing things. Crop rotation on one side, and I kept calling it rotation farming until I looked it up. Vegetables and fruit on another. A fishery in a corner. Cocoa somewhere in the mix, at least enough to test whether I can export something that does not live in a Git repo.

People have to eat.

That sentence is doing a lot of work in my head lately.

AI can write a frightening amount of the code I used to treat as my identity. It does not 100% replace a software engineer. Anyone who has shipped a real system knows the last 10% is still full of judgment, context, ugly constraints, and people. But it almost does the part we spent years getting paid for. "Almost" is a strange place to stand when you are still good at the job.

So I am not shutting the laptop.

I am also not going to pretend the only rational next move is to become the world's best prompt engineer and hope the market stays kind.

The thing I don't want to lose is the entrepreneurial part. The part that likes creating something and actually delivering it. Value you can point at. Not a slide. Not a sprint demo that evaporates after the next rewrite.

Code is wonderful. Code is also weightless. A model can swallow a lot of weightless work.

A pond full of fish is not weightless. Cocoa that has to be harvested, dried, and sold is not weightless. Food is an old industry with new tools, but the product still has to exist in the world. Manufacturing is like that. Farming is like that. A lot of "boring" sectors are like that. They can use AI. They are not made of AI.

That is the thought, not a manifesto.

Stay in software while it still makes sense. Use the tools. Get better with them. Keep the craft.

And in the same season, put some of that restlessness into something AI cannot fully eat. Land. Water. Crops that follow a calendar instead of a changelog. A small business that fails or works in daylight.

I don't have the farm yet. I have tabs. I have questions about soil and seasons I cannot autocomplete my way through. I have the slightly embarrassing feeling of a senior engineer discovering that "retirement plan" for me does not mean a beach. It means I might want to grow food and sell cocoa and still write software on the side because I like both.

If you have been in this industry long enough to remember working without Copilot, and you now work with it every day, you probably know the feeling. Gratitude and unease in the same week.

I got sick recently and almost let a whole week pass with nothing published. That is a small version of the same fear. Time moving, output thinning, the sense that if you don't plant something on purpose the season just goes.

So this is the plant for today.

I am not announcing that engineers should all become farmers. Please do not @ me with a tractor.

I am saying that if AI is doing more of the work you used to guard with your chest, it is not crazy to ask a second question next to "how do I stay employable?"

The second question is: what can I build that still needs a human when the code is cheap?

For me, right now, that question smells like soil. And fish. And cocoa. And a software career I have not finished.

The engines are still on. I am just pointing one of them at a field.

69 Tests. All Passing. Zero Bugs Caught.

Marvin Okafor — Thu, 17 Sep 2026 22:19:31 +0000

An AI model wrote 69 tests for a Python module. Every one passed. Together they caught zero of the eleven bugs I had deliberately planted in that module.

A second setup, pointed at the specific bugs rather than at the module, used 17 attempts and caught ten.

That contrast is the whole project.

What I built

A mutation-testing harness. It changes source code in small ways, flips a comparison, alters a constant, deletes a raise, then runs the existing test suite and records which changes the suite fails to notice. A change nothing catches is a fault your tests cannot detect.

Coverage tells you a line ran. This tells you whether anything would have failed if that line had been wrong. Those are very different numbers. A toy module with one happy-path test showed 47% line coverage and caught 2 of 21 mutations.

Then I pointed an agent at the mutations the tests missed. For each one it writes a single test, and that test is kept only if it both passes on clean code and fails on that specific mutation. Ground truth is a subprocess exit code. No model judges any outcome.

What I found

Three approaches, same model, same token ceiling, twelve widely used Python libraries including cachetools, toolz, tenacity and boltons. 455 mutations generated, 53 of which survive the existing tests and sit on lines those tests actually execute.

approach	caught
one prompt, "write more tests" (556 tests)	9 of 53
one test per call, no targeting (53 tests)	2 of 53
targeted at the specific fault, with a pass/fail gate	44 of 53

I hand-audited the nine it missed. Seven are provably unkillable, six of those being type annotations inside TYPE_CHECKING blocks that never execute at runtime. So 44 of 46 that could be caught at all.

Three findings I think matter more than that number.

Most undetected faults are unreached code, not weak assertions. Only 53 of 133 surviving mutations sit on a line the tests execute at all. I suspected my test commands were scoped too narrowly, so I widened every one of them by 6 to 40 times. The count went from 54 to 53. Down. In mature, human-written code, tests do not mostly execute code without checking it. They mostly do not execute it.

The gate never once rejected a broken test. Across 74 attempts, every rejected draft was a valid, passing test that simply failed to detect the fault. Not one was broken. The gate turns out to have exactly one job in practice, and it is not the job I designed it for.

The tests it writes catch the fault they were shown and almost nothing else. Zero cross-function transfer across 44 kept tests. 36 of the 44 catch exactly one mutation. That is the uncomfortable result, and it belongs next to the 44 rather than underneath it.

Why it actually mattered

Building the agent took a weekend. The rest of the time went into discovering that my measuring instrument kept lying to me.

Eleven bugs in the harness itself. Editable installs that made mutations invisible. Parallel execution corrupting one target. A classifier running on the wrong unit. A bytecode cache serving stale results. An outcome bucket that had never once filled, because it matched on a string my version of pytest does not emit. I had published that empty bucket as a finding.

Two things they had in common.

Every one made the results look better, or made an absence look like evidence. That is selection rather than conspiracy. Debugging is triggered by surprise, and a pleasing result is not surprising. So the filter that removes measurement bugs gets applied unevenly, hard against results you dislike and softly against results you like. The ones that flatter you survive to publication.

Not one was found by reading the code. Every single one was caught by running a check whose outcome I had predicted in advance and getting the wrong answer.

Then I published, and three readers found three more. All of them by pointing at my checks, never at my numbers. Nobody has disputed a single result. Every correction landed on the instrument.

What I would tell you

If you build evaluations for your own work, the harness is the part worth publishing. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works.

I got more out of three comment threads than out of forty hours of my own review.

Repo, with all eleven bugs documented and every result reproducible from a clean clone: killcheck repo

The longer write-ups, if you want them:

Thanks to Vinh Nguyen (@vinhnguyenthanhdn), Ahmet Özel (@ahmetozel) and Zain Dana Harper (@zaindanaharper). Three findings, three checks improved, zero numbers moved.

Six Checks to Run Before You Trust a Number Your Own Harness Produced.

Marvin Okafor — Sun, 13 Sep 2026 21:27:38 +0000

I built a measurement harness. I found ten bugs in it. Every one made my results look better than they were, and not one was found by reading the code.

Here is the checklist that came out of that. It is meant to be used rather than admired, so each check comes with what it proves, what it does not prove, and roughly what it costs to write.

Why the bugs point one way

Debugging is triggered by surprise. A pleasing result is not surprising.

So the filter that removes measurement bugs from your work is applied unevenly. Hard against results you dislike, softly against results you like. Every pass removes more unflattering bugs than flattering ones, and after a few days your instrument has drifted in one direction while nothing in your process is designed to notice.

That is the whole mechanism. It does not require anyone to be dishonest, which is why being careful does not fix it.

I wrote that argument up in full, with all ten bugs and the direction each one pushed, here. The rest of this post is the checklist rather than the case for it.

Check 1: the canary

Replace the thing under test with something unparseable and assert that your harness notices.

def test_canary_reaches_interpreter(target):
    with swapped_source(target, "this is not valid python"):
        result = run_suite(target)
    assert result.failed, "harness did not notice a broken module"

Proves: your change actually reaches the interpreter.

Does not prove: that the change is isolated, or that your scoring is correct, or that a valid change reaches the interpreter. Mine passed happily while a same-length mutation was being silently ignored, because unparseable garbage has a different byte length and therefore invalidates Python's bytecode cache. Vinh Nguyen (@vinhnguyenthanhdn) found that one. A byte-size-preserving variant of the canary closes it.

Caught in my project: editable installs. pip install -e on src-layout packages resolved imports back to the original checkout, so modifications written to a temp copy never executed. Three targets scored 0.000, which reads as "these test suites are terrible" rather than "my harness is broken."

Write this before you write any measurement at all. It is the cheapest check on the list, and it catches the most embarrassing class of failure.

Check 2: the determinism gate

Run the same scoring three times, serially, and require byte-identical output.

runs = [score(target) for _ in range(3)]
assert runs[0] == runs[1] == runs[2], f"nondeterministic: {runs}"

Proves: execution is isolated between runs.

Does not prove: that you are executing the right thing. Three identical runs of the wrong file are still perfectly deterministic.

Caught in my project: parallel execution. Running mutants concurrently produced three different results across four runs on the one target doing real async I/O. I had already recorded an improvement off that data. It was noise.

The important thing is that checks 1 and 2 do not substitute for each other. The canary passes happily while concurrency corrupts your results. The determinism gate passes happily while your imports resolve to the wrong file. You need both, and you need to write down what each one actually proves so you do not talk yourself into believing one covers the other.

Check 3: the negative control

Build one case where the score should be near zero.

# a suite that imports, calls, and asserts nothing meaningful
def test_negative_control():
    score = run_scoring(vacuous_suite, target)
    assert score < THRESHOLD, f"vacuous suite scored {score}"

Proves: your scoring is not generous in a way you would otherwise never look for.

Does not prove: anything about the top of the range.

This is the one I did not have, and it came from Ahmet Özel (@ahmetozel). The logic is worth stating carefully. Everywhere else in a harness, high is good and low prompts investigation. On a negative control, high is the alarm. That gives you exactly one place where a flattering failure is the surprising one, which is the only condition under which debugging fires reliably.

There is a critical design constraint. The control must fail loudly if it passes for a boring reason. A near-zero score is also what you get when nothing loaded, which is precisely what my first bug produced. So the control has to assert separately that the suite passes on clean source, that the module actually imported and executed, and that your other checks still fire.

Mine did not come back at zero. It came back at 7 of 51, and the explanation was a genuine calibration fact: assert x is not None is a real detector, just an extremely narrow one. It catches exactly one class of fault. All seven kills were that one operator and zero were anything else.

So a suite of vacuous tests has a non-zero floor rather than a zero one. A control that came back clean would have taught me nothing.

Check 4: assert the property, not the proxy

When you write a regression test, check the thing you care about rather than a signal correlated with it.

# proxy: correlated with correctness, until it isn't
assert count_cache_files(workdir) == 0

# property: the thing you actually need to be true
before = observe(target)
with mutated_source(target):
    after = observe(target)
assert after != before, "source changed, but behaviour did not"

Proves: the behaviour you depend on actually holds.

Does not prove: that you picked the right property. But it removes a failure mode the proxy has for free.

Caught in my project: my fix for the bytecode problem asserted a cache-file count of zero in the temp copy. Set PYTHONPYCACHEPREFIX and bytecode goes to a central tree keyed on the copy's absolute path. Zero cache files arrive in the copy, and the stale read happens anyway. My test would have passed while the harness lied. Vinh found that too, and ran three branches on his own machine to isolate which variable mattered.

The proxy held under my configuration and failed under a supported environment variable. The property costs the same to assert and does not have that failure mode. That is the entire argument.

Add a meta-test that reproduces the broken case, so you know the regression test has teeth. A regression test nobody has ever seen fail belongs in the same category as a canary that cannot detect what it claims to.

Check 5: predict the outcome before you run it

This is not a check you write. It is a habit, and it caught four of my ten.

Before running any diagnostic, write down what you expect to see. One line is enough. Then run it, and treat any contradiction as stop-and-investigate rather than as something to rationalise on the spot.

There is no code for this one, which is precisely why it is easy to skip. The prediction has to exist before the output does, and nothing in your tooling will remind you.

Proves: nothing on its own.

Does what nothing else does: it manufactures surprise where none would otherwise exist. A check you run without a prediction just produces another number, and you will interpret that number the same way you interpret all your other numbers.

Caught in my project: I knew, independently, that one specific generated test was broken. So I predicted that removing it would make the suite go green. It did not. That contradiction is the only reason I found a bug where a reconstruction step was dropping shared imports and manufacturing failures that were not real. The scores it produced looked entirely plausible, and plausible in a direction I liked.

Check 6: run it where you don't develop it

Install into a fresh environment, outside your own tree, and point it at a repository you did not build around.

cd $(mktemp -d)
pip install <your-tool>
<your-tool> run --target some-repo-you-never-tested-on

Proves: the thing works for someone who is not you.

Does not prove: correctness. It proves that your correctness is reachable by someone else, which is a separate problem. Your checks can all be sound and still never run for a user, because the path they take through your tool is not the path you take.

Caught in my project: my tenth bug, and the one I find most instructive. I had fixed a problem across twelve curated repositories. Then I shipped a CLI where the check that catches that problem was not wired into the command my own quickstart tells people to run first. A stranger pointing it at a normal project would get a confident 0.0000 with no warning at all.

Everything looked correct from inside the repository where I develop it. That is the point. Your development environment is the one configuration you have accidentally optimised for, and it is the one your users are least likely to reproduce.

What this checklist does not cover

Every check above validates the execution path. That your change reached the interpreter, that runs are isolated, that the behaviour actually differs. None of them validates the scorer: the code that reads a result and decides what it means, then aggregates that across everything else.

That gap is not theoretical. Three of my ten bugs were scorer-level. A classifier run on the wrong unit, a reconstruction step dropping shared imports, an assertion style the classifier couldn't see. Not one was caught by a check on this list. All three came from check 5, predicting an outcome and hitting a contradiction, which is a habit rather than a control.

Zain Dana Harper (@zaindanaharper) put the general version of this better than I can: an intact artifact tells you nothing about whether the thing interpreting it is correct. A report can be perfectly well-formed, every byte verified, and carry a wrong number.

I'm building three checks for this now. Known-outcome fixtures that assert the pipeline reports an answer you know by construction. Conservation invariants so counts have to reconcile at every stage. Explicit unit metadata on anything carrying a count, so a consumer can assert what it's being handed rather than assume.

The first one found a bug on its first run. My outcome classifier decided a result was a collection error only if the output contained a specific phrase, one that the pytest version I'm on never actually emits. So that branch had never fired, for any target, ever. Worse than the bug: I had published the empty bucket as a finding. "Zero error outcomes across all twelve targets" read as reassurance about data quality when it was the signature of dead code. A uniform zero across twelve independent targets should have been suspicious on its own. Real data is rarely that clean.

That one doesn't fit the pattern in the rest of this post, and it's worth naming why. It didn't inflate a metric, and the primary number is unaffected. What it did was make a null result look like evidence. Until the scorer checks are finished, treat this list as covering execution and leave interpretation to check 5.

The order to do these in

Canary first, before any measurement exists at all.
Determinism gate before you trust any number that comes out.
Negative control before you interpret a good result.
Fresh-environment test before anyone else runs your tool.
Property-not-proxy every single time you write a regression test.
Predict-then-run on every diagnostic, forever.

The ordering matters because each one gets harder to run honestly once you have results you are attached to.

Three traps that no check catches

These are scoring decisions rather than checks. Nothing on the list above will catch them, so you have to decide them deliberately and then say what you decided.

Batch versus per-test scoring. If one broken item invalidates a whole batch, your number is a floor rather than a measurement. Both are defensible. Publishing one while implying the other is not. Say which you have.

Budget matching. When you compare approaches that spend resources differently, there is no neutral unit. Matching calls starves one arm of output. Matching output tokens effectively rebuilds a different arm. Every choice advantages someone. Report the full resource vector, and say plainly that the unit was chosen before you saw the results, assuming it was.

Controls die at small n. I built two versions of a held-out control and abandoned both. One ended up with a denominator of 1. The right move is to state that you have no control, not to dress up something weaker and call it one. A weak control that nobody flags is worse than an admitted absence, because it transfers confidence you have not earned.

What this costs and what it bought

Ten bugs, all flattering, none found by inspection.

Then two readers found two more by pointing at my checks rather than at my numbers. Neither time did a result move. Nobody disputed a single finding. All of the scrutiny landed on the instrument, and both times the instrument was wrong in a way that could not be seen by reading it.

That is the argument for publishing the instrument rather than the finding. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works.

I am considering putting the full version of this together, with the checks as drop-in code, the pre-registration and commit-ordering templates, and the eval set as a worked example.

If you build evaluations, what would you want in it that isn't here, and what would you not bother with? The second question is the more useful one to me.

Two Readers Broke My Checks. Not One of My Numbers Moved.

Marvin Okafor — Fri, 11 Sep 2026 23:20:40 +0000

Last week I published a post about nine measurement bugs I found in my own evaluation harness, all of which happened to favour my results. Two readers replied with things I hadn't found.

Both were about my checks rather than my numbers. I verified both and fixed both. My results did not change at all.

That turned out to be the interesting part.

Reader one: a canary that could not detect what it claimed to

Some setup. The harness mutates a source file in a temp copy and runs the test suite against it. To prove that mutations actually reach the interpreter, I had a canary: overwrite the module with unparseable garbage and assert the suite fails.

Vinh Nguyen pointed out that CPython's default bytecode invalidation keys on source mtime in whole seconds plus source size. A mutation flipping < to > is byte-size identical. If a stale .pyc is sitting there with a matching header, Python skips recompilation and imports the original bytecode.

The mutation is on disk. The original behaviour executes. The suite passes. The mutant gets recorded as surviving.

Here is why my canary could not catch that. Unparseable garbage has a different byte length. So the pyc genuinely does invalidate, so the canary fires, so it reports success. It was answering a question I wasn't asking.

I checked the mechanism first.

# compile a pyc from this
a == b

# overwrite with a byte-identical mutation, force mtime back
a != b

# import: stale bytecode executes, original comparison runs

Reproduced on the first try.

Then I checked whether it reaches anything in my harness. It doesn't. Every temp-copy call site passes ignore_patterns("__pycache__", "*.pyc", ...), so no pyc ever arrives in the copy. I verified it by copying a target with 19 real pyc files and confirming zero arrive.

Here is the part I keep thinking about. I wrote that line to avoid copying junk. It turned out to be load-bearing for correctness, and I had no idea until someone pointed at the mechanism.

The reason an instrument is correct is often not the reason you wrote the code that makes it correct. A line added for tidiness that silently does safety work is a line someone removes during a cleanup, with no way to know what it was holding up.

I added a byte-size-preserving canary alongside the original. Eight of twelve targets are eligible and all pass. The other four report N/A rather than being silently skipped.

Reader two: the negative control I didn't have

Ahmet Özel came at it from a different direction. His reading of the original post was that "debugging is triggered by surprise" explains a one-directional bias without anyone needing to be dishonest, which is exactly the argument I was making.

His counter-measure was one I didn't have. Keep a deliberate negative control. A case where the score should be near zero.

Everywhere else in a harness, high is good and low prompts investigation. On a negative control, high is the alarm. That gives you one place where a flattering failure is the surprising one, which is the only condition under which debugging reliably fires.

He also made a sharper point about pre-registration, which I was doing. Writing down your expected number helps when you're wrong. It does nothing when the instrument is broken in a way that produces the number you predicted.

One of my original nine was exactly that. A classifier misread self.assertEqual as "no assertion" while I was testing a hypothesis that models write assertion-free tests. It would have handed me my own prediction with a confirmation attached.

So I built the control. A suite of deliberately vacuous tests. Import the module, call things, assert nothing meaningful. I expected near zero.

It killed 7 of 51 mutants. 13.7%.

That is not a bug. assert x is not None is a real detector, just a very narrow one. It catches exactly one class of mutation: a function that started returning None. All seven kills were that operator. Zero were constant mutations and zero were comparison mutations.

So a suite of vacuous tests has a non-zero floor rather than a zero one. That is a calibration fact about my scoring that I would never have learned from a control that came back clean.

The design constraint is the practical part, and it took me a second attempt to get right. A negative control has to fail loudly if it passes for a boring reason. A near-zero score is also what you get when the module never loaded, which is precisely what bug 1 in my last post produced. Three targets at 0.000 that read as a finding rather than as a fault.

So the control asserts separately that the suite passes on clean source, that the module actually imported and executed (coverage-confirmed, 37 lines), and that both canaries still fire.

Round two: my fix asserted a proxy

Vinh came back after I described the fix and found a hole in it.

My regression test asserted pyc_count == 0 in the temp copy. But set PYTHONPYCACHEPREFIX and bytecode goes to a central tree keyed on the copy's absolute path. Zero pyc files arrive in the copy and the stale read happens anyway.

My test would have passed while the harness was lying.

He ran three branches on his own machine to isolate the real dependency:

no prefix, fixed work path: fresh
prefix, same fixed work path: stale, zero pyc in the copy
prefix, unique directory per mutant: fresh

I reproduced all three. What actually protects me is a second accidental thing. Every copy site uses tempfile.TemporaryDirectory(), which gives a unique path per mutant, which is branch three. I verified it across every call site and ran a full twelve-target sweep with the prefix set. It diffs byte-identical against the committed results.

Two accidental protections. Neither written for the reason it works. ignore_patterns to avoid junk. TemporaryDirectory because it cleans itself up. Both are now commented as load-bearing and name the environment variable explicitly, because anyone adding a --work-dir flag would remove one of them and have no way to know.

The rule underneath this is the most useful thing I got out of the whole exchange.

My test asserted a proxy. Pyc count is a proxy. "The observed behaviour actually changed" is the property. The proxy held under my configuration and failed under a supported environment variable. The property costs the same to assert and does not have that failure mode.

So I rewrote the test to push a byte-size-preserving mutant through the real code path and assert the observed outcome isn't "survived." I added a meta-test that reproduces the stale branch to prove the check has teeth. A regression test nobody has ever seen fail sits in the same category as the canary that couldn't detect this in the first place.

One result that came along the way

While I was in there I closed something I had deferred: hand-labelling the nine mutants the agent couldn't kill. Seven are provably equivalent, six of them type-hint mutations inside TYPE_CHECKING blocks that are never evaluated at runtime. Two are genuine misses, traced to the agent testing the wrong call pattern and the wrong function. So the claim moves from 44 of 53 with an unbounded equivalence caveat to 44 of 46 killable, 95.7%, with the per-mutant arguments published.

Update: the tenth bug, three weeks later.

After publishing this, I packaged the harness as a CLI so other people could run it, then field-tested it against three repos I hadn't built it around: pytest, attrs, and jsonschema.
It returned 0.0000 on pytest. Zero of 35 mutations detected, reported confidently, with no warning that anything was wrong.
It's the src-layout bug from item one of this list. The editable install resolves imports back to the original checkout, so the mutated copy is never executed. Exactly the bug I said I'd found and fixed, which I had - for the twelve repositories I curated by hand.
What I hadn't done was wire the check into the command people actually run. The canary exists. It works. It catches this every time. But it only ran under verify, and score is the command my own quickstart puts first. So a stranger installing the tool, pointing it at a normal src-layout project, would get a confidently wrong number reading "your test suite detects nothing" when the truth is "my harness never executed your code."
The shape is worth sitting with. Both reader findings in this post were about checks that couldn't detect what they claimed. This one is a check that could, and simply wasn't called. Same outcome for the user. And I found it the same way as all the others - not by reading the code, but by running it somewhere I hadn't run it before and getting an answer I couldn't explain.
score and harden now abort on canary failure rather than returning a number. There's a --skip-canary for people who know what they're doing, and it says loudly that the results are unverified. The error message names src-layout as the likely cause and suggests the fix, because "mutations are not reaching this target's test process" tells a user nothing.
One more thing that came out of it. pytest is a hard case - it imports itself before test collection runs, so the usual -o pythonpath=src fix doesn't work. Setting PYTHONPATH before the interpreter starts does. With that, pytest scores 0.657 instead of aborting. So the tool needed a way to set environment variables, not just command-line flags, which I'd never have discovered without a target that broke in an unusual way.
Nine bugs became ten. The tenth is the one I'd have been least likely to find on my own, because everything looked correct from inside the repository where I develop it.

What I actually took from this

Two rounds. Two readers. Four checks improved. Zero numbers moved.

Nobody disputed a finding. All of the scrutiny landed on the instrument, and both times the instrument was wrong in a way I could not have seen by reading it.

That is the argument for publishing the harness rather than the headline. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works. I got more out of two comment threads than out of forty hours of my own review.

Two things worth stealing:

Assert the property, not the proxy. Same cost. One fewer failure mode.

Build one check where a good number is the alarm. Everywhere else, surprise is what triggers debugging, and a flattering bug never surprises you.

Nine Bugs in My Own Evaluation Harness. Every One Made My Results Look Better.

Marvin Okafor — Thu, 10 Sep 2026 21:24:21 +0000

I built an evaluation harness. I found nine bugs in it. Every single one would have made my results look better than they were.

Nine out of nine, all pointing the same way. That is not a coincidence, and I do not think it is specific to me or to my project. I think it is a structural problem with any evaluation you build for yourself.

Here is the mechanism, before any of the evidence.

Debugging is triggered by surprise

You do not audit numbers. You audit numbers that bother you.

When a result comes back disappointing, you go looking for the reason. You check the setup, you re-run it, you add logging, you find the bug. The bug gets fixed and the number moves.

When a result comes back good, none of that fires. Nothing feels wrong. There is no surprise to investigate. You write it up.

So the filter that removes measurement bugs from your work is applied unevenly. Hard against results you dislike. Softly against results you like. Every pass through that filter removes more unflattering bugs than flattering ones.

Run that loop for a few days and your instrument has drifted in one direction, and nothing in your process is designed to notice. The bugs that survive to publication are disproportionately the ones that helped you. Not because anyone was dishonest. Because they never triggered the thing that catches bugs.

I knew this argument in the abstract before I started. It did not stop me from writing nine of them.

What the harness did

Only enough context for the bugs to make sense.

Mutation testing changes your source code in small ways. Flip a comparison. Change a constant. Delete a raise. Then it runs your test suite and checks whether anything failed.

  def withdraw(balance, amount):
-     if amount <= 0:
+     if amount < 0:
          raise ValueError("amount must be positive")

If the suite stays green, that is a fault your tests cannot detect.

Coverage tells you a line ran. This tells you whether anything would have complained if the line were wrong. Those are very different questions. A toy module with one happy-path test sits at 47% line coverage and a 9.5% mutation kill score.

The harness generated mutations, ran suites against them, and scored how many got caught. Roughly 40 hours of work.

The nine bugs, and which way each one pushed

Editable installs made mutations invisible. pip install -e on src-layout packages resolved imports back to the original checkout, so mutations written to a temp copy never executed. Three targets scored 0.000. Direction: reads as "these test suites are terrible" rather than "my harness is broken." It made the problem I was solving look bigger.
Parallel execution corrupted one target. Running mutants concurrently produced three different results across four runs on the one target doing real async I/O. A recorded improvement of 0.27 to 0.77 was noise. Spurious failures get counted as the mutation being detected, and detection was the number I was maximising. Direction: inflated the result.
A file picker chose an unrelated test file for the hardest target, feeding the model irrelevant context exactly where context mattered most. Direction: made a baseline look worse than it was.
A classifier categorised batches instead of individual tests. One good test in a batch of 69 would have marked all 69 as good. Direction: inflated quality.
A reconstruction step dropped shared imports and manufactured test failures that were not real. Direction: understated a baseline's capability.
An extractor only scanned top-level functions, so a valid unittest.TestCase response was discarded as "no test found." The retry loop then received a harness error instead of real pytest output, which disabled the exact mechanism I was measuring. Direction: understated the agent.
self.assertEqual(...) was classified as "no assertion." I was testing a hypothesis about models writing assertion-free tests at the time. Direction: would have manufactured my own hypothesis and handed it back to me.
A pre-registered metric was not computable on dunder-dispatched code. It read as a real near-zero rate instead of as undefined. Direction: false signal.
A documentation figure that had already survived two audits. One arm was recorded as having zero clean-pass failures when it had three. Direction: flattered a comparison.

Note that they do not all inflate the headline number. Three of them understate a baseline or an arm. That still counts as favourable, because a worse baseline makes the thing I built look better by comparison. "Favourable" means favourable to the story, not favourable to one metric.

Bug 9 is the one I find hardest to be relaxed about. It had been looked at twice. Two audits, both of which read past it, because the figure was consistent with what we expected to see and nothing about it invited a third look.

None of them was found by reading code

This is the part I would most want someone to take away.

Not one of those nine was caught by re-reading the function. I had already read the functions. Reading code that you wrote, looking for a bug you do not yet believe exists, is close to useless.

Every single one was caught the same way: by running a check whose outcome I had predicted in advance, and getting a different answer.

The clearest case was bug 5. I knew, independently and from earlier output, that one specific generated test was broken. So the prediction was simple. Remove that one test, and the suite goes green.

I removed it. The suite did not go green.

That contradiction is the only reason I found the dropped-imports bug before its numbers went into anything. There was no other signal. The scores it produced looked entirely plausible. They were plausible in a direction I liked, which is why nothing else would have prompted me to look.

The prediction is what does the work. A check you run without a prediction just produces another number, and you will interpret that number the same way you interpret all the others.

Two checks worth stealing

Both are cheap. Both caught things.

The canary. Overwrite the file under test with unparseable garbage and assert that the suite fails. This proves your mutation actually reaches the interpreter. If the suite passes while the file is syntactically invalid, you are not testing what you think you are testing. This is what would have caught bug 1 on day one.

The determinism gate. Run the same scoring three times, serially, and require byte-identical output. This proves execution is isolated. This is what catches bug 2.

The important part is that these prove different properties, and neither substitutes for the other.

The canary passes happily while concurrency silently corrupts your results. Your mutation reached the interpreter, so the canary is satisfied, and the numbers are still garbage.

The determinism gate passes happily while imports resolve to the wrong file. Three identical runs of the wrong thing are still perfectly deterministic. The gate is satisfied and the score is meaningless.

You need both, and you need to write down what each one actually proves, so you do not talk yourself into believing one covers the other.

The check that caught one of my own targets

Three hours before my deadline I did a clean-clone reproduction run to verify the reproducibility claim.

The determinism gate quarantined one of the project's own targets. Eleven of twelve reproduced exactly. The twelfth varied.

I reported it in the README instead of fixing it.

A check that has never caught anything is indistinguishable from a check that cannot catch anything. Mine had just caught something, and the something was mine. Removing that from the record would have made the project look better and the instrument look worse, which is exactly the trade this entire post is about.

What to actually do

Before you measure anything you built:

Write down what your instrument would look like if it were lying to you. Then build the check that catches specifically that, and run it before you have any results you are attached to.

Then write down which direction each possible lie would push your result.

That second list is the one that matters, because it is a list of the checks you will be least motivated to run. Every item on it is a place where a bug will feel like a finding. You will not notice those on your own. Nobody does. That is the whole reason the nine came out nine for nine.

I Grade AI Agent Code for a Living. Here's the 12-Point Checklist I Run Before Trusting Any of It.

Marvin Okafor — Mon, 07 Sep 2026 23:17:07 +0000

Your agent produced 400 lines in nine seconds. It compiles. The tests pass. The demo works.

None of that tells you whether it's correct.

I evaluate agentic AI coding output against structured rubrics professionally - correctness, instruction adherence, edge-case handling, the whole grid. Separately, I've spent seven years shipping production systems: high-traffic e-commerce, multi-tenant healthcare data, a national eLearning platform handling tens of thousands of concurrent applications. The checklist below is what happens when those two things collide. It's the pass I run before I'll put my name on anything an agent wrote.

None of it is exotic. All of it is stuff agents get wrong constantly, and reviewers skip because the code looks fine.

1. Does deleting the error handling break any test?

If not, your error paths are untested decoration. This is the single fastest way to find out whether a test suite is real. Comment out a catch block and run the suite. Green? You have no coverage of the thing most likely to hurt you.

2. What happens on the second call?

Agents write beautiful single-execution logic. Ask: if this runs twice - retry, duplicate message, user double-click - does it produce one side effect or two? Idempotency is seldom in the generated code unless you asked for it explicitly, and it's almost always required in production.

3. Is every dependency pinned, and does the lockfile exist?

"Latest" is not a version. I've root-caused a bug where a minor version bump silently collapsed TypeScript types to never across a monorepo - semver protects runtime behaviour; it promises nothing about type inference or subtle behavioural edges. Agents love unpinned ranges. Pin them.

4. Is it mocking the thing it's supposed to be testing?

The most common fake-coverage pattern I see: mock the database, assert the mock was called correctly, declare the data layer tested. If the correctness property is enforced by the database - constraints, row-level security policies, transaction isolation - a mock verifies your assumption about the policy, not the policy. Run against a real, disposable instance. I maintain a security suite that validates RLS policies against actual Postgres for exactly this reason.

5. Does it check what should be invisible, not just what's visible?

Access-control tests written by agents almost always assert "the authorised user can see their data." The security-relevant half is "the unauthorised user cannot." Test negative cases explicitly - and remember that some systems (RLS being the classic) filter silently rather than erroring, so "no exception raised" is not the same as "correctly denied." Assert the actual result set.

6. How many database round-trips does the happy path make?

ORMs plus agents produce N+1 queries at an impressive rate, because each line looks perfectly reasonable. Log the queries for one request. Count them. The number is usually higher than anyone guessed, and it's where a large share of real latency wins hide.

7. What are the timeout and retry policies - explicitly?

Not "does it retry," but: how many times, with what backoff, and what happens when retries are exhausted? Unbounded retries against a struggling downstream service is how a partial outage becomes a full one. Agents default to either no retries or naive infinite ones.

8. Does it handle partial failure, or only total failure?

Total failure is easy - the call throws, you catch it. Partial failure is the hard case: three of five writes succeeded, the response timed out but the operation actually completed, the queue delivered twice. Generated code is overwhelmingly written as though operations either fully succeed or fully fail.

9. Are the IAM permissions scoped, or is it wildcards?

Agents reach for permissive policies because permissive policies make the demo work. Any wildcard in a generated permission set is a finding, not a default. Same for over-broad database roles - a correct policy can still leak data if the connecting role has privileges that sidestep it.

10. Is there anything in here that only works because of a race the tests never trigger?

Concurrency is where "semantically wrong about failure" gets most expensive. Check-then-act patterns, non-atomic read-modify-write, missing transaction boundaries. Tests run sequentially; production doesn't.

11. Can someone else reproduce this from a clean clone?

One command, fresh machine, same result. If setup requires tribal knowledge or an undocumented sequence of steps, your correctness is unverifiable by anyone but you - which, for anything that will outlive your attention span, means it's unverified.

12. Is the reasoning written down anywhere?

Which ambiguity did you resolve, and how? What did you deliberately not handle? Agents produce code without provenance - no record of what was considered and rejected. That gap is a real maintenance liability, because the next person can't distinguish a deliberate decision from an accident. Write down the why, especially for the non-obvious calls.

The pattern underneath all twelve

Every item is a variant of the same thing: agents are excellent at code and unreliable about consequences. They handle the path you described and quietly assume the paths you didn't. The failure mode people complain loudest about - hallucinated APIs - is the easy one, because it's loud and any test catches it. The dangerous one is syntactically perfect code that's confidently wrong about what happens when something breaks.

Which means the review skill that matters now isn't "can you spot bad code." It's "can you enumerate the failure modes nobody wrote down." That's not a new skill. It's the thing senior engineers have always done. It just got a lot more load-bearing.

What I'm building from this

I'm turning this checklist into something executable - a fault-injection harness that stress-tests agent-generated infrastructure code against realistic failure conditions (retries, partial outages, IAM misconfigurations, concurrent access) with deterministic pass/fail checks instead of eyeballing. Chaos engineering, pointed at AI output. It'll go up on my GitHub and portfolio as I build it in the open, along with what breaks and why.

What's on your list that isn't on mine? I'm collecting failure patterns for the harness's scenario set, and the ones that come from people who've been burned in production are worth more than anything I can invent. Drop them in the comments.

Part of an ongoing series on production debugging, performance engineering, and evaluation infrastructure for AI systems.

I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.

Marvin Okafor — Tue, 01 Sep 2026 18:01:34 +0000

Here is a Python function and a test for it.

def withdraw(balance, amount):
    if amount <= 0:
        raise ValueError("amount must be positive")
    if amount > balance:
        raise ValueError("insufficient funds")
    return balance - amount


def test_withdraw():
    assert withdraw(100, 30) == 70

That test gives you 47% line coverage. It gives you a 9.5% mutation kill score.

The harness generates 21 small breakages of that module. The suite notices 2 of them.

That gap is the whole reason I started this project. Line coverage is the default test-quality signal in most of the industry, and it measures whether a line ran. It does not measure whether anything would have complained if the line were wrong. AI-generated tests are unusually good at producing that shape: high coverage, low detection.

Mutation testing measures the real thing. You break the code in small ways and check whether the tests notice.

  def withdraw(balance, amount):
-     if amount <= 0:
+     if amount < 0:
          raise ValueError("amount must be positive")

If no test fails, that is a bug your suite cannot detect. It has been sitting there the whole time.

Mutation testing never went mainstream, and I think the reason is simple. It hands you a wall of surviving mutants and no path to fixing any of them. It tells you that you have a problem and then leaves.

So the idea was an agent that closes the loop. Find the survivors. Write tests that kill them. Gate each generated test on a hard criterion: keep it only if it passes on clean code and fails on the mutant. Ground truth is a subprocess exit code. No model judges any outcome.

I built it over about 30 hours for the micro1 Frontier Engineering Challenge, which had around 7,800 registrants.

The tool works, sort of. It is incomplete and I will get to the numbers. But that is not the interesting part of the weekend.

The interesting part is that my measuring instrument kept lying to me, and it lied in a consistent direction.

The finding that broke my own premise

Before I ran a single agent call, I ran the harness across 12 widely-used, well-maintained Python libraries: cachetools, validators, natsort, dictdiffer, toolz, voluptuous, python-slugify, python-dotenv, shortuuid, boltons, aiofiles, tenacity.

455 mutants generated. 133 survived the existing test suites.

Then I checked something I had assumed I would not need to check. Of those 133 survivors, how many sit on a line the tests actually execute?

53.

The rest were never run at all. Not weakly tested. Not vacuously tested. Untested.

I suspected my test commands were scoped too narrowly, so I widened them per target, between 6 and 40 times more test code. If the "executes but does not assert" category was real and I was just missing it, that number should climb.

It went from 54 to 53. Down.

And where widening changed anything, it converted unreachable mutations directly into kills. It did not move them into the middle category. It skipped it.

So in mature, human-written Python, the vacuous test failure mode is rare. Where these suites fail, they fail by not running the code at all.

The story I had absorbed about tests that execute everything and assert nothing is a story about AI-generated tests. It is not a story about human ones. I had to reframe the project before I had built the main part of it.

That was the first sign that what I was actually building was a measuring instrument, and that I had not been treating it like one.

Eight bugs in the instrument

Every one of these would have produced a confident, wrong, publishable number.

Editable installs made mutations invisible. pip install -e on src-layout packages resolves imports back to the original checkout. My mutations were written to a temp copy, so they never executed. Three targets silently scored 0.000. That would have read as "the agent fails on src-layout packages," which is a finding. It is just not a true one.
Concurrency corrupted one target. Running mutants in parallel gave me three different survivor sets across four runs, on the one target doing real async I/O. I had already drafted a result of "0.27 to 0.77" off that. It was noise. Note which way it pointed: spurious failures get counted as kills, and kills are the number every arm is trying to increase.
A file picker chose the wrong test file. On the hardest target. Which means the model would have been shown irrelevant context in exactly the place where context mattered most.
A classifier was about to run on the wrong unit. It classified batches, not individual tests. One strong test in a batch of 69 would have marked all 69 as strong.
A reconstruction step dropped shared imports. This manufactured test failures that were not real failures.
An extractor only scanned top-level functions. So a perfectly valid unittest.TestCase response got discarded as "no test found." Worse, the agent's retry loop then received a harness error instead of real pytest output. That quietly disabled the exact mechanism I was trying to measure.
self.assertEqual(...) was classified as "no assertion." This one would have manufactured precisely the finding I was hypothesising. It would have handed me my own conclusion.
A pre-registered metric was not computable on dunder-dispatched code like __call__ and __or__. It read as a real near-zero rate rather than as undefined.

The two things they had in common

They all pointed the same way.

Every single one of those eight would have made my result look better, cleaner, or more publishable. Not one of them would have made the agent look worse than it was.

I do not think that is a coincidence, and I do not think it is a conspiracy either. It is attention. When a number disappoints you, you go looking for the reason. When a number pleases you, you write it up. So the measurement bugs that survive all the way to publication are disproportionately the ones that helped you.

That is a selection effect operating on your own debugging, and you cannot fix it by being careful. Careful people are exactly as motivated to stop investigating when the number looks good.

None of them was found by reading code.

Every one was caught by running a check whose outcome I had predicted in advance, and getting the wrong answer.

The clearest case was bug 5. My prediction was: remove this one known-bad test and the suite goes green. It did not go green. That contradiction is the only reason I found the dropped-imports bug before I trusted the numbers it was feeding me.

I would not have found it by rereading the function. I had already read the function.

The results, with the caveats attached

Three arms, same model, same token ceiling.

Arm	What it does	Mutants killed
A	One prompt: "write as many tests as warranted." The brief's specified baseline.	—
B	One test per call, same call count as the agent. No mutation hint, no gate, no retry.	1
C	The agent: mutation diff in context, execution gate, one retry with real pytest output fed back.	9

On the 15 mutants the agent covered, B killed 1 and C killed 9. Keep rate was 60%, so the gate was genuinely filtering rather than rubber-stamping. Nine retries fired and three succeeded.

Now the parts that matter just as much.

The agent ran on 2 of 10 targets. The API budget ran out mid-run. I did not swap in a substitute model to finish the sweep, because then the comparison would not be a comparison.

Those two targets are the ones where the baseline performed worst. That is not a random sample, and I have no way to argue it is representative.

All 9 kills were on the two cheapest mutation types. There was zero cross-function transfer. Seven of the nine kept tests kill exactly the one mutation they were written for, and nothing else. That is a real limitation, not a rounding error.

One result that inverted my hypothesis

I had pre-registered a prediction that gate-passing tests would mostly be vacuous. Bare existence checks. Tests with no assertion that "kill" a mutant by crashing rather than by detecting anything.

That is not what happened. The none category was empty. Eight of the nine kills were real assertion failures.

And of the six discarded drafts, zero failed on clean code. All six were valid, passing tests that simply did not detect the bug.

So the gate was not catching broken tests. It was catching working tests that miss. That is a more interesting failure mode than the one I predicted, and I only know it because the prediction was written down first and was wrong in a specific way.

The determinism check that quarantined my own target

Three hours before the deadline I did a clean-clone reproduction run to verify the reproducibility claim.

The determinism check runs each target three times serially and requires the survivor sets to be byte-identical. Eleven of twelve reproduced exactly. The twelfth varied.

I reported it in the README instead of fixing it. A check that has never caught anything is indistinguishable from a check that cannot catch anything, and the first thing mine ever caught was one of my own targets. Removing that from the record would have made the project look better and the instrument look worse.

I missed the submission by 11 minutes.

That is annoying in a way I do not want to dress up. But the reproduction run is what found the twelfth target, and running it is the reason I trust the other eleven.

What I would tell you to take from this

Before you measure an agent, write down what your instrument would look like if it were lying to you. Then build the check that catches exactly that.

And write down which direction each possible lie would push your result. That second list is the important one, because it tells you which checks you will be least motivated to run.

The repo is private while I finish the write-up. If you build evaluations for agents and you have hit this, I would like to compare notes, particularly on catching measurement bias before it reaches a number you have already started believing.

5,900 Engineers Just Registered for a Hackathon Where Using AI Is the Point. Here's How It Will Actually Be Judged.

Marvin Okafor — Fri, 28 Aug 2026 11:16:11 +0000

Most hackathons treat coding agents as something between a crutch and cheating. micro1's Frontier Engineering Challenge, which kicks off today, inverts that entirely: you're expected to use coding agents. The competition isn't about whether you can generate code. It's about whether you can generate code that survives scrutiny.

That's a fundamentally different game, and judging by how most engineers approach AI-assisted work, a lot of the field is about to optimise for the wrong thing.

The details

What: micro1 Frontier Engineering Challenge 2026 - a free, global, online competition, run as a three-day sprint where you use coding agents to tackle a real-world software engineering problem
When: August 28–31, 2026. The full challenge statement is released at kickoff - August 28 at 15:00 UTC
Format: Online, individual (team size 1), free
Registrations so far: ~5,900
Why it matters beyond the prize: micro1 has indicated that top-performing participants get considered for paid opportunities with them

The problem statement is deliberately withheld until kickoff, so nobody gets to pre-build. Everyone starts cold.

The line in the announcement that tells you everything

Buried in the challenge description is the sentence that should reframe your entire strategy: AI can produce convincing code in seconds - real engineering begins when convincing is not enough: incomplete requirements, hidden dependencies, difficult edge cases, failure modes, and decisions that require technical judgment.

And the deliverable: a solution that is correct, reproducible, testable and clearly explained.

Read those four words again - correct, reproducible, testable, explained. Not "impressive." Not "feature-complete." Not "shipped fastest." If you've ever built or graded an evaluation rubric, you recognise immediately what that list is: it's a rubric where three of the four criteria have nothing to do with whether your code runs.

Where I think most of the field will lose points

I evaluate agentic AI coding outputs against structured rubrics professionally - rubric construction, adversarial prompt design, deciding which checks can be programmatic and which need human judgment. So I'm reading this challenge less as "what should I build" and more as "where does this rubric bite." My honest read:

1. Reproducibility is the silent killer. "It works on my machine after four hours of undocumented agent conversation" is not reproducible. If a judge can't clone your repo and get the same result, correctness is unverifiable - and unverifiable correctness scores as zero, not as partial credit. Pin your dependencies. Commit your lockfile. Containerize. Make the setup a single command.

2. "Testable" doesn't mean "has tests." It means the tests actually prove the thing that matters. A test suite that only covers the happy path proves your agent could write a happy path. The edge cases and failure modes the challenge explicitly calls out - those need tests that would fail if the behaviour were wrong. If deleting your error-handling doesn't break any test, you don't have error-handling coverage; you have decoration.

3. Agents are confidently wrong about failure, not about syntax. This is the pattern I keep seeing in evaluation work: the generated code is syntactically perfect and semantically wrong about what happens when things break. Retries, timeouts, partial writes, duplicate messages, concurrent access. Given the challenge explicitly names failure modes as part of the frontier, this is almost certainly where the separation happens.

4. "Clearly explained" is a scored deliverable, not a README afterthought. When ambiguity is baked into the problem statement - and the announcement promises incomplete requirements - the judges cannot read your mind about which interpretation you chose. Your write-up needs to name the ambiguity, state the interpretation you picked, and justify it. An engineer who documents "the spec didn't define whether delivery is at-least-once or exactly-once; I assumed at-least-once and made the consumer idempotent, here's why" is demonstrating exactly the technical judgment being tested. An engineer who silently picks one and says nothing looks identical to an engineer who never noticed.

5. Time will go where you don't expect. With agents, generating a working first draft is the fast part. Verification, reproducibility, and documentation are where three days actually go. Budget accordingly - a working solution with no test suite and no write-up will lose to a slightly narrower solution that's fully verified and clearly reasoned.

How I'd structure the three days

Rough plan, adjust to the actual problem:

Day 1 - Specify before you generate. Read the problem twice. Write down every ambiguous term before you touch an agent, because every ambiguity is a decision you'll otherwise make accidentally. Decide what "correct" means, concretely, in a form you could test. Set up the reproducible environment (container, lockfile, one-command setup) first, not last.
Day 2 - Generate, then attack your own output. Use agents aggressively for the implementation. Then switch hats: try to break it. Inject the failure modes. Write tests that would fail if the behaviour were wrong. Every defect you find yourself is one a judge doesn't find for you.
Day 3 - Write the explanation as if the reader is sceptical. Architecture, edge cases considered and rejected, ambiguities and your resolutions, known limitations. Stating a limitation honestly reads as judgment. Hiding one reads as an oversight when someone finds it - and someone will.

The thing I'd most want to internalise: stating what you deliberately did not do, and why, is a strength. Rubrics reward demonstrated judgment. Scope honesty is judgment.

Why this format is a signal about where engineering is going

There's a real argument that this is what technical hiring looks like in a couple of years. Not "can you invert a binary tree without autocomplete," but "given agents that generate plausible code instantly, can you specify, verify, and defend a solution?" That's a senior-engineering skill set, and it's notably not the skill set that LeetCode grinding builds.

Registration and the full brief are on HackerEarth - the problem statement drops at 15:00 UTC today. I'm going in, and I'll write up what I learn regardless of how I place, including anything I get wrong in the read above.

If you're competing too, say so in the comments - I'd like to compare approaches afterwards, especially on how people handled the ambiguity-documentation piece.

I write about production debugging, performance work, and building evaluation environments for AI systems. Previous posts in this series cover deterministic RL environments for cloud infrastructure and what months of grading agentic code taught me about where models actually fail.