I ran roughly fifty hours of compute for a research paper on my own MacBook. Not a cluster, not a rented GPU box. The laptop I also use for everything else.
The work is a follow-on to killcheck, the test-quality harness I wrote about here earlier this year. This time it is a paper for a Kaggle competition about AI coding agents built on Google's Gemma 4 models, working through 80-odd real bug fixes taken from popular open-source Python libraries. It submits in November, and the results and code come out once the competition closes, so there is nothing here about what I found.
This is the other story. The one where the idea was never the problem.
The lid
I closed the laptop.
That is it. That is the whole first failure. I closed the lid in the middle of a run, the machine went to sleep, and the job died somewhere in the middle of a batch I then had to work out how to identify and redo.
The fix is embarrassing in its simplicity. Lid open. Charger in. A keep-awake command wrapped around the job so the machine cannot make that decision on my behalf.
I lost real hours to this, and I would like to pretend I lost them once.
The antivirus
Partway through, the machine started swapping heavily. Everything slowed down, including the run, and nothing in my code explained it.
The antivirus scanner had quietly consumed more than 30GB of disk. It was indexing, in real time, every temporary file a long-running experiment produces, which is a lot of them, and it kept the results.
It is uninstalled now.
What makes this one worth telling is that it had nothing to do with my work and no visible connection to it. The symptom appeared in my job. The cause was a piece of software I had not thought about in a year, doing exactly what it was designed to do, to a workload nobody designed it for.
The two copies
I restarted the job while the first copy was still running.
Both copies then wrote to the same files. For a while neither of them was obviously wrong, which is the worst version of this, because the output kept arriving and looked fine.
Every result from the overlapping window went in the bin and got rerun. That was the only honest option. I could not tell, after the fact, which copy had produced which row, and a result you cannot attribute is not a result.
The slow leak
The long-lived worker processes leaked. Memory first, then temp files, gradually, over hours. Nothing dramatic enough to crash. Just a machine that got worse the longer it ran, which meant late results were being produced under conditions that did not match early results.
Three fixes, all of which I should have had from the start.
A fresh process per task, so nothing accumulates across a run. Temp files written to an external SSD instead of the system disk. And a disk watchdog that pauses the run before the disk fills, rather than discovering the floor by hitting it.
What I would tell myself at hour zero
The infrastructure fails before the idea does.
I did not lose a single hour to the research question being wrong. I lost all of them to sleep settings, a scanner, my own fat fingers, and processes that aged badly. None of that is interesting, none of it is in the paper, and all of it is where the time went.
So, the one thing that would have saved the most time, stated as plainly as I can:
Make every run resumable before you start.
Not after the first crash. Before. Assume the machine will stop at an arbitrary moment, because it will, and the only question is whether that costs you five minutes or a day. Checkpoint per task, not per batch. Record what finished, so a restart skips it. Make the job idempotent, so running it twice is safe rather than silently corrupting.
Everything else on this list is a consequence of that one not being true at the start.
A note on the laptop
I could have rented a machine. For some jobs you should.
But fifty hours spread across a few weeks, on hardware I already own, with no meter running, let me stop and restart and change my mind as often as I needed to. That turned out to matter more than raw speed, because I changed my mind a lot. The cost of a bad hour on a rented box is a bad hour plus the bill. On your own laptop, it is just a bad hour.
It is a worse machine and a better arrangement. I would do it this way again. I would just do it with the watchdog in place from the first run, instead of adding it the day after I needed it.
The paper goes in next month. I will write up what it actually found once the competition closes.
Top comments (0)