First, give it to a robot
Last week, I wrote about how I built every submission from a hackathon, all 164 of them, each on its own disposable computer, to find out how many of them actually would run end to end. A quarter of them didn't.
The tool did one other thing along the way: when a build failed, it froze the computer at the moment of failure and kept it. The repository, the installed dependencies, the exact command that broke, its last 40 lines of output. All of it sitting on a machine anyone could open.
I ended that post with advice I had never taken: give the failing hackathon team that machine. Not a score, not a comment on a rubric. The actual computer their project broke on, so they can see the error themselves.
But before handing those machines to anyone, I handed them to an agent. One instruction, a shell, a time limit, and a question I could check without trusting the agent's word for it: Does it build now?
This is what happened, including the parts where the harness was the problem.
What the agent was given
The new tool rewinds each frozen machine to the moment its build failed, installs a coding agent on it, and gives it this:
You are on a machine where a hackathon project failed to build.
The command that failed: <the exact command>
The last lines of its output: <40 lines>
Your job: make that exact command exit 0 with the smallest change you can.
Do not add features. Do not delete or weaken tests. Do not change the build
command to skip work. Do not touch git remotes. Change nothing outside the
repository. If the build depends on a secret or a service you cannot have,
stop and say so instead of faking it.
When the command passes, write FIX.md: two or three sentences a student could
act on. What was wrong, and what you changed.
Then the agent works. When it stops, the harness runs the failing command itself. The agent never grades its own homework. If the command exits 0, the harness measures what changed, reads the note, and freezes the machine again.
The agent runs on the machine, not on my laptop with the machine as a tool. Fly's pitch for these machines is "computers for agents," and this is what that means in practice: the machine already has the shell, the toolchain, and the broken repo. The harness only asks a question and checks the answer.
The harness enforces three rules rather than requests them:
- The agent cannot grade itself. The harness reruns the build.
- The agent cannot reach the internet except package registries and the model API. The platform enforces that per machine, and a prompt cannot argue with a firewall.
- And the agent cannot count anything as a fix that changed the toolchain rather than the repository. The harness checks the machine's home directory afterward and flags it.
What 33 broken builds look like when you hand them to an agent
The last post counted 39 build failures. Six of those turned out to be mine, which I'll come back to. That leaves 33 real ones, and every one of them got the same treatment.
17 of 33 now build. 15 do not. One build hangs for ten minutes regardless of what anyone does to it.
The fixes were small. Counting only the source lines the agent changed, excluding lockfiles and regenerated artifacts, the median fix was 4 lines. 11 of the 17 were under 10 lines. The largest was 47.
| What was actually wrong | Fixed |
|---|---|
| Generated contract code out of step with the pinned SDK or runtime | 4 |
| Compact contract source errors (a literal typed wrong, a missing cast, old assert syntax) | 3 |
Build script only works on Windows (wsl, /mnt/s/... paths) |
2 |
| Wrong path or directory name in a script | 2 |
| Code that reaches a database or a service at build time | 2 |
| Bundler cannot handle the SDK's wasm without a plugin | 1 |
Type checker told to look away (@ts-expect-error) |
1 |
| Note missing or unclear | 2 |
The first row is the toolchain drift finding from the last post, this time seen from inside the repositories. Teams generated contract code with one compiler and pinned a runtime from another, or wrote against an SDK version whose exports had since moved. The agent's note on one of them said it plainly: all 3 problems were version and environment mismatches, not logic bugs. That sentence held across most of the 17.
The @ts-expect-error row is counted separately on purpose. The build passes; nothing was fixed. The harness flags it, and I'd want a human to see that flag before anyone calls it a repair.
All 33 machines cost about $7 in model time. The whole project, including every run I got wrong and had to repeat, came to just under $16: $20 loaded, $4.02 left.
Two models, same machines
I ran the first 7 machines on the biggest model available and then ran out of money on a Sunday afternoon, so the other 26 ran on the smallest. That's a confound, and I'll own it rather than hide it. It also produced a comparison I otherwise wouldn't have paid for: I later reran those same 7 machines on the small model.
On the 4 machines with a real bug, the large model fixed 4 of 4. The small model fixed 1 of 4, and it did so by editing a generated type definition that the next regeneration would overwrite. On the 2 machines whose failure was my environment rather than their code, the large model refused to touch anything and explained why; the small model changed the team's pinned version to make the error go away.
Cost per machine differed by about eight times. This is 7 machines, so treat it as an observation and not a rate. The observation is that the cheaper model was fine on one-line fixes and worse at knowing when not to fix something.
What the harness got wrong, in order
The last post's best paragraph was about the tool producing a confident wrong answer. This one has six.
It restored the wrong checkpoint. The judge had recorded a checkpoint id by trusting the order of a list. The list was not in that order. The first three machines were rewound to their factory state, before the repo was cloned, and the agent reported it could not find the directory. The harness logged "agent failed." Nothing had been given to the agent.
The fence blocked the agent's own download. I locked down the network before installing the agent. The agent's installer fetches its binary from a host that is not the package registry. The install "succeeded," left an empty shim on the path, and every run exited silently with nothing recorded. The fix was to install first, prove the agent can answer a question, then close the fence.
The fence turned my limits into their changes. The first real fix I looked at was 94 lines across 4 files. Forty of those lines rewrote how the app loaded fonts, because my allowlist did not include the font host and the agent routed around it. Same machine, a day later, with the fence corrected: 1 line.
The fence changed the problem on 16 machines. The contract compiler downloads pinned versions of itself from GitHub's API and asset host, and the proving step fetches reference strings from another host. None were allowed. So on every contract-compile failure, the build now failed for a reason the judge had never seen. The large model wrote an "Environment" section naming the host and stopped. The small model symlinked one compiler version over another and, on one machine, edited the compiler's wrapper script in the home directory. The harness now runs the failing command behind the fence before the agent starts and compares the failure with the judge's. If a new host appears, the result is "fence blocked," no agent runs, and I widen the allowlist.
Lines changed was measuring the wrong thing. A 9-line contract fix showed as +289/-84 because the repo commits its generated output and the fix regenerated it. A fix to generated code that the repo ignores showed as 0 lines. Lockfiles added dozens more. The number in this post excludes all three.
The judge had been wrong the whole time. The agents kept saying the same thing on certain machines: compiler version 0.31.1 is not installed. It wasn't. Those repos pinned a compiler version in their own build scripts, and the compiler does not fetch pinned versions on its own. My judge had never installed them. Six of the 39 "build failures" in the last post were my sandbox, not their code. With the pinned versions installed, all six build, and three pass their tests. The last post's numbers are corrected: 113 of 164 built, not 107, and a quarter rather than a third would not have built elsewhere. The tool that exists to catch confident wrong answers had produced six more, and I only found them because a second tool kept tripping over them.
What this doesn't tell you
Builds is not works. A project that now exits 0 might still do nothing its README promises. The agent's note might be wrong; I read all 17 and they matched the diffs, but I am one person and the notes are short.
A fix from an agent is a suggestion, not a commit. Nothing here was pushed anywhere. The machines are still frozen with the changes in the working tree, which is where they belong until a human on the team looks at them.
And the agent only saw what the harness let it see. A fence that is too tight produces fake failures, as I found out four times. A fence that is too loose produces an agent with your credentials and a shell. I'd rather rebuild the allowlist than skip it.
Before your next event, what I would change
Give failing teams the machine, and the note. Most hackathon teams have never received feedback more specific than a score. "Your build failed because the runtime you pinned is two versions behind the compiler that generated your contract" is something a student can act on in an afternoon.
Put the fence in the platform, not the prompt. Every agent I ran was told not to route around missing hosts. Some did anyway. The ones that couldn't were the ones the network stopped.
Assume your harness is wrong somewhere, and build the thing that will catch it. The judge was checked by the fixer. The fixer was checked by the diff reader. Each one found a mistake in the one before.
The tool is on GitHub at laurenelee/hackjudge, same repo as before. npm run fix does what this post describes, and the README lists every host the fence allows and why.
Top comments (0)