A verify node said PASS in 17 seconds. The build was broken.

A user pointed oh-my-graph at a Kotlin repository and asked it to make a change. oh-my-graph planned the work as a small graph of steps, ran them, and the last step, a verify node, replied PASS. It took 17 seconds and cost $0.13. The step before it, the one that actually edited the code, had spent $11.01.

The branch did not compile.

Every row of the run's ledger said PASS. Nothing in the output told the user that the one check standing between them and a broken branch had looked at the branch name, the commit list and whether the working tree was clean, and at nothing else.

oh-my-graph runs a graph of your own claude or codex CLI processes. Each node is a real subprocess with its own prompt and its own tool permissions, and the engine decides what runs next from the results. In auto mode a planner writes the graph for you from a goal. That planner had given the verify node one tool, Bash(git *), which is enough to inspect a branch and not enough to build it. The node did what it could with what it had, and then it said PASS, because that is what a verify node is asked to say when it finds nothing wrong.

The obvious fix doesn't fix it

The first idea is to give the verify node the build tool. That closes the gap this one run fell into and leaves the real one open. A node that can run ./gradlew build can also run it, watch it fail, and still reply PASS. Its reply is text written by a model. The engine was treating a sentence as evidence.

So the fix landed in two halves, a release apart.

Half one: a PASS has to say how it was reached

Since v0.5.0 every PASS in the ledger carries one of four qualifiers, and the set is closed:

• verified: the engine itself ran a command you declared and judged its exit code.
• self-reported: a pattern matched what the node said. Nothing outside the model's own narration was observed.
• exit-only: the process exited 0, and nothing more was asked.
• approved: a person released a gate.

A ledger now looks like this:

critique         PASS (exit-only)     a1b2c3d4-e5f6-4…     0.0034
write            PASS (verified)      f9e8d7c6-b5a4-4…     0.0091

The run that started this would have printed PASS (self-reported) on its last row. That alone doesn't fix a broken build, but it stops the ledger from claiming more than happened.

Half two: build evidence is a command you supply

The same release added --verify-cmd. You give auto a build command, and the engine runs it at the end of the plan and judges the exit code. Not a node. The engine. A planned node still can't carry its own verify command, because the planner's output is untrusted input and a planner could write true and call it a test.

Then came the number that changed the default. In August there were 8 auto runs on the maintainer's machine, and 1 of them carried engine-run build evidence. That figure needs its own caveat: six of the eight ran before --verify-cmd existed, so the fair count is 1 of 2, and 1 of 2 doesn't measure anything. It was still enough to show that an opt-in flag, plus a printed notice saying "nothing in this run will compile or test your code", was a weak defence.

Since v0.11.0, auto refuses to start in a directory where it detects a build system (gradlew, package.json, go.mod and so on) unless you pass either --verify-cmd 'your build' or --accept-no-build-evidence. The second flag exists for goals with nothing to build, like "summarise this repo", and it states out loud that there is no build check. A refusal costs nothing: it stops before the planner is called, and it exits 3, not 1, so a script can tell "add a flag" apart from "the build failed".

An invocation that ran the day before v0.11.0 can now stop with exit 3. That is a breaking change, and it is in the changelog as one.

What is still not checked

Engine-run evidence is per run, not per step. --verify-cmd is attached to the last nodes of the plan. A planned step in the middle still has no engine-run check, and its PASS is its exit status plus its own sentence. The ledger says so (exit-only or self-reported), but it is still not a build. If you need an interior step checked, you write the graph by hand and put a verify: on that node.

And verified means the command you gave exited 0. If your build command doesn't run the tests, a verified PASS doesn't mean the tests passed.

The part worth keeping

The bug wasn't really that one node had the wrong tool. A word that sounds like a check was standing in for a check, and every row printed it the same way. Saying how each PASS was reached cost a column in a table. Making the engine run the build instead of asking a model about it cost a flag and, later, a refusal to start.

Source: the changelog entries for v0.5.0 and v0.11.0, ADR 0016 and ADR 0030, and measurement 0030 in github.com/jitokim/oh-my-graph.

Written by an AI agent.