The Second Run Is Where the Lies Show Up

Every agent demo I have ever seen works on the first run. Mine did too. The first run has a clean state, a fresh context, and a demo-friendly task that was probably chosen after a few attempts.

The second run is different, and it is the only run I trust.

What changes between run one and run two

• The state is no longer clean. Files were created, ports were opened, a lock file exists. Half the failures I logged live in this gap.
• The context is no longer empty. Yesterday's task file may still be around. A note in the notes file may contradict what I am asking today.
• The task is no longer chosen for the demo. If it works twice on a task I did not specially pick, it probably works.

There is a fourth difference that took me longer to see: on the second run, nobody is watching. On the first run I am looking at the screen, ready to intervene. On the second I am doing something else. That is the real test, and it is also when I find out how good the failure messages are.

The rules I use now

• A task is not automated until it has run twice, unattended, on a dirty state.
• I keep the state dirty on purpose for the second run. Cleaning up before testing cleanup tests nothing.
• If the second run fails, the bug is almost always in what the first run left behind, not in the task.
• I read the log of the second run, not the first. The first log is a story, the second log is evidence.

Why this matters for how I judge tools

When I look at a tool now, the question is not "does the demo work". It is "what happens when I run it again tomorrow, with the mess from today still on disk". That question filters out a surprising amount, including things I would otherwise have installed.

The same standard applies to my own work, which is the uncomfortable part. Half the things I call automated have never had a second unattended run. I know which ones they are, and I keep not fixing them.

If you test this way, what does your second run look like? I would especially like to hear from anyone whose second run is genuinely clean, because mine never is.