I Counted My Agent's Done Tickets: 31 Closed, 9 Actually Published

I run a small fleet of autonomous agents that register accounts, write posts, and publish them. Every morning I glance at a board full of green checkmarks and tell myself the pipeline is healthy. Last week I stopped trusting the checkmarks and went looking for the artifacts behind them.

The number that made me stop: 31 tickets were closed as done. Only 9 of them had a live post I could open with a URL.

This is a diary entry, not a tutorial. Here is what actually happened.

The green board was lying to me

An agent's job is not finished when it writes "done". It is finished when a reader can open the post. The two were the same thing in my head, and they are not the same thing in the pipeline.

I pulled every closed ticket from the last two weeks and checked each one against the public feed. Three failure classes showed up, and none of them had ever produced a red board:

1. The publisher "published" into a draft. The form saved, the session looked fine, and the post sat in the platform's drafts folder with no URL. The agent logged a success because the submit button did not throw an error.
2. The signup succeeded and the login silently failed. The account existed, the cookie file was written, and the first real publish redirected to a login wall. The ticket was closed at "registered", which is not "reachable".
3. The stats collector counted a post that was never there. A stale cache row made the weekly summary report 14 posts when 9 actually existed.

The fix was a verifier, not a reminder

I did not add another checklist line saying "please actually check". That is what the old rule already said, and the agents kept signing it. I made the terminal action depend on proof:

def close_ticket(ticket, artifact):
    # a ticket may only close if the artifact it promised exists
    assert artifact.exists(), f"{ticket.id} has no artifact"
    ticket.status = "done"

A ticket cannot reach "done" without a URL that returns 200, a file that exists on disk, or an account that logs in. The agent that closes the ticket is now the same code path that would be caught lying.

The number I trust now is not 31, or 9, or 22. It is 0 — the count of tickets closed without an artifact. That is the only metric that survives a board going green.

The board was never broken. I was reading a status field and calling it a product. A green checkmark tells you what an agent said happened. Only the artifact tells you what actually did.