The enum value that had never been written

I found 27 workflow branches that were being skipped while every run still finished as COMPLETED.

The condition on those branches could never match. So the step was skipped, the run was marked success, and nothing ever told me. They had been running like that for weeks.

That is the part worth sitting with. Not that there was a bug — there is always a bug. That the system reported success 27 times a day, honestly, while doing nothing.

The question that made it worse

I wrote about this on a forum and someone asked a question I could not answer:

Does it distinguish "green but semantically idle" from "green and actually processed", or is that still something you catch by comparing runs?

I went to check, expecting to say the data is one level down, in the step rows. Each step has its own status, and SKIPPED is one of the values. So the information had to be there.

It was not there. StepExecutionStatus.SKIPPED had existed in the enum since the beginning of the project and had never once been written. Zero occurrences in the codebase.

A skipped step did not create a database row at all. The loop appended a dict to an in-memory results list and continued. The only trace of a skip lived inside a JSON blob on the execution row.

So:

• the step view reads rows, which meant a skipped step was invisible in the UI too
• "which runs skipped something?" had no answer short of parsing JSON
• a run that skipped every branch and a run that did all the work were identical at every level a human or an alert would look

I had built an enum value to describe a thing, and then never recorded the thing.

The fix, and the flaw in the fix

The fix took an afternoon. A skipped step now writes a real row with the reason. The run reports how many steps it processed, skipped and failed, derived from those rows so the numbers cannot drift from what the step view shows.

Then I added a badge: when a finished run processed nothing, say so.

The same reviewer killed it within the hour, and he was right:

A zero can be legitimate. Quiet day, nothing to send, COMPLETED with 0 processed is correct. The alert gets teeth when the count is two-sided: the run says what it processed, the source says what it handed over, and the two have to tie out.

A scheduled workflow with nothing to do processes nothing every quiet day. My badge would have fired on all of them, been muted inside a week, and then not been there on the day it mattered.

That is worse than showing nothing, because it looks like coverage.

The badge came out. The counts stayed, as plain facts, and the judgement went with the badge.

The same shape, one level up

Two days later I built a deploy script that runs the tests locally and refuses to deploy if any are red. It ends by verifying the deploy.

It printed five green lines:

✓ deployed
✓ server on c15b730
✓ all containers running
✓ agent-mesh.org/health healthy

Every one of them true. The feature I had just deployed was dead. The containers had started before I added the environment variable it needed, so the code shipped and did nothing.

I found it by curling the endpoint and reading the value, which the script had not done.

A deploy verifying that it deployed is not the same as verifying the thing works. Health checks are generic. Effect checks are specific, and specific is the part nobody writes.

What three other people added

I posted the 27-branch story and the thread turned into something better than the article I meant to write.

On maintenance. Per-step assertions do not scale past a handful of workflows, because every assert is another thing to maintain. The answer that survives is to move the assert out of the workflow and into the engine. One place, and every workflow gets it without anyone adding a node.

On the zero. History is a partial answer to the legitimate-zero problem. Do not look at one execution; look at the workflow's own baseline. What does Tuesday 9am normally produce, against Saturday? Zero on a normally-busy slot is a signal. Zero on a normally-quiet one is not. It is probabilistic rather than certain, but it turns blind into suspicious, which is usually enough to know where to look.

The limit, and it is mine: a new workflow has no history, which is exactly the period when someone is most likely to have built the condition wrong. My 27 branches were dead from the first run. There was never a healthy baseline to deviate from.

On cascades. Someone described a self-hosted setup where a request to a local model dropped without a loud error. The node completed, passed an empty payload downstream, and every subsequent node executed successfully against nothing.

That is worse than my case. Mine were 27 independently dead branches. That is one silent failure manufacturing more of them, each of which succeeded honestly, because each did do its job on the nothing it was handed.

It is also why recording what a step received matters as much as what it returned. A step that returns nothing is suspicious. A step that received nothing tells you where the rot started, and those are usually different steps.

And the one I have no answer for. A dedup node with a logic edge case swallowed an entire dataset. The node was correct. The code was correct for the cases it was written for. Nothing in the run is wrong except the number of rows.

Output shape validation does not catch that, because [] and a thousand rows have the same shape.

The pattern

Every one of these is the same thing wearing different clothes: a check that cannot report failure.

A run status that only knows whether an exception was thrown. A test that performs actions and never asserts. A health check that confirms a process is listening. A skipped step with nowhere to be recorded.

There is a smaller version of this I hit in the same codebase. I stored cost as an integer number of cents:

cost_cents = int(
    (prompt_tokens * 0.5 / 1000) + (completion_tokens * 1.5 / 1000)
)

A typical run costs about 0.0955 cents. int(0.0955) is 0. Almost every run I had cost under a cent, so almost every run recorded as exactly zero, and my own analytics page reported 2 cents total across about a hundred executions. I believed it for a while. That is not rounding drift. That is the data being destroyed at write time, by a cast.

What is still broken

The two-sided count. My runs say what they processed; nothing says what the source handed over. The HTTP tool already computes how many records a list endpoint returned and throws it away, because tool results are not persisted per step. Until that changes, a run that processed zero looks the same whether the day was quiet or the condition was broken.

The baseline idea is in the backlog and the data for it already exists. Both are worth doing, and they catch different bugs: history finds drift, source counts find born-broken.

I build a hosted agent platform, which is where all of this happened, so treat the whole thing as biased. But the bug was not exotic and neither was the fix. The enum value was right there the whole time. Nobody had ever written it.