Flaky is a claim only declared inputs can back

Flaky-test detection is one of the features that usually lives behind
a cloud login. Nx sells it. It is also one of the easiest things to get
wrong, because "flaky" is used to mean "failed and I do not want to
look," and a tool that agrees with that use is teaching a team to
ignore red.

vx defines it narrowly, and the narrow definition is the feature.

The definition

A task is flaky when one of two things is on record:

• its exact cache key failed after it had passed, this run included (a cache hit counts as a pass; it replayed one), or
• it needed a retry this run (exec.retries or --retry) and then passed.

A failure on a key that never passed is a break. The inputs
changed and the result is red, which is what a red run usually means,
and it is not listed. Nor is the first pass after only failures: that
is a recovery (a missing tool installed), not a flake. Only tasks with a cache block are judged at
all: "same inputs, different outcome" is a claim only declared inputs
can back. A task without them is keyed on its config alone, and two
runs of it are not the same inputs in any meaningful sense.

What you see

A flaky task says so on its own row in the run's task list, in a dim
note after its name:

 ◼︎   4.21s failed  miss     app#test flaky - passed 3× before
 ⏺︎  12.84s success miss     api#e2e flaky - failed 1× before · 2 attempts

A task that was not flaky carries no note. vx info keeps the
standing list across runs, --summarize carries it as typed data
(flaky: { passes, failures, attempts } on the task, so a consumer can
tell a break from a flake without parsing text), and the MCP server
reports the same signal to an agent — getRunHistory as each task's
failure mode, getWorkspaceInfo as the standing list — so an agent
does not learn to shrug at a repeated failure on changing inputs.

Why it is free

The information was already there. Every cache entry records its key,
its outcome and its run. Asking "has this key ever had the other
outcome" is one probe of the failed-row index on a green miss, and
nothing at all on a run that executed nothing. There is no upload
because there is nothing to upload to; the history is the local
cache.db that vx why and vx last already read.

That is also why it works on the first day. A service-side detector
needs a fleet of runs before it can say anything. vx's needs the second
run of the same key on your machine.

What it is not

It is not a retry policy. exec.retries is separate and explicit, and
a task that passes on retry is reported as flaky rather than quietly
made green. It is not a quarantine: nothing is skipped, ever, because a
skip is a silent pass. It is a label on a fact, kept where you can act
on it, and the act is yours.

---

Originally published on the vx blog. vx is an MIT task runner and build cache for JS monorepos: vznjs.github.io/vx · GitHub.

_{Written with AI assistance.}