My coding agent kept telling me "done." The build was broken. The tests it claimed to pass didn't exist. I started calling these hollow completes: the agent reports success, but nothing was actually verified.
The problem isn't the model. It's that "done" was never defined as something checkable. So I defined it.
What I built
FlowSkill is a small open-source harness that sits between your coding agent and the word "done." Two kinds of gates:
-
Mechanical gates (
flow.sh) — does it build? Do the tests pass? Does the linter stay quiet? Things a script can check without judgment. -
Semantic gates (
SKILL.md) — does the change actually do what was asked? This is the harder one, and it's where most harnesses give up and let the agent grade its own homework.
The key rule: a kill at a gate is a valid outcome. If the work doesn't pass, it doesn't ship — no partial credit, no "mostly done."
Why existing tools didn't solve it
I looked at 21 agent harnesses. Every single one let the agent evaluate its own work. That's like letting students write their own report cards. The failure mode is structural, not a prompt engineering problem.
Benchmarks have the same disease from the other direction: agents score 90+ on examples they've memorized, then collapse to single digits on unseen tests. Memorized answers, not capability.
How it works in practice
npx @manhquy/flow-skill@latest
You describe the task. The agent works. Before anything is called "done," it runs /flow check: mechanical gates first (fast, deterministic), then semantic review against the task definition. If a gate fails, the attempt dies there and the failure gets logged to DEBT.md — a record of what didn't work, so you stop paying for the same dead ideas twice.
There's also /flow doctor (12 environment checks — the machine checks the machine before the model judges anything).
Honest limitations
- The semantic gate is only as good as the task definition you write. Vague task in, vague verdict out.
- It adds minutes to each run. If you're prototyping something throwaway, the gates are overhead you don't need.
- 9 stars on GitHub. This is early, and the edges are rough.
Why I'm writing this
Not to sell you a tool. To name the problem: every AI-written line is a loan against future debugging time. Someone — you, your teammate, future-you at 2 AM — pays it back with interest. Gates are just a way to pay some of it upfront, when it's cheap.
Repo: github.com/manhquydev/flow-skill · MIT licensed. If you've been burned by hollow completes, I'd like to hear how you verify your agents' work.
This is the first in a series on gated build harnesses. Next: "21 harnesses, one hole." For release notes and longer essays, subscribe to the newsletter (link in profile).
Top comments (0)