Ran into this twice on data pipeline projects - the agent said everything was done because the files were there, and we only found the content was wrong on the third review. Your point about separating who defines done from who does the work is what actually mattered to us. We tried keeping the same session going across all the steps to save time but it got unreliable around step four or five. Starting fresh each pass was slower but we could actually trust what came out.
Yes Tae, that example is false completion in its cleanest form. Your observation on unreliable behavior around steps 4-5 also matches what I’ve seen: accumulated context initially saves time, but then carries stale assumptions forward and causes rapid deterioration. Fresh instances cost more, but a compact and explicit handoff doc makes their output way more trustworthy.
Hi Tae, your example of output files being present while their contents were wrong is exactly the distinction I’m trying to understand.
What check finally exposed the problem on the third review, and is that check now automated? I’m also curious whether starting fresh sessions resolved the issue or mainly made the remaining problems easier to catch.
I’m exploring developer tooling for checking agent completion. A short exchange here would be valuable; there’s no need to share private code or data.
Solution architect, 16 years in software, last 3 on AI automation. I write about making automated work verifiable: evaluation harnesses, audit trails, and results published even when the answer is no.
Rule 2 has a gap with the same shape as the failure the whole process exists to catch, one level up. A locked test the implementation cannot satisfy and a locked test that is wrong produce the identical signal: the test fails and the run stops. The escalation carries nothing that separates them, so the "separate decision" gate is exactly as strong as whoever reads it, and an agent that cannot reach the bar has the same incentive to claim a bad oracle that it had to claim completion. Routing failures back to planning, as the thread suggests, does not close that either, because the router is the one who cannot tell.
The fix looks like it already lives in rule 1 rather than needing an eighth rule. If the product book has to name, per acceptance condition, one construction that must pass it, then "the test is wrong" becomes checkable: run the named construction against the test. It failing is evidence about the oracle. It passing means the implementation is the thing that missed, and the run does not get to restart from the specification boundary.
Good catch, Vinh. I agree that Rule 2 alone cannot distinguish a failing implementation from a broken oracle. Each critical test should first be checked against something we already know should pass. That lets us verify the test itself before development begins. I would not require a complete working example for every end-to-end requirement, since that could mean building the feature twice. But the basic principle belongs in Rule 1: important tests must be validated independently.
Solution architect, 16 years in software, last 3 on AI automation. I write about making automated work verifiable: evaluation harnesses, audit trails, and results published even when the answer is no.
A known-pass construction only rules out one of the two ways an oracle can be wrong. A test that is too strict fails the construction and gets caught. A test that is too loose passes it, because a test that passes everything passes that too, and too-loose is the false completion case itself: the locked criteria agree with the agent that the work is done. So the thing worth attaching per critical test is a pair, one construction that must pass and one that must fail, and the must-fail half is usually the cheaper of the two to write.
On the cost, the must-pass half is not the feature. An acceptance test asserts over an artifact - a response body, a file, a resulting state - so a hand-written artifact validates the oracle with no implementation behind it. It only approaches building the thing twice where the condition is over a process rather than an artifact, and that set is small enough to name in the product book rather than exempt by default.
You’re right, Vinh. A test that's too loose is exactly what enables false completion. Pairing a 'positive' case with a 'negative' case gives the oracle both a positive and a negative control.
And your distinction between checking an artifact and checking a process also resolves most of my cost concern. For artifact-level checks, a hand written response, file, or state is enough; it does not require implementing the feature twice.
The refined rule becomes: "each critical acceptance condition in the product book should include both a must-pass and a must-fail case" or so.
Building AI systems and fixing AI-generated apps before they collapse in production.
Working with agentic AI, RAG pipelines, AI orchestration, and startup MVP scaling at IT Path Solutions.
The “done” state being outside the agent is probably the most important architectural point here. I’d frame the acceptance layer as an independent control plane: the builder can produce code and evidence, but it should never control the definition of success. One nuance worth adding is that hidden tests alone don’t solve false completion the oracle itself can be wrong or incomplete. The strongest setup is locked acceptance criteria + independent evaluation + deterministic checks, with failures routing back to planning rather than letting the coding loop redefine the target.
Indeed, Mateo. 'Independent control plane' is a good name for what I was trying to describe here. Hidden tests can also be wrong. If that happens, someone outside the coding loop i.e. human, should review the tests; and in any case the builder should not be allowed to change the definition of success.
Your article on false completion addresses a distinction I’m investigating: an agent can produce plausible work without satisfying the original requirements.
I’m exploring developer tooling in this area and would value your experience with the acceptance process you described. Which parts are enforced by software today, and which still depend on someone manually checking that the rules were followed?
Would you be open to a 20-minute conversation? I’m especially interested in where additional checks reduce work - and where they just create more process.
Hi Devin, just saw this. Sorry for late reply and thanks for the careful read.
In the harness I use, the mechanical parts are enforced by software: acceptance tests are locked and hashed before coding starts, the coder can't read the final suite, each step gets a fresh coder instance, and the coder/reviewer loop has a hard limit.
What still needs human judgment is whether the spec and tests describe the right thing. Vinh's suggestion above helps: give each critical test a must-pass and a must-fail case. But even that doesn't settle whether we're testing what the user actually needs. Note that this point is not even AI related, but rather correct engineering- tests and checks built around a mistaken spec can all pass but with zero effectiveness.
For me, extra checks are useful when they test something we haven't actually established yet. Checking file contents against expected results, for example, can catch the kind of problem Tae described. Another reviewer going over the same evidence without checking anything new is harder to justify.
Happy to have a 20-minute chat. What are you building? The easiest way to reach me is on LinkedIn: search giladhaimov
The scary one isn't a red test. It's the agent saying done while the thing a person would actually use is still missing.
I lock what done looks like before it starts writing, the click or flow a person hits. Then a separate go to build. The session that wrote it doesn't get to say it's finished.
Spot on, Ezra. That is the core of the 'False Completion' trap. Locking the definition of 'done' before any code is written and ensuring the builder never grades its own work are exactly how we stop an agent from moving the finish line. And as you said, if that pre-locked path is still wrong, it simply ain't done yet.
We're a place where coders share, stay up-to-date and grow their careers.
Ran into this twice on data pipeline projects - the agent said everything was done because the files were there, and we only found the content was wrong on the third review. Your point about separating who defines done from who does the work is what actually mattered to us. We tried keeping the same session going across all the steps to save time but it got unreliable around step four or five. Starting fresh each pass was slower but we could actually trust what came out.
Yes Tae, that example is false completion in its cleanest form. Your observation on unreliable behavior around steps 4-5 also matches what I’ve seen: accumulated context initially saves time, but then carries stale assumptions forward and causes rapid deterioration. Fresh instances cost more, but a compact and explicit handoff doc makes their output way more trustworthy.
Hi Tae, your example of output files being present while their contents were wrong is exactly the distinction I’m trying to understand.
What check finally exposed the problem on the third review, and is that check now automated? I’m also curious whether starting fresh sessions resolved the issue or mainly made the remaining problems easier to catch.
I’m exploring developer tooling for checking agent completion. A short exchange here would be valuable; there’s no need to share private code or data.
Rule 2 has a gap with the same shape as the failure the whole process exists to catch, one level up. A locked test the implementation cannot satisfy and a locked test that is wrong produce the identical signal: the test fails and the run stops. The escalation carries nothing that separates them, so the "separate decision" gate is exactly as strong as whoever reads it, and an agent that cannot reach the bar has the same incentive to claim a bad oracle that it had to claim completion. Routing failures back to planning, as the thread suggests, does not close that either, because the router is the one who cannot tell.
The fix looks like it already lives in rule 1 rather than needing an eighth rule. If the product book has to name, per acceptance condition, one construction that must pass it, then "the test is wrong" becomes checkable: run the named construction against the test. It failing is evidence about the oracle. It passing means the implementation is the thing that missed, and the run does not get to restart from the specification boundary.
Good catch, Vinh. I agree that Rule 2 alone cannot distinguish a failing implementation from a broken oracle. Each critical test should first be checked against something we already know should pass. That lets us verify the test itself before development begins. I would not require a complete working example for every end-to-end requirement, since that could mean building the feature twice. But the basic principle belongs in Rule 1: important tests must be validated independently.
A known-pass construction only rules out one of the two ways an oracle can be wrong. A test that is too strict fails the construction and gets caught. A test that is too loose passes it, because a test that passes everything passes that too, and too-loose is the false completion case itself: the locked criteria agree with the agent that the work is done. So the thing worth attaching per critical test is a pair, one construction that must pass and one that must fail, and the must-fail half is usually the cheaper of the two to write.
On the cost, the must-pass half is not the feature. An acceptance test asserts over an artifact - a response body, a file, a resulting state - so a hand-written artifact validates the oracle with no implementation behind it. It only approaches building the thing twice where the condition is over a process rather than an artifact, and that set is small enough to name in the product book rather than exempt by default.
You’re right, Vinh. A test that's too loose is exactly what enables false completion. Pairing a 'positive' case with a 'negative' case gives the oracle both a positive and a negative control.
And your distinction between checking an artifact and checking a process also resolves most of my cost concern. For artifact-level checks, a hand written response, file, or state is enough; it does not require implementing the feature twice.
The refined rule becomes: "each critical acceptance condition in the product book should include both a must-pass and a must-fail case" or so.
The “done” state being outside the agent is probably the most important architectural point here. I’d frame the acceptance layer as an independent control plane: the builder can produce code and evidence, but it should never control the definition of success. One nuance worth adding is that hidden tests alone don’t solve false completion the oracle itself can be wrong or incomplete. The strongest setup is locked acceptance criteria + independent evaluation + deterministic checks, with failures routing back to planning rather than letting the coding loop redefine the target.
Indeed, Mateo. 'Independent control plane' is a good name for what I was trying to describe here. Hidden tests can also be wrong. If that happens, someone outside the coding loop i.e. human, should review the tests; and in any case the builder should not be allowed to change the definition of success.
Hi Gilad,
Your article on false completion addresses a distinction I’m investigating: an agent can produce plausible work without satisfying the original requirements.
I’m exploring developer tooling in this area and would value your experience with the acceptance process you described. Which parts are enforced by software today, and which still depend on someone manually checking that the rules were followed?
Would you be open to a 20-minute conversation? I’m especially interested in where additional checks reduce work - and where they just create more process.
Hi Devin, just saw this. Sorry for late reply and thanks for the careful read.
In the harness I use, the mechanical parts are enforced by software: acceptance tests are locked and hashed before coding starts, the coder can't read the final suite, each step gets a fresh coder instance, and the coder/reviewer loop has a hard limit.
What still needs human judgment is whether the spec and tests describe the right thing. Vinh's suggestion above helps: give each critical test a must-pass and a must-fail case. But even that doesn't settle whether we're testing what the user actually needs. Note that this point is not even AI related, but rather correct engineering- tests and checks built around a mistaken spec can all pass but with zero effectiveness.
For me, extra checks are useful when they test something we haven't actually established yet. Checking file contents against expected results, for example, can catch the kind of problem Tae described. Another reviewer going over the same evidence without checking anything new is harder to justify.
Happy to have a 20-minute chat. What are you building? The easiest way to reach me is on LinkedIn: search giladhaimov
The scary one isn't a red test. It's the agent saying done while the thing a person would actually use is still missing.
I lock what done looks like before it starts writing, the click or flow a person hits. Then a separate go to build. The session that wrote it doesn't get to say it's finished.
If that path is still wrong, it isn't done.
Spot on, Ezra. That is the core of the 'False Completion' trap. Locking the definition of 'done' before any code is written and ensuring the builder never grades its own work are exactly how we stop an agent from moving the finish line. And as you said, if that pre-locked path is still wrong, it simply ain't done yet.