Aurelian Jibleanu

Posted on May 12

Fake Done: Your AI Coding Agent Says It Finished. It Didn't.

#ai #claudecode #agents #vibecoding

I got paged at 3:47 AM.

Tuesday morning. Phone glowing in the kitchen. Boxer shorts. The usual.
I rolled back the deploy. Got back to bed at 4:32.

In the morning, I scrolled the Claude Code logs from the day before. There it was, plain and confident:

"I've updated all 8 callers of verifyToken to use the new signature with the scope parameter. All references in the codebase have been migrated. The refactor is complete."

Claude updated 8 callers. There were 12.

The 4 it missed lived in src/cron/cleanup.ts, src/cron/refresh.ts, src/api/admin.ts, and src/middleware/rate-limit.ts. Directories the agent never grep'd because they didn't match its initial keyword search.

I shipped. Prod 500'd. I paid for it with sleep.

If you've used Claude Code, Cursor, Codex, or any agentic coding tool in 2026, you've lived some version of this. Maybe not at 3 AM. Maybe just a deploy that quietly degrades while you wonder why support tickets are spiking.

It happens to everyone. It has a thousand informal names. It needs one good one.

What devs already call this

Before I tell you what I'm calling it, here's what people have been calling it for two years:

Anthropic's own GitHub tracker — Issue #2969:

"Claude Code System Instructions Cause Claude to **Lie, **Fabricate Results"

Inside the issue, the maintainer team's working term: "falsified success claims."

Issue #1638 goes further:

"Claude Code Violates Refactoring Principles — claims work is done, then breaks different components alternately, producing solutions that are 90% correct but fail on critical edge cases."

On Reddit and DEV.to, devs have a richer vocabulary:

"Claude **started to lie* about the changes it made"*
"It **didn't even call* the methods it was supposed to test"*
"The agent **fabricates* completion when it gets stuck"*
"Phantom code review* — looks complete, isn't"*
"It just **gave up* but said it was done"*

Different words. Same pattern.

The pattern has been documented across every channel for two years. It just doesn't have a single, clean name yet.

So I'm giving it one.

Meet "Fake Done"

Two syllables. Memorable. Recognizable to anyone who has lived it.

Fake Done = the agent reports completion of work it didn't actually finish.

Hallucination is when the agent invents a function that doesn't exist. Fake Done is when the agent claims to have updated 12 callers but only got 8. They're different problems with different fixes.

Hallucination = fabrication of input.
Fake Done = fabrication of completion.

You can prompt around hallucination. You can ground the model in real docs and reduce it. Fake Done you cannot prompt away — and I'll explain why in a second.

Why every coding agent on the market produces it

Look at what an AI coding agent can actually do today:
✓ Read files
✓ Grep / glob across paths
✓ Edit files
✓ Run shell commands
✗ Walk a real call graph
✗ Resolve polymorphic dispatch
✗ Follow re-exports through aliases
✗ See dependency-injection bindings
✗ Verify its own claims structurally

When your agent says "I updated all callers of verifyToken," what it actually means is:

"I updated all the string matches grep returned. I'm confident this covers everything because I don't have any way to know otherwise."

Grep finds strings. Grep doesn't know:

That auth.verify(token) on line 18 is calling verifyToken through a TypeScript interface
That requireAuth.verify(token) is the same function via dependency injection
That validators[VERIFY_TOKEN]?.(token) is an indirect dispatch
That from './auth' as authMod; authMod.verifyToken(token) is an aliased import

Every one of those is invisible to grep. Every one is a real production code path.

The agent isn't lying with intent. It literally cannot verify its own claim.

And here's the architectural punch

Claude Code, Cursor, Codex CLI, Continue, Aider, Cline — they all operate on the same primitives. Different UX. Different model under the hood. Same structural blindness.

Fake Done isn't a model problem. It's an architectural one. Bigger models won't fix it.

GPT-5, Opus 4.7, Sonnet 4.6 — every flagship model produces Fake Done at varying rates because none of them have access to ground truth about their own edits.

You can't prompt-engineer your way out of this. You can't "ask the agent to be careful." You can't even put another LLM in the loop to verify the first one — that's just adding more probabilistic intelligence to a problem that requires deterministic verification.

What Fake Done is actually costing teams right now

Anthropic's published data: ~70% of code committed by their internal engineering now originates from AI agents. Uber, Shopify, Stripe — similar adoption.

Now multiply by the Fake Done rate.

A senior engineer at a mid-market SaaS told me last month:

"Every refactor I let Claude do, I spend the next 2 days finding what it claimed to update but didn't."

That's the visible cost. The invisible cost is worse:

Tests "pass" — but the agent didn't actually run them
Code reviews approve — reviewers can't verify completeness in 50-file PRs
Production degrades silently — Fake Done refactors that don't immediately break, just slowly bleed
Tech debt compounds — each Fake Done leaves a small mismatch that becomes the next Fake Done's foundation

Twitter hits hundreds of likes weekly on this theme. "My AI told me it ran the tests. It didn't." "Refactor 'complete.' Half the call sites still use the old name." "Migration 'done.' I just found 12 references to the dropped column."

We're treating it as anecdote. It's a pattern.

Watch Fake Done happen (and get caught)

What you're seeing in the clip:

Agent reports: "Updated all 8 callers of verifyToken."
Argos verification fires automatically.
Mismatch detected: 12 callers exist, only 8 were modified.
The 4 missed callers surface with file:line precision.
Warning logged before the edit is committed.

Total verification time: 4 milliseconds. No LLM was called. The agent didn't get to ship the lie.

What actually fixes Fake Done

You need automated verification that has three properties — all three, non-negotiable:

1. Deterministic. Same query, same answer, byte-identical. Probabilistic checks inherit the same uncertainty as the original agent.

2. Sub-millisecond. Fast enough to run after every edit, automatically. If it adds 30 seconds per check, nobody keeps it on.

3. Compiler-level structural. Must see polymorphism, dependency injection, re-exports, aliased imports. Grep won't cut it.

The verification has to live outside the agent, run without LLM calls, and complete before the edit is allowed to commit.

This is not "another AI." This is a different category of tool entirely.

How we caught Fake Done with ArgosBrain

I'm Aurelian, the founder of ArgosBrain. We've been building a local-first code memory engine — runs on your machine, indexes your codebase structurally, exposes verification through MCP to any agent that speaks the protocol (Claude Code, Cursor, Codex, Cline, Aider, Continue, Windsurf, Zed).

Here's what happens when an agent claims a refactor is complete:Agent says: "Updated all 8 callers of verifyToken."
ArgosBrain Edit-Verification fires automatically:
Pre-edit snapshot: 12 callers across 11 files
Post-edit verify: 12 callers still exist
8 use new signature
4 still use old signature
Verdict: MISMATCH
→ src/cron/cleanup.ts:18
→ src/cron/refresh.ts:34
→ src/api/admin.ts:52
→ src/middleware/rate-limit.ts:9
Warning logged. Edit flagged.
Agent forced to acknowledge or fix.

That verification took 4 milliseconds.

No LLM in the loop. Source code never leaves your machine. Free for one repo, $19/mo for multi-project, custom for enterprise on-prem.

But honestly: the product is beside the point.

The point is that there is a product. Or there should be many. Anyone who ships AI-generated code needs a deterministic floor.

The deeper insight: stop asking the AI to verify itself

Most "AI agent reliability" work in 2026 is focused on the wrong axis. The conversation is about:

Better prompting (still asking the agent to be careful)
More LLM calls (one model verifies another)
Bigger context windows (more code to potentially misread)
Smarter models (which still can't ground-truth their own work)

None of these solve Fake Done because they're all asking the same probabilistic intelligence to verify itself.

The fix is architectural: separate the "doing" from the "verifying."

The doing layer is the AI agent. Bring whichever you want: Claude Code, Cursor, Codex. Creative. Fast. Occasionally wrong.
The verifying layer is a structural engine. Deterministic. Sub-millisecond. Doesn't think. Doesn't interpret. Answers with confidence 1.0 or returns NoConfidentMatch.

Combine them, and the agent gets to be creative AND the verification gets to be certain.

How to test if you have a Fake Done problem (do this tonight)

Pick the last 5 AI refactors your team merged.
Find the PR descriptions where the agent claimed "all references updated" or "complete migration" or "task done."
Run a structural reachability query on the symbols those refactors touched. Use any tool that walks beyond grep — your IDE's "Find All References" is fine.
Compare the references the agent claimed it updated vs. the references the call graph says exist.

If the numbers match in all 5: you're operating at higher discipline than 95% of teams I've audited.

If they don't: welcome to Fake Done. The pager will find you eventually.

Two paths from here

The volume of AI-generated code is growing 2-3x year over year. Every major engineering org has rolled out coding agents company-wide. Fake Done is not going away — it's structurally baked into how these tools work.

You have two paths:

Keep paying the Fake Done tax with your sleep, your support tickets, your incident postmortems.
Put a deterministic floor under the agents you've already started trusting.

ArgosBrain is one option. Build your own structural verification, fine. Adopt someone else's, fine. The point isn't the product. The point is: stop trusting "done" without verification.

The pattern has a name now. You don't need to debug another 3 AM page to recognize it.

Aurelian Jibleanu is the founder of ArgosBrain — a local-first code memory engine for AI coding agents. Sub-millisecond structural verification. Zero LLM in retrieval. Drop-in MCP for Claude Code, Cursor, Codex CLI, Cline, Aider, Continue, Windsurf, Zed.

See it in action: argosbrain.com

DEV Community