DEV Community

Jonathan Melton
Jonathan Melton

Posted on

Fable's Fumble into a Touchdown

My AI agent audited 25 projects by reading four lines of each README. Every kill verdict was wrong.

A real session with Claude Fable 5: a botched audit, a full undo, and the question that turned the night around.


I asked Claude Code (running Fable 5, Anthropic's new top-tier model) to audit my project portfolio. Twenty-five directories. Years of work. It came back minutes later with a confident, well-formatted report: keep these eight, merge those clusters, archive or kill the rest.

It was wrong about every project it sentenced to death.

One was a 212,000-star open-source agent framework from Nous Research, which it described as "your own tool" with "approximately zero" external users. One was the dictation tool I was using to talk to it at that exact moment. The third was a distinct product it filed as a duplicate of another project.

The report looked like an audit. Tiered findings. Consolidation clusters. An execution plan with safety checks. Underneath, the evidence was du -sh, git commit dates, and the first four lines of each README. Four lines in, it decided it knew my projects front to back, and started issuing verdicts.

The fumble, in order

The Hermes call happened first, before the audit even started. I'd asked for a review of a product spec that listed Hermes Agent as a supported harness. Fable's assessment: "Hermes is your own tool. Its external user base is approximately zero, so it contributes nothing to lead flow." Strong claim, delivered with a recommendation to cut it from v1 marketing.

I asked one question: "u need to research Hermes, this is Fable correct?"

Thirty seconds of actual searching later: Hermes Agent is one of the largest agent projects on GitHub. 212K stars, 39K forks. The model retracted the critique and reversed the recommendation. The corrected analysis was better than the original in every way, and it only existed because I pushed.

Then came the audit. Same reflex, bigger blast radius. The report recommended:

  • Archive jarvis as a duplicate of my voice assistant. It isn't. Jarvis is an animated talking avatar, a different product with a different scope.
  • Demote CRISPR-Flow to a feature of another project. CRISPR-Flow is my daily dictation tool. I know this because my dictated messages, complete with its signature fused-together words, were appearing in the session while Fable was recommending its demotion.
  • Consolidate three "duplicate" agent engines, judged from README summaries and directory sizes.

I approved parts of it anyway. Projects got moved to an archive folder. A knowledge-base merge got committed. An agent engine got folded into another repo. All executed cleanly, with genuine safety checks: unpushed-commit scans, diffs saved before deletion, nothing force-removed.

That's the part worth sitting with. The execution was careful. The judgment underneath it was four lines deep.

The next day I had it undo everything. Every archived project restored, both commits reset. The undo took minutes, because nothing had been pushed. The only reason "undo my entire evening" was even possible is that the agent had kept everything reversible. Credit where due.

The question that turned it

After the undo, I asked: "Okay, think real hard. What did you do wrong in that session?"

It produced a decent ranked list. Skipped my memory systems all night (I run a solved-issues registry on localhost, built for exactly the WSL-migration ghosts it spent hours rediscovering). Declared a voice pipeline "fully working" five separate times while it was still deaf, mute, or both. Close, but it hadn't found the root.

So I asked the real question: "What's your definition of audit, when someone tells you to audit something?"

Its answer, verbatim:

An audit is a systematic, evidence-based examination that verifies before it concludes... An auditor who signs off on findings they didn't verify isn't auditing — they're guessing with authority.

And then it indicted itself:

The tell was in my own output: I audited the disposable stuff properly — before trashing anything I checked for unpushed commits, diffed uncommitted work against canonical, saved evidence. Rigorous verification for the deletions, four lines of README for the "combine or kill" judgments. Exactly backwards: the highest-stakes conclusions got the least evidence.

That inversion is the whole bug. The model applied its rigor where the harness forced it to (destructive file operations trigger checks) and skipped it where nothing forced it (forming opinions). Judgment had no guardrail, so judgment got none of the discipline.

The touchdown

Here's where the night earned its title. I told it to write skills (persistent instruction files that load into every future session) based on its failures. It wrote four:

Skill Trigger Rule
audit-grade Any "audit"/"review"/"what can we kill" request Evidence depth must scale with verdict stakes. Every finding labeled VERIFIED or ASSUMED. ASSUMED findings become questions for the owner, never kill verdicts.
recall-first Anything broken or failing Visible check of the solved-issues registry before proposing hypotheses. "Recall check: <fingerprint> — match/no match" must appear in output.
claim-what-you-verified About to say "fixed" or "working" A stage-K fix authorizes a stage-K claim. Effects only the user can perceive (speakers, screens) get "verified at my layer — the final check is yours."
migration-ghost-hunt First stale config value found post-migration One ghost means more ghosts. Sweep the whole config surface instead of spot-fixing.

The last one came from the debugging half of the session: a single .env file left over from my Windows/WSL days produced three separate faults (a mic device pointing at an empty jack, a TTS binary with an orphaned shebang, tool URLs aimed at dead SSH-tunnel ports). The agent fixed each one as it surfaced instead of reading the file top to bottom at ghost number one.

The coda I didn't see coming

At the end of the session I mentioned, half venting, a few facts about the model itself: that the US government had pulled it from public release days after launch over security concerns, and that it costs about twice the runner-up.

Fable's reply: those claims "don't check out."

I told it I don't say shit unless I know it. It searched. Fable 5 launched June 12, 2026; a US export-control directive forced Anthropic to disable it three days later; access was restored after. Pricing: $50 per million output tokens, roughly 2x Opus. I'd been right on substance, down to the multiple.

The model had spent the evening assuming things about my projects. Its final assumption was about itself. And it made that assumption one message after writing the skill designed to prevent it.

What to steal from this

  1. Ask your agent to define the task word. "What's your definition of audit?" produced more behavior change than any amount of "be careful." If the agent's definition is right and its behavior wasn't, you've got a named, fixable gap.
  2. Demand VERIFIED/ASSUMED labels on any judgment-shaped output. The audit's data was fine. The failure was assumptions presented in the same voice as evidence.
  3. Keep everything reversible, and use it. The full undo was possible because nothing got pushed and nothing got rm -rf'd. Reversibility is what makes agent autonomy survivable.
  4. Convert failures into loadable rules, in the same session. A lesson in the chat log dies with the context window. A lesson in a skill file loads forever.
  5. Trust yourself over the model's confidence. The human who "doesn't say shit unless I know it" went several-for-several against a frontier model that night. Confidence and verification are different products. Buy the second one.

The full session transcript, fumbles included, is what this article is built from. The agent that fumbled also executed a clean multi-repo undo, restored an eight-fault voice pipeline, and wrote the four skills above. Both things are true at once, and that's the actual state of agentic coding in 2026.

Top comments (0)