I've lived through the progression of AI from the start: copying code challenges into ChatGPT, then Gemini, then Claude, and now everything agentic, with its harnesses and its tools.
AI is getting more and more independent. For my personal projects it works well. Starting something from scratch or putting out a new website, inside a small codebase without much shared domain, isn't bad. The problem shows up in day-to-day work: the ticket, the task, the fix, the bug.
What happened to me
Since Opus 5.5 came out, my impression is that it does worse at that work. I'm not saying it as a fact: it's what I've seen, in my workflow.
It gets too clever when creating tasks and sets aside the harness I had built. Before, with TDD, I would check whether the tests were written and what they implied, and then we would look for the bug. That harness was going quite well for me until recently. In the last ten bugs I've shipped, I've had to guide it step by step myself, because the solutions it gave me weren't correct.
It's not that I would have done it better or worse. It's that I had the knowledge to know that wasn't what had to be done.
There's a second thing I don't fully understand: it finds more and more things risky. Querying some logs or running a query against a database that isn't production, something it used to do without any problem, now seems dangerous to it. For now I do it by hand and that's it.
And a third, deeper one: they sell you that AI can do everything, that it fixes everything and ships everything. In my work, no.
The answer: a hard harness
My harness is no longer a soft one, made of suggestions. It's a hard one, where the AI has to follow strict steps and can't step outside them.
I've published it as rstack. They are Claude Code skills: according to the documentation, a skill is a SKILL.md file with instructions that Claude loads when needed, or that you invoke yourself. My skills aren't advice, they are sentences of the kind "don't do X until Y". They were born in the same spirit as gstack, with one design difference: each skill stands on its own, with no shared document that all of them point to.
The one this post is about is rstack-investigate, the one for bugs. The repository also has others for planning features, reviewing PRs and doing QA in the browser.
The rule that isn't negotiable
The skill starts with a rule: no root cause is proposed and no fix is written without having reproduced the bug with real data. Reading the code and reasoning about what should happen is not proof.
It came from something that happened to me twice in the same session with the AI: it proposed a cause without checking the account's real data, and then it built an entire fix by reading code, which was useless in production because the data it depended on didn't exist. Both times I had to stop it myself.
The seven phases
The cycle has seven phases, and at the end of each one the skill stops, tells me what it found and asks me before continuing. It doesn't run on its own from start to finish.
The first is reproduce. It reads the symptom as it arrives, without reinterpreting it, traces the code backwards and looks at the git history in case it's a regression. Then it reproduces the bug with real data, as an automated test inside the repo itself, and watches it fail. It doesn't first write a loose script "to understand": if the project has tests, the reproduction is that test. If a piece of data is missing, it asks for it, it doesn't make it up.
The second compares with bugs already fixed and rules out the typical patterns: a race condition, a null that propagates, a half-finished state, a failing external integration, configuration that differs between environments, a stale cache. A bug that comes back to the same place is a sign of architecture, not a coincidence.
The third tests the hypothesis. It adds temporary logs right at the suspected point, runs the same test and reads the output line by line. If three hypotheses fail in a row, it stops and asks me, because it may be an architecture problem and not a wrong hypothesis. Before fixing anything, it removes all the logs.
The fourth is the fix. The fifth reproduces the original bug again and delivers a report. The sixth checks what looks like the bug, without dismissing it or covering it by ear: if it doesn't apply, it says why.
The seventh is the audit. A clean subagent, with no context at all, reads all the project's ADRs (architecture decision records) and all my principles documents, the knowledge/ folder, and checks the solution against each one. Not against the ones that sound relevant: against all of them, with an N-of-N count so it's visible that none was skipped.
The red flags
In the third phase there is a list of signs that force a stop:
- "A quick fix for now". There is no "for now": you fix it properly or you escalate.
- Proposing a fix before having traced the complete flow of the data. That's guessing.
- Trusting a subagent's reading of the code, or your own, without checking it against the real case. A piece that fits "in theory" can depend on data that doesn't exist in production.
- Doing a find and replace over a shared file to isolate an experiment, without scoping it to your block. After any bulk edit, you look at
git diff --stat: if it touches more lines than expected, you revert it and redo it scoped. - Wanting to delete or consolidate shared code as a first idea, without having checked who else calls it. It's an assumption dressed up as elegance.
The fix: the least dangerous, not the cleverest
When there is more than one way to fix it, the skill prefers the one that can break the least if I'm wrong. In this order:
- A localized fix, scoped right where the bug lives, with its test. It's the default when I don't know who depends on the shared code next to it.
- A new, small, isolated piece, if the localized fix would force me to dig into logic I don't fully understand. It's more code, but it can't break what I haven't looked at.
- Touching the shared code (consolidating, deleting or adding a guard) only after having checked every place that calls it. It's the strongest fix, because it makes that kind of bug impossible, but if I miss a caller, I break more than I fix.
The regression test is the same one from the first phase, the one that was already failing with real data. You don't write a new one: you apply the fix and check that this one goes green, and then the whole suite. And if the fix touches more than five files, the skill asks me before continuing, because that's too much for a bug.
Why an integration test
I work with e-commerce flows and, in that kind of system, a bug rarely lives in a single place. It depends on several contexts at once: contacts, taxes, invoicing.
Each piece on its own is usually fine. Even so the bug exists. That's why an isolated test doesn't work for me: I need an integration test that replicates the whole path. The skill prefers it to a unit test with mocks, because a test that fakes exactly the dependency where the bug lives can make a wrong fix look correct. It's an old debate, classical style versus mockist style, which Fowler sums up in Mocks Aren't Stubs.
The report
The fifth phase ends with a report shaped like this:
Symptom: [what the user reported]
Root cause: [what was actually wrong, confirmed with evidence]
Fix: [what changed, with file:line]
Evidence: [test/reproduction output before and after]
Regression test: [file:line]
Status: DONE | DONE_WITH_CONCERNS | BLOCKED
The skill won't let me say "this should fix it". It proves it with the test and with the original reproduction. And if the test used real production data, it says so, without passing it off as synthetic.
When I don't use it
It isn't for a typo or a one-line fix that I've already diagnosed myself. And it's slow by design: seven phases with a question each. That's the price of not letting anything get skipped.
To start something from scratch, AI on its own is still fine. For a bug, this.
Top comments (0)