You Can't Fix A Bug With A Prompt

Gil Zilberfeld — Thu, 06 Aug 2026 10:46:27 +0000

Let's talk about fixing bugs. My second favorite activity after creating them.

How do I know the bug is fixed? Well, I've been taught by wiser and more experienced people than me (way back when), that if you want to make sure the bug never shows its face again - write an automated test for it.

Ok, I admit that there were times I didn't write a test. Some fixes are so trivial, that sometimes a test is a luxury.

But even then, I was working on a hidden assumption. Just like in Fallout, code never changes. Until it does. But between those times it never changes.

I fix the bug. From this point until it actually changes by a human, or a bot or someone in-between - that code will compile, or transpile, or whatever we call "runs" the same way. A statement will execute, a condition evaluated, an exception caught - every time in the same way.

And now we come to modern programming languages: Prompts. Agents, prompts, workflows - they don't run the same way every time. What they do is run their interpretation of the request. If they run tools - they run the deterministic parts. If they run sub-agents, they run interpretations based on other interpretations.

Interpretations work mostly the same way, until they don't. When I was teaching my agent to develop in TDD, I had a couple of requests. One of them was to not create code without a test.

Which I thought was a very normal request from an agent. In fact, when I started out by telling it - we're working in TDD - I assumed it knows what TDD is. Ha.

Always remember that what LLMs know is exactly what most of the population knows. And usually the "don't create any code without a test" falls through the cracks.

So I made it official: One of the agent rules was exactly that. This was a legitimate bug fix. At least I thought so.

But it really was a suggestion. Which the agent considered, and depending on its mood, sometimes did, and sometimes didn't.

I won't go into the full solution (still in progress, if you want me to elaborate, comment), but part of it was to run a targeted coverage tool - a deterministic one, check it and stop the process if it found extra code. The TDD sequence looks like this:

Per step:
1. Write test(s).
2. `node scripts/tdd.mjs red <test-file>` — all newly added tests must fail. Pre-existing passing tests in the same file are allowed to remain passing.
3. Implement minimum to pass.
4. `node scripts/tdd.mjs targeted <test-file>` — targeted coverage + pass check.
5. `node scripts/tdd.mjs lint` — ESLint.
6. `node scripts/tdd.mjs full` — full suite + coverage.
7. Script says STOP. Human reviews.
8. Human runs `node scripts/tdd.mjs commit "message"`.
9. Human confirms next step. AI runs `/clear`.

But each step here is the suggestion. The real enforcement is done in the tdd.mjs code. Real code.

But this is just an example. The real problem is that more and more "code" is not programmed. It's interpreted. That means that bugs are a lot more flaky to catch, but also are not permanently fixable.

And don't get me started on companies switching model capabilities every other Tuesday. In the past, updating versions was a whole ceremony because we were worried something would break.

Now LLM providers do it for us without us knowing.
We can live with that. We should make sure we know.

And remember - bug fixes in prompts are not real bug fixes. They are more like wishes. Which may or may not come true.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

DEV Community: Gil Zilberfeld

You Can't Fix A Bug With A Prompt