DEV Community: Gil Zilberfeld

In a Few Weeks, I Won't Know How My Own App Works

Gil Zilberfeld — Wed, 09 Sep 2026 12:53:49 +0000

I’ve been doing a lot of code reviews recently, and I’ve noticed something about my experience during the reviews. And what it means for code in general, but specifically for AI generated code.

But before I do that, I want to take you back to the ’00s. A simpler time. Back then I started doing something I only read about in books – pair programming. Ok, that’s a lie. Everyone who grabbed someone and brought them to look at your code because it’s doing something weird – you’ve done pair programming.

But this was pair programming for like 80% of my work. And of course, not just me. This links directly to my topic – our understanding of the code. And in the age before AI, where teams really knew their code, that would be the biggest grade I’d give for code understanding.

Meaning, if you and I work both at the same time on the same task, and finish it. I’d say we’re both at the highest level of understanding of that feature – what it needs to do, how it works, what it depends on, what depends on it, and how it’s written and tested.

From here that level of understanding is going to drop. If you’re part of a team, but didn’t work on that feature, you probably know it exists. If we have dependencies, you probably know about them. But the rest is either non-existent or close to it.

If you’re on another team, that level of understanding drops, and if you’re on another project – you may not even be aware.

That makes sense, but we’re not talking about that today, I want to focus on my capability to do a proper review. And I go back to understanding for that. In order for me to give you proper feedback, catch your mistakes and offer alternatives, I need proper understanding. The more understanding I have, I can review better and make better changes.

So far, so good. It makes sense.

Two more things make sense. First the use of tools for review. Every tool, from syntax analyzer to the smartest security checker, work according to patterns, not understanding. All the tools check for generic stuff. Newer tools that use LLMs, can find more elusive patterns. But in the end, it’s like finding a missing semi-colon.

The second thing is async code review. Or PR review, as people call it. This is done not with the person who wrote the code, but at my spare(?) time. I pull the PR, look at the code and based on my understanding, I need to approve it. Note that at this point, I can’t ask questions, so guess what I do? Look for patterns. If I’m the team leader and I know the feature, I can review better. If I’m on another team – I will not. Not take risks. Not initiate another coding and code review. Who has time for that?

And that brings me to reviewing code my genie wrote.

Because if you think about it, I’m the team leader, and the code agent is on my team. I’m supposed to know more about the feature, but I’m reviewing code I didn’t write.

How well can I do it? There’s another part that contributes to this, which is the size of the diff. Humans, before AI know their limitations (most of them), and so they don’t write a lot of code, because they know what it’s like to review a lot of code. But even if they do, the reviewer, unless she gives up, can still ask questions.

Well, my genie does a lot of work and creates a lot for me to review. Now, I know I can’t review a lot of code in one go, so I make my genie work in chunks. Chunks I can read and understand. For example I break features into workflows, and ask the genie to do a workflow or two, including all tests and whatever’s needed.

Which is still a lot. And I tried to explore what happens that makes me give up. Yes I do give up sometimes.

It goes back to understanding. I’m reviewing pieces of code – tests, bits of FE, bits of the API code, bits of logic and bits of database code. I can review these at the lowest level – does it do what I need? Can I read it?

But when it comes to bigger things – like dependencies and side effects and performance – I give up. First there’s a lot of code to absorb, and then I need to piece everything that I know (or think I know) of how the rest of code works and designed. I’m building a code model in my mind. This helps me identify design issues, find the side effect traps. Maybe the performance issue hiding in there.

But, if it’s hard to build that puzzle, I’m not going beyond “does it look ok” and “do the tests run”.

So, I’m taking risks. Risking that there’s code in there that I don’t know what it does, how it works and what effect it might have on the system today. And tomorrow, I won’t even consider it for any co-dependence with my new feature, because I don’t understand how it works.

This accumulates over time. Within a few days or weeks – I won’t know how most of my app works.

So what do I actually do about it?

Two things, and neither of them is a fix.

The first one you’ve already seen. I keep the genie working in workflows, not in features. It makes the diffs smaller, and keeps them inside a context I can understand. The second one is a decision. Before I review a chunk, I decide what is important for me to go deep into, and what to go over lightly. Or not at all.

It doesn’t come close to the old level of understanding. But my overall feeling of understanding has improved.

Now, take that, and think about teams and groups. How much code and tests that nobody knows and understands can be accumulated in one year. Let alone three.

Do you understand this risk? And how are you handling it?

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

Testability Is a Feature. Does Your Code Agent Know About It?

Gil Zilberfeld — Wed, 09 Sep 2026 12:48:21 +0000

I always say that testability is a feature.

It needs a customer – usually testers. We need to define what it means. And we need to build it in.

The question is what happens to testability when someone else builds the code.

Yup. Genie talk again.

Let’s look at just 3 aspects of generated code, or any code really, that impact testability. But when our genie doesn’t get directions, it will skip those and cause us a big headache.

1. Complexity

Of course, complexity affects testing. The more complex the code, the more work it takes to verify. More cases, more time – time that we don’t really have.

My classic example is recursion. Recursion seems simple (in code lines), but it can hide all kinds of bugs in there. Edge cases galore.

But that’s usually a function. That can be confined and handled. Today we’re generating systems.

And they are as complex as the genie wants. Or the systems it was trained on. And those are not a model for simplicity.

Complexity is the nemesis of testability. And unless we ask for simplicity, and make sure we got it, we pay for that in more expensive testing.

2. Observability

Observability is a key part of testability. Without it, we may be able to operate the system, but may not see the impact of those operations.

For example, if you had one POST API in the world that saves data in the database, and you want to check if it works, you’ll need to choose between looking inside the database (which may not be possible), or rely on the status code (which doesn’t tell you anything about the actual impact). Adding a GET to read helps, and presto – you’ve got observability.

Of course, you need to ask for that GET API.

In CRUD systems, you’d get that API, I’m not worried. Even in generated code. The problem starts with the not-so-intuitive stuff. What gets logged and where. Knowing how long it takes to see the result, and maybe how much time to wait between operations and states.

The more the system exposes state and data, the easier it is to understand, and the cheaper it is to debug and reproduce issues in it.

Now – what are the chances you get the things you need for your non-generic needs, from a generic code generator?

Exactly.

3. Security

I don’t need to explain how much generated code suffers from security issues. First – not new – it comes from the code examples out there, which are not that secure.

But here’s one thing we don’t talk about. We usually plug security holes by tools. They scan the code and find patterns that are possible entry points.

But now, we have a problem. First – the tools are ok, but they now also use models. Which are not perfect, they created the code in the first place. Plus, we use them to fix the code. Again, the same tools that created the code.

This is where I get trust issues.

But it’s not enough to call APIs and see the result. We need to plan and design cases to identify security issues. The impact of these issues are inside different systems – our app, the database, internal network, the billing system.

Even if our app is testable (and we know how often that’s true), what about the extended environment? How easy is it to design security cases, run them, and see the results?

The risks of code we don’t know don’t only increase the impact of what we know we need to check. They increase the hidden security issues too.

Ok, the devs use those code agents, not you. What can you do?

Talk to the developers. If you don’t, that’s a good line to start with: “Hey, I heard you use code agents”. Works great at parties.
Ask to see the code. Even non-coders understand simplicity. And they can see and understand reports from scanning tools.
Explain how you’re going to test the app, and wait for the “you can’t do that” moment. This is where you make your request.
Define what testability means for you and communicate it to them. They can ask (they would call it “tell”, but don’t fall for that trick) the genie to generate the code with those guidelines.

I’ve said it before, AI amplifies problems and solutions we had before. We need to adapt to the quantities and new risks.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

AI vs CI: Nobody Told the Pipeline

Gil Zilberfeld — Thu, 20 Aug 2026 11:17:28 +0000

CI is so commonplace these days, we don’t often think about why it’s there. And that reasoning really matters these days, because of how we use it.

Especially when you’re writing AI agents and expect the CI system to run a regression test suite, and sound the alarm when something goes wrong.

Here’s a funny story you may not know. CI - Continuous Integration – was not originally a tool category. It’s a process - it’s how you continuously integrate pieces of code together.

Anyway, why do you need this process anyway? Because you’re scared. You’re terrified that integration broke something.

So what do you do? You look for approval. An automation system that runs all the tests all the time is our approval of choice.

That’s how CI became a tool category. Because in its heart, it’s a simple automation pipeline.

So, that automation’s holy grail is speed, right?

Wrong.

The mission was to kill “Works on my machine”

Remember when we wrote software in a cave, and we had a sticker on our computers – “Works on my machine. Don’t run it on another cave”?

Ah, the good old days.

Because the CI automation #1 OG mission was to kill “Works on my machine”. How? By creating a repeatable process. Independent of all the weird installs on my machine. Multiple versions of libraries. And different configurations. And admin hacks.

The CI builds, packages, runs and tests the software every time the same way. That way, if a test turns red, you know something bad happened, because it was green until now. And why your stomach turns, when it flakes between red and green – that’s the feeling of lost repeatability.

Yes, repeatability brings bliss. Well, it did, before AI.

Then AI showed up

If your app does not touch AI (although building with code agents, can sometimes count as “touch”), you’ll continue to feel that bliss.

But if you’re developing AI-based features, or agents, or vibe-coding, you’re in for a new experience every time you push your code (or prompts) into CI.

Things are not repeatable anymore. Models change, sometimes without warning. And always under your feet. And when a model changes, does it trigger a run?

You wish! Your alert system is malfunctioning.

And the worst part is not a regression. Although if it’s a bug in a prompt, you may not be able to fix it.

No, the worst part is that Green today doesn’t tell you what kind of Green it is. Things are no longer Working or not, they are Work-ish. But even that’s not the same work-ish every day.

So we’ve got fewer triggers, results we can’t read, and on top of it – every run means something else.

What’s the solution?

Repeatability gives us confidence. CI is the process, and tools, that give us the approval we seek, and the confidence we have comes from that repeatability.

We can’t rely on the trigger anymore, but we can initiate our own runs. And since one run is not enough, we need to run more and look at the trends. Catch drift before it ships.

What happens to quality when AI meets CI? We need to get back to the reasoning, and change how we manage quality.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

It's Almost Like You Need an Actual Programmer

Gil Zilberfeld — Thu, 13 Aug 2026 07:07:33 +0000

Vibe coding ain’t what it used to be.

Back in the old days (2 years ago), we conjured up apps with our own programming language.

Turns out, it does work, but more for prototypes. If you want an actual working production app, you need that thing from before. How do the old timers call it? Ah yes. Code.

So our builder says to the genie: “Code me this”.
And the genie does build this. Mostly.

When we’ve got a prototype and it’s working, two things happen. The first thing is management thinks they have an actual product. In our case, it’s the builder that thinks “code me this” is simply a translation layer.

But then, another thing happens – we now have to take care of our puppy. I mean the code.

As long as we don’t need to touch it, it doesn’t matter what it looks like. But then we do.

So what does our hero think? What worked before, will work again. He tells his favorite code agent – “build me my greatest feature yet”.

But then weird things start happening. Bugs that weren’t there. Multiple tries of code changes – costly ones, mind you. Performance slows down.

It’s almost like you need an actual programmer to do the work.
And you do. You know what helps developers? And coding agents?

Clean code. Readable, maintainable code. The classics are still with us.

Now, coding agents are getting better at producing code. However, they are only as good as the code they are trained on. Now think – what do you think the average level of code out there is?

Exactly. That’s what you’re getting.

So if you are on a transition from prompts to code, it may look like continuing vibing is the shortest way to more features. Yet, the first thing you need is a good platform to grow your code.

Now it’s time to take control.

Read the code.

You’ll do this first anyway. The code is the only truth you have. The prompts you used before are scattered across forty chat sessions and there were never any docs.

Just know what you’re up against. Agents drop a lot of code, and after ten files your eyes glaze over. You won’t read all of it, but try to understand the important bits.

Write down what it should do.

Ask the agent to derive documentation from the codebase. It’s fast and it beats no docs at all.

But know what you got. Those docs describe what the code does, not what it should do. If there’s a bug in there, it appears as a requirement. Review and sieve out the wrong stuff.

If you can find them, collect your prompts. They were the intent for building the app.

Review both, and drop everything that doesn’t add up. Now you have a documented system.

Now write tests. Against the requirements.

It’s tempting to generate tests for what’s already there. And for legacy apps, I recommend capturing existing behavior of undocumented knowledge built up for ten years.

But here? You’re better set. The intent is still around. Capture what the code does today and you’ve just made every mistake a requirement.

Then you can change it.

Including asking the genie to change it for you – which is the part everyone wants to skip to.

But now, with documentation and tests in place, now it’s a lot safer to make changes.

Next you need to understand the principles of Clean Code. If you don’t, you may ask the genie to change the code for you, but you’ll get more of the same. We don’t want more of that mess.

Exactly like programming. Who knows, maybe you’ll become one eventually.

So what do you say – Are you ready for clean up?

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

The New Technical Debt

Gil Zilberfeld — Thu, 13 Aug 2026 07:06:32 +0000

Is AI code debt the new technical debt? As I keep learning, generating code and examining it – it’s worse.

In the old days, we worked hard to create technical debt. Just kidding, it was easy.

Technical debt has many definitions, and origin stories. But one thing is common: We leave the code as it is, knowing it could be better. The debt is the gap of effort of making it better.

And why the gap? Because we know we’ll see that code again. And when we do, it’ll be hard to change. Better code would have made it easier.

Now we’re in the age of genies. They can write any code. They can change any code. Do we need to worry about AI code debt?

Sure we do.

AI code debt is exactly the same as technical debt – we’re leaving the code as it is, knowing it could be better. The gap is still the effort of making it better.

But this time the gap is a lot bigger. There’s a lot more code, it’s probably a lot more complex, and some of it – let’s be frank – is code we didn’t review. So the gap is a lot bigger than we guess.

But that’s a bot problem, right? The code agent will deal with all the needed changes. We don’t need to even look at the code.

Nah, you know you will. You know what code is generated, and it’s not how you would have written it. And coding agents have the same problem of making sense of complex code bases as us.

And they’ll make mistakes.

Can the code agent clean the code up by itself?

Cleaning is not just “make it readable”. It’s preparing it for more changes, reducing dependencies and isolating interfaces – all in the context of future plans. And the agent doesn’t have this context.

In fact, it will assume another context, and we get stuck with the code and the assumptions.

How do you pay down AI code debt?

Same as always: find the code you know you’ll touch again, and make that code easier to change. The problem starts when you don’t know where that code is.

Because, you didn’t write it. And you didn’t review all of it. If any.

Putting a price on technical debt before was an exercise in imagination. Now it’s pure fantasy.

How do you avoid it next time?

First know what code was generated. And for that you need to enforce smaller code generation.

If you don’t enforce it, you’ll have a lot more to review. And if that happens you won’t review it all. It’s a human thing.

Then, you can wish the genie to refactor it to take the shape you want. Genies are good at transformations. And if the genie breaks something, your tests will tell you.

You do have tests, right?

The old technical debt was based on maintenance work. Now we have bigger maintenance queued up, along with risks of unverified code. Not cool.

So, first, we need to be aware of the AI code debt – we’re creating code that will cost a lot more to maintain, than “regular” code.

Before, we thought “it’s ok, we’ll take care of that later”, and then, when the bill came it was a lot more than we thought.

Now? Expect a much bigger one.

Questions people ask about AI code debt

What is AI code debt?

AI code debt is technical debt created by generated code: you leave the code as it is, knowing it could be better, and the debt is the effort of making it better. The difference is scale. There’s more of it, it’s more complex, and some of it was never reviewed.

Is AI code debt different from technical debt?

Not in kind, only in size and visibility. The old debt was something you put there and remembered. This debt arrived while you were reading something else, so you don’t know where it is.

Can an AI agent clean up its own code debt?

Not on its own. Cleaning up means preparing code for changes you plan to make, and the agent doesn’t have that context. It will assume a different one, and you’re left with the code and the assumptions.

How do you check that an AI refactor didn’t break anything?

With tests you wrote before the refactor. Directing the genie to reshape code is fine as long as something independent tells you when it breaks.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

You Can't Fix A Bug With A Prompt

Gil Zilberfeld — Thu, 06 Aug 2026 10:46:27 +0000

Let's talk about fixing bugs. My second favorite activity after creating them.

How do I know the bug is fixed? Well, I've been taught by wiser and more experienced people than me (way back when), that if you want to make sure the bug never shows its face again - write an automated test for it.

Ok, I admit that there were times I didn't write a test. Some fixes are so trivial, that sometimes a test is a luxury.

But even then, I was working on a hidden assumption. Just like in Fallout, code never changes. Until it does. But between those times it never changes.

I fix the bug. From this point until it actually changes by a human, or a bot or someone in-between - that code will compile, or transpile, or whatever we call "runs" the same way. A statement will execute, a condition evaluated, an exception caught - every time in the same way.

And now we come to modern programming languages: Prompts. Agents, prompts, workflows - they don't run the same way every time. What they do is run their interpretation of the request. If they run tools - they run the deterministic parts. If they run sub-agents, they run interpretations based on other interpretations.

Interpretations work mostly the same way, until they don't. When I was teaching my agent to develop in TDD, I had a couple of requests. One of them was to not create code without a test.

Which I thought was a very normal request from an agent. In fact, when I started out by telling it - we're working in TDD - I assumed it knows what TDD is. Ha.

Always remember that what LLMs know is exactly what most of the population knows. And usually the "don't create any code without a test" falls through the cracks.

So I made it official: One of the agent rules was exactly that. This was a legitimate bug fix. At least I thought so.

But it really was a suggestion. Which the agent considered, and depending on its mood, sometimes did, and sometimes didn't.

I won't go into the full solution (still in progress, if you want me to elaborate, comment), but part of it was to run a targeted coverage tool - a deterministic one, check it and stop the process if it found extra code. The TDD sequence looks like this:

Per step:
1. Write test(s).
2. `node scripts/tdd.mjs red <test-file>` — all newly added tests must fail. Pre-existing passing tests in the same file are allowed to remain passing.
3. Implement minimum to pass.
4. `node scripts/tdd.mjs targeted <test-file>` — targeted coverage + pass check.
5. `node scripts/tdd.mjs lint` — ESLint.
6. `node scripts/tdd.mjs full` — full suite + coverage.
7. Script says STOP. Human reviews.
8. Human runs `node scripts/tdd.mjs commit "message"`.
9. Human confirms next step. AI runs `/clear`.

But each step here is the suggestion. The real enforcement is done in the tdd.mjs code. Real code.

But this is just an example. The real problem is that more and more "code" is not programmed. It's interpreted. That means that bugs are a lot more flaky to catch, but also are not permanently fixable.

And don't get me started on companies switching model capabilities every other Tuesday. In the past, updating versions was a whole ceremony because we were worried something would break.

Now LLM providers do it for us without us knowing.
We can live with that. We should make sure we know.

And remember - bug fixes in prompts are not real bug fixes. They are more like wishes. Which may or may not come true.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.