Years ago, I discovered I had a superpower.
Unfortunately, I couldn’t throw heavy things for a long distance. Which is sort of a blessing when matched with my nearsightedness. But, it was useful in some areas, and lucky for me, the ones in my chosen profession.
I was known as “Debugger-Man”!
Ok, I wasn’t called that, but when people needed to debug the undebuggable they called me.
Now you probably think, every developer, and automation people – everyone can use a debugger.
First of all, I really hope so. But then, debugging is like any activity, it’s a lot of skills, and one of them is using the right tools for the job. Like debuggers. And logs. And others.
What was I talking about? AI.
One of the things that I find fascinating about AI agents is their ability to debug.
You see, it can do a lot of things Debugger-Man does. It reads logs. It looks at external effects and compares to the code, and reasons about what the problem is.
It also knows how third-party things work, and therefore, its hypotheses become more confident. And if the code is clean and has tests, it has tools to understand and test those hypotheses.
Cool stuff, and it can do it without a human. Sure, it sometimes makes mistakes, but we’re there to help it along the way.
Hmm. Not sure I didn’t lie about the last part.
You see, even Debugger-Man admits that his skills are eroding due to using AI. My genie created a whole harness just to check a multi-threading issue it was sure about. I totally believed it. It didn’t find the bug. But I have a harness.
The genie does a lot of the work right. And if it doesn’t, I catch it on its way down, pick it up, and direct it in the right direction.
But that requires a couple of things. First – I need to know what I’m doing. This is knowing the app, the design, the architecture, and connecting more information to make a model in my brain to see if that genie has the same one, and it follows it.
I also need to know what the agent is doing. Read all the sub agent logs, in real time if things are important. And they usually are.
Then I also need to decide if what it’s seeing and doing is the right way. If the hypothesis is true, and if the test that should corroborate it is fit for this purpose.
And if there’s a fix in the code, if it really solves the problem.
Or I just nod my head at whatever it’s doing and trust it.
Which brings me to our main issue. Debugging is a way to know the code. You can’t be Debugger-Man if you don’t know it.
We used to talk about code ownership in organizational terms, who should know the code well enough to maintain it. Who can answer questions about it. Who can we trust to make the bug fixes without causing ripple effects.
If all “who” questions now answer “that genie in the corner”, we have a problem.
First, as lawyers can tell you, “the genie did it” is not a defense. But more importantly, the assumption that the genie knows it all – that we, the real experts, have given it everything we know, and that’s everything you need to know about the app, the users, the assumptions, and a million other things – is… not a good one to rely on.
In an organization, we don’t hold everything in our heads. We have different people, docs, systems that hold the information. But mostly people.
They have the answers, and the skills and tools to dig for them, if they don’t.
Well, they should. The more we trust AI to do this, the more we lose the skills, and rely on the genie for the rest of the app’s life.
That’s adding features, fixing bugs, keeping the code clean, and integrating with other systems. And identifying the problems and fixing them.
AI is better than humans in collecting information and analyzing it at great speed and not so bad accuracy. But imagine your reliance on tokens to do all this in five years. Where people don’t know why the code looks like that. And who the hell decided to write the C functions in javascript.
Debugging is a set of human skills. Using them allows us to keep being system experts.
One day a bug will appear. And a big Debugger light will be seen over the clouds.
But Debugger-Man will not appear.
You can delegate the debugging. But the knowledge has to stay with you.
Originally published at testingil.com.
I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.
Top comments (0)