Before AI we had this problem: The person who wrote the nifty algorithm left the project months ago.
There’s a bug we have to fix. And you don’t know the code. Good luck! We’re right behind you!
Now with AI, we’re creating the problem ourselves. We generate code that we barely look at. And it didn’t go through our brains, so it didn’t leave a trace. As the old joke goes – the good news is you always see new code.
But that’s not funny when you need to fix that bug.
I’ve talked about losing debugging skills before, but you probably want more down-to-earth advice.
So, how to debug code I didn’t write. And that goes for both AI and human generated code.
Is it really a bug? How do you know?
You see, if there’s a test – it’s a great start, we’ll get there in a second. But if not – we’re relying on knowledge in people and in docs. We need to know what’s the expected behavior and what we don’t see.
With AI code, there’s no one to ask. The prompt is gone, and the intent is gone. So you may need to dig deeper.
You know what should happen, you don’t understand why it doesn’t
Do you have a test that broke? Excellent. It doesn’t matter what kind of test – API, unit – whatever. Something that worked before but doesn’t now and has a proof means you don’t need to debug, just fix!
I know, it doesn’t work like that. But it can.
You don’t have a test? Write one. With the expected behavior, and it should fail. What kind of test to write? This depends on how much you “know” the code. If you have a suspicion the problem is in the backend, don’t write a UI test. If it’s probably in the logic, maybe a unit test will be enough.
The smaller a test, which fails and gives you the proof that it really doesn’t behave as it should, the better. A smaller test is easier to fix and pass.
Before digging, run your failing tests a couple of times, to make sure that you’ve captured the failing behavior.
The problem may not happen all the time. Or the setup may be right for the first run, but not the second. And if you’re running AI powered features, the assertions may not capture the right things.
Don’t forget the greatest technique of debugging.
Binary search debugging. If your test proves the problem is not in part A, change it to start testing B.
The rest is proper old school debugging.
Breakpoints, printouts and logs. Building models of how the bug can occur, printing state and data, and hoping we catch it when we’re looking.
Not all bugs are like that.
AI bugs are problematic in a couple of aspects.
First there’s the repeatability problem. They don’t return the same answer every time. Then there’s the decision of “how correct is the answer”. This may not be a true/false result, but a lot of gray areas.
That makes it hard to capture a failure, and to understand if what we have is a failure or acceptable.
It’s the same techniques though – find how to set up the trap, how to spring it, and how you know it caught something. Hopefully the right thing.
Yes, the classics never die. The difficulty level just goes up.
Originally published at testingil.com.
I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.
Top comments (0)