Testability Is a Feature. Does Your Code Agent Know About It?

I always say that testability is a feature.

It needs a customer – usually testers. We need to define what it means. And we need to build it in.

The question is what happens to testability when someone else builds the code.

Yup. Genie talk again.

Let’s look at just 3 aspects of generated code, or any code really, that impact testability. But when our genie doesn’t get directions, it will skip those and cause us a big headache.

1. Complexity

Of course, complexity affects testing. The more complex the code, the more work it takes to verify. More cases, more time – time that we don’t really have.

My classic example is recursion. Recursion seems simple (in code lines), but it can hide all kinds of bugs in there. Edge cases galore.

But that’s usually a function. That can be confined and handled. Today we’re generating systems.

And they are as complex as the genie wants. Or the systems it was trained on. And those are not a model for simplicity.

Complexity is the nemesis of testability. And unless we ask for simplicity, and make sure we got it, we pay for that in more expensive testing.

2. Observability

Observability is a key part of testability. Without it, we may be able to operate the system, but may not see the impact of those operations.

For example, if you had one POST API in the world that saves data in the database, and you want to check if it works, you’ll need to choose between looking inside the database (which may not be possible), or rely on the status code (which doesn’t tell you anything about the actual impact). Adding a GET to read helps, and presto – you’ve got observability.

Of course, you need to ask for that GET API.

In CRUD systems, you’d get that API, I’m not worried. Even in generated code. The problem starts with the not-so-intuitive stuff. What gets logged and where. Knowing how long it takes to see the result, and maybe how much time to wait between operations and states.

The more the system exposes state and data, the easier it is to understand, and the cheaper it is to debug and reproduce issues in it.

Now – what are the chances you get the things you need for your non-generic needs, from a generic code generator?

Exactly.

3. Security

I don’t need to explain how much generated code suffers from security issues. First – not new – it comes from the code examples out there, which are not that secure.

But here’s one thing we don’t talk about. We usually plug security holes by tools. They scan the code and find patterns that are possible entry points.

But now, we have a problem. First – the tools are ok, but they now also use models. Which are not perfect, they created the code in the first place. Plus, we use them to fix the code. Again, the same tools that created the code.

This is where I get trust issues.

But it’s not enough to call APIs and see the result. We need to plan and design cases to identify security issues. The impact of these issues are inside different systems – our app, the database, internal network, the billing system.

Even if our app is testable (and we know how often that’s true), what about the extended environment? How easy is it to design security cases, run them, and see the results?

The risks of code we don’t know don’t only increase the impact of what we know we need to check. They increase the hidden security issues too.

Ok, the devs use those code agents, not you. What can you do?

1. Talk to the developers. If you don’t, that’s a good line to start with: “Hey, I heard you use code agents”. Works great at parties.
2. Ask to see the code. Even non-coders understand simplicity. And they can see and understand reports from scanning tools.
3. Explain how you’re going to test the app, and wait for the “you can’t do that” moment. This is where you make your request.
4. Define what testability means for you and communicate it to them. They can ask (they would call it “tell”, but don’t fall for that trick) the genie to generate the code with those guidelines.

I’ve said it before, AI amplifies problems and solutions we had before. We need to adapt to the quantities and new risks.

---

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.