I've noticed that people are using AI mostly to generate code. New code is easy to test, right? AI excels at it, so thousands and thousands of tests are created and beautifully green.
But not all of us only work on new code, right? Most of us deal with legacy, brownfield projects with years and years of neglect and bad decisions, and also, no tests.
The business doesn't care. Execs want that feature and can't understand why it's so hard. Well, we touch one thing and another breaks. We give AI a job and it goes and touches more than it should!
What is left for us to do? People often jump to the "refactor" conclusion: "If I write this better, then I can implement it." Don't get me wrong, the idea is gold but often gets reality checked. Refactoring is hard, error-prone and doesn't add any short-term value, only risk.
Should we just ask AI to create tests for the existing code? Yes, we can. Does it work? No.
Why, you might ask? Well, code is problematic because it is entangled and often mixes different layers and responsibilities, so if you ask AI to test it, it will get it wrong. Tests will be coupled with the implementation, because guess what, someone is accessing data inside business logic. There's a transaction scope being created while calculating amounts and persisting a payment. It happens.
What is the solution, then? What if we asked AI to exercise live code top-down? Instead of worrying about testing each unit, mocking dependencies and coupling your assertions to that (which limits your refactoring power), why don't we ask AI to write characterization tests instead?
Why is AI great at helping with characterization tests?
I like to think that whatever AI writes is wrong until someone validates it. Often, validating is as hard as writing the thing yourself! And that's why I can't fully trust AI to write code that isn't tested.
If we ask AI to write the characterization tests, guess what, it is not writing production code! The risk is lower. But you may ask: if we don't trust production code, why would we trust the tests?
Good question — we shouldn't. But we can introduce mutation testing into the mix. That changes everything.
Usually, I ask AI to do the following, without modifying production code:
Sometimes I go for 90% coverage, sometimes less. Why? Coverage can get tricky when there's lots of try/catch flows, and since you aren't mocking anything, you can't exercise them.
What do we gain from that?
So far, AI hasn't written anything that could impact production (you need to ensure that nothing is dangling from the mutations!), but now we have a signal (or noise) that tells us if we've changed any existing behavior.
This confidence can help us navigate the existing code without being too afraid of breaking it. Even better, we can tell AI to change production code now!
Are we adding value?
In my book, just the tests alone are a huge improvement. Is it perfect? No. Just better. If previously we had 0% confidence in a particular code flow and now we have 5%, that's winning.
Refactoring? Possible. New features? Possible. We win, the business wins, AI wins (we paid for those tokens), everybody wins.
All we need to think about is testing first. And I'm not asking anything hard, am I? It's just a simple flow you can ask any LLM to follow, and I guarantee you won't go back.
A point of attention, though: make sure you review the output of that flow at least once — then you are golden.
When should you not do this?
This process is time-consuming. It takes time for AI to parse the code, come up with the testing flows and validate itself. Because we aren't mocking anything, running the tests is slow, flaky and far from perfect.
In my travels, it usually takes 4-5 hours to get into a good state if you start testing from a controller.
Therefore, if the change is small enough and you know the codebase and how to test it, then don't do this.
Also, I will be honest with you. I don't usually commit all tests. They are slow! With confidence, I am able to refactor the code and test it properly with unit tests and focused integration tests.
Top comments (0)