The myth of “unstable” code

The most common reason I get for people delaying test automation is that the code is “unstable”, and automating too soon will result in a lot of re-work of automation. Better to only automate after a feature is “stable”, i.e. development work is all done, they say, so you won’t have to rework any of the automation. This belief is a myth that makes automation harder in the long run, not easier.

Your code is not unstable

This might feel like quibbling with semantics, but consider these two statements:

• Our code is unstable
• Developers are changing the code

When I probe what people mean by “unstable”, they often consider these two statements to be the same. To me, they could not be more different. “Unstable” to me means unpredictable, that it exhibits different behaviour at different times, that there are unknown variables at play, that it is unknowable.

This description should, hopefully, bear little resemblance to how your developers change code. There can be a period when developers have to experiment with how to implement something and things truly changing quite a lot, but in 90% of development work developers are working against specific requirements, acceptance criteria, and planning. You, as a QA, should be a part of those planning discussions. If you are so disconnected from the changes the developers are making that it makes the code feel “unstable”, then you have a teamwork problem, not a software problem.

If your code truly is “unstable” in the sense that it doesn’t behave deterministically, then that’s a product issue. Either it’s deliberate and you need to figure out how to test for that, or it’s not deliberate and it’s a problem that needs to be fixed.

Your code will never be “stable”

Let’s say we really do mean that we want to wait for the developers to be finished before doing automation. What does it mean for developers to be finished?

In a waterfall or project development model, where there is a limited scope and a well defined beginning and end to the project, you might be able to do this. There are other reasons why you shouldn’t want to (which I’ll touch on below), but it could work if you can get a real commitment that development stops and you’ll have a long enough “test implementation” phase where you can automate.

In product development, though, where you have a continuous backlog of work that the team moves through with incremental releases, this is impossible. If you wait until devs are done one set of features to write test automation, that set of features might be stable but the code as a whole will still be unstable because the devs are still changing it for the next set of features. Again, you might be able to work this way if you have good controls to keep separate environments for each version of the application.

With ongoing product work, the product is never finished, so it will never be “stable”. This no longer works for a reason to delay automation.

Waiting doesn’t prevent re-work

The third part of this myth claims that by waiting to automate, you will prevent re-work.

Again, this depends on the same fallacy: the only way automation will never require re-work is if nothing about the application ever changes. For well defined standalone projects, maybe this can be true. But I’ve yet to experience that.

The re-work required for automation is actually worse when you delay it based on an unattainable standard of “stability”. As I’ve written before, waiting to automate triples the amount of extra effort required.

Debunking the myth

To summarize, given the mistaken belief that “we have to wait to automate because the code is unstable and we want to avoid re-work”, consider instead:

1. Your code isn’t unpredictable or unknowable: it is being changed deliberately by other developers on your team. Work with them so that you know what changes are coming and can plan for them.
2. Your code is likely never going to be “stable” because software development is usually an ongoing exercise in maintaining and extending products. In most cases, if you wait for your code to stop changing, you will be waiting forever.
3. Re-working automation is unavoidable and only gets more expensive the longer you delay keeping your automation up to date.

In limited circumstances, such as an early POC that will be thrown away before going to production or a waterfall-style standalone project, waiting to automate may make sense. In most cases, however, you will be able to have a much bigger impact on the quality of your product by automating early.