Why there is no way around test automation (except one)

Right now, manual testers in software quality assurance basically fight a lost cause. With every sprint and every iteration, the number of features increases as the software grows, because the work of developers (ideally) adds up.

But the work of manual testers doesn’t add up. New features have to be tested as they are introduced. This part of the work scales with the amount of developers. But software can break in interesting ways. So all features should be tested before a release, including existing ones. If a team has a fixed number of developers and testers, over time the testers are bound to fall behind.

There are only three possible solutions:

1. Add more manual testers as the software grows
2. Let users find your bugs
3. Automate tests

Adding more manual testers

Adding more manual testers means to increase the cost. More people in a team do not scale linearly, as the overhead for organization and communication increases. So even with unlimited budget (which most companies don’t have) this is only a limited option. And even if it was, there are many reasons, why doing manual regression testing is a bad idea.

Let users find your bugs

Accepting the risk by testing only a small sample of the features or testing them insufficiently is only acceptable in certain situations. Google and Facebook do not implement life-critical software, so showing an error to some small percentage of their users is a viable solution – for them. This is what let’s them implement continuous delivery. But this possibility tends to be the exception. If your software is installed locally or is critical in any way, this is not an option.

Automate tests

So eventually you end up in the situation that you have to automate the tests in order to deal with this inequality. This is probably the reason why test automation has seen such a boom in recent years, and why test automation engineers are in such demand right now.

There is a huge and recurring discussion between testers, whether test automation will eventually replace human testers. Whoever thinks that got both the reason for test automation and the capabilities of test automation wrong. You have to automate tests. But not in order to replace testers. You have to automate tests in order to enable testers to do their job: proper testing of new features.

Test automation is despised by some. Or at least it appears that way. They repeat all over again, that test automation is not automated testing. And that test automation is of very limited value. I understand where this comes from and why they stress this point so much. I also understand, that

a test tool is not a strategy; test automation is a development process; test automation is a significant investment; and test automation projects require skills in programming, testing, and project management.

While all of that is true, it mostly refers to specific tools and specific experiences. And it doesn’t address the issue I stress here. I acknowledge that most of today’s test automation tools are far from perfect, but that is another discussion.

---

What test automation can’t do

Test automation is very much comparable to a version control system. It highlights changes of the behavior of the system and asks the user to verify or undo those changes. Thus it cannot find bugs that existed at the time of the creation of the test (historical bugs). Because these kinds of bugs are already baked into the tests. Hard-to-change existing tests can even be detrimental, as they enforce defective behavior.

Test automation can only find new bugs in old functionality and only a specific kind of such bugs. It can only find bugs that manifest in changed behavior. It cannot reason or understand the software, so test automation doesn’t find if the system becomes inconsistent. It does not find functionality that should have changed to preserve consistency. So even when using test automation, this is not a no-brainer. You still have to manually test and review old functionality, to make sure it stays consistent with the overall system.

Of course, testing exhaustively or testing everything is impossible. I would argue that testing is a risk–cost calculation. How much risk are you willing to take, or as others have put it “How long do you look into the rearview mirror”? So the number of tests to automate is always a cost-function.

What test automation can do

This is what test automation really does, albeit sub-optimal: it highlights changes.

Test automation helps to detect when functionality that once worked (and was tested and approved) ceases to work. In other words, test automation is a way to help you find unwanted changes to the behavior of the system under test. These unwanted changes are also called side-effects or regressions. Seen that way, regression testing and test automation are version control in disguise, i.e. version control of the behavior of the software.

Test automation is no silver bullet, but it can be of help. Test automation is a tool, a support, a utility function to help testers with what would otherwise be a sheer unmountable amount of effort.

TL;DR: With ever growing software, there is no way around test automation, unless you want your users to find your bugs for you. But this is meant to enable testers to do their real work: critically challenge the system instead of becoming routine-blinded.

This post has also been published on medium. If you liked it, please press heart, twitter or otherwise spread the word.