One of the more frequent arguments I hear against automated testing, and by extension, continuous delivery, is the idea that some problems are too important to trust to automated tests. Someone might die if a bug gets through. Or maybe billions of dollars would be lost.
Common examples are things like: firmware for pacemakers, self-driving cars, and Mars landers.
I guess this question seems reasonable on the face of it, because it comes up all the time.
But hidden within this question is the erroneous assumption that manual testing is more reliable than automated testing.
What makes it worse is that the assumption isn’t always wrong. The problem lies in the fuzzy definition of “testing.” Some kinds of testing definitely do require an actual mind, which can reason about things, apply intuition, etc.
But that’s not the kind of testing we’re talking about when we discuss automated tests, or continuous delivery, is it? We’re talking about rote regression testing. “Run through these 10,000 scenarios, and make sure none of them behave in unexpected ways.”
This is exactly the sort of testing that humans are notoriously bad at doing. And not only because they’re dreadfully slow, but also because they easily make mistakes. They also tend to get bored, and the better ones require salary increases every now and again.
Would you rather hop behind the… eh, wheel(?) of an autonomous vehicle that had the last software update tested automatically, by a computer that can examine 10,000 corner cases per second, or by a human sitting in the, er… driver’s(?) seat, trying to trigger unusual scenarios to ensure that the car behaves correctly?
When you stop to think about it, this common question “How can we trust automated tests in high-stakes scenarios?” is exactly backwards.
How can we possibly trust manual tests in high-stakes scenarios?
If you enjoyed this message, subscribe to The Daily Commit to get future messages to your inbox.
Top comments (0)