There's a belief that, when I share it, will sometimes make other folks in software tilt their heads at me.
The goal of QA teams isn't good testing.
The goal isn't even testing the "right" things in the "right" way.
The goal is doing our part to help teams develop software that does what it's supposed to do. Testing is one way we get there, but certainly not the only one and often not the best one. And yet, the go-to quality conversation at many places is still about testing and coverage.
Testing is a great tool for a specific set of jobs. Automated tests are your reliable, repeatable checks on Very Important Functionality (VIFâ„¢) that give you fast feedback as code changes. Exploratory testing builds understanding of how software actually behaves and gives space for testers to go off the map and follow their intuition (i.e. years of experience and pattern recognition passing itself off as a hunch).
At a lot of places though, testing has become the everything tool. It fills gaps left by quality work that didn't happen earlier + it stands in for the visibility that monitoring and observability should be giving us instead. That's a lot of weight for one tool to carry. And relying on test coverage as your primary quality signal is like waiting to reshape a loaf of bread until after it's already out of the oven.
The bread is your software. The ingredients are the things that have to be right before you even get to shaping it: a clear definition of what the software needs to do and for whom, alignment on what "good" looks like, clarity on what you're optimizing for, risks and constraints mostly known going in, etc. If you use the wrong flour or leavening, or skip the salt entirely, no amount of reshaping will transform that dough into what you intended to bake. Think back to the last time someone asked an "obvious" question and everyone on your team had a different answer. That's an example of the work. It's almost invisible (no artifact or dashboard to point to) so it's easy for teams to skip it or assume it happened at some point with some combination of folks.
In dough form (early in development, before the bulk of the code gets written) software is much easier to shape. By the time the dough has hardened into bread (after code is written and in testing) you need a lot more force to make changes. And if you do get the shape right eventually, it costs more time and effort to get there than it needed to.
So why do so many teams approach software quality like shaping a loaf of mystery bread after it's been baked?
In cases like this, test coverage is often used to fill other gaps. Test coverage tells us where we've looked (kind of). It doesn't tell us whether what we're looking at matters. An 80% coverage number could mean a well-understood codebase with risk-driven testing and thoughtful gaps. Or it could mean way too many tests that dont meaningfully exercise the paths your team most needs signal on. The number doesn't distinguish. And test coverage leaned on just before release is a slow, expensive way to learn things you could have learned much cheaper. Technically operational. Not the best tool for what you're trying to do.
Something I want quality-minded teams sitting with instead of chasing test coverage: where are the places you don't know how your software is going to behave and need to + what's the fastest, cheapest way to find out? Sometimes that's a test. Many times it's not. Many many times it's someone asking the questions everyone assumed were obvious that nobody actually had the same answer to, in a conversation that should have happened two weeks earlier.
There's a downstream side to this too, because most software breaks eventually. When it does, what's most important is how fast you find out and respond. Production monitoring and observability tell you something is wrong before your users do. Even the best testing won't give you this. Not cheaply, anyway, although I guess you could technically just run your 100% coverage automated test suite constantly against production, if you've got money and time to throw around.
So, yes: the goal of QA teams isn't good testing. It was never supposed to be. What we're here to do is help teams ship software that works, and the most useful version of that work isn't happening in a test environment.
And we know that AI raises the stakes here. You can now generate code and tests without the understanding ever happening. Products ship. Coverage numbers go up. Stakeholders are happy. Skip straight to the oven and it looks like progress. And when the pressure to generate goes up, invisible work is the first to get cut.
Testing was always a means. Coverage was always a proxy for that means. Teams can now produce both at scale and end up no closer to their actual goal. What doesn't get generated is the dough stage. That part's still on us.
Top comments (0)