I popped into an online panel about testing today, and a question along these lines was asked: what is the ratio between automated testing and exploratory testing at your company?
I get the gist of what is being asked here of course, and wouldn’t get too pedantic in answering it in the moment, but this is my blog and I can get pedantic if I want. The interesting thing to me about this question is that even confining it to a single company, single team, or single product, there are many different ways to answer it. The crucial point is that for you to measure a ratio, you need to attach the same units to both quantities.
One of my pet peeves is that tests aren’t countable things. Even strictly scripted manual tests usually have relatively arbitrary boundaries. Automated tests, especially UI tests, can typically be refactored in ways that change the number your tool reports at the end but contain the same set of assertions. (Even if you’re strict about saying one test has one explicit assertion, there are implicit assertions everywhere. Some tests might have 0 explicit assertions!) Even if you could count tests, how do you count exploratory tests in any comparable way? Each session? Each charter? Certainly not per assertion because a human will make a hundred different “assertions” in their head every minute. No, counting tests can’t be done.
Even asking about relative coverage of the two approaches requires some underlying quantifiable thing to cover. I generally work in an environment where implicit requirements far outnumber anything explicitly written down. You may have some set of user stories, BDD scenarios, or other itemized list you can go through, but it’s unlikely many of them fall squarely into automated vs exploratory buckets. (I’ve rarely automated tests for features I didn’t also explore myself.) Can you even say a user story is 80% tested with automated tests and 20% with exploratory? 20% of what?
Maybe you can try to use time spent. I should be able to roughly say how much time I spent exploratory testing, at least if I’m good about time boxing and not getting distracted too much. How do you measure the time of automated testing? Should I could how long I spent writing test automation, or running it, or analyzing the results? Let’s say, I typically spend 1 hour writing code and 2 hours doing an exploratory session, so the ratio would then be 1:2. Does that ratio tell me anything interesting?
One thing it certainly doesn’t tell me is how much of my testing work is done by automation, or the relative importance of those two activities. In that one hour I might have added a handful of scenarios to an enormous test suite that takes 8 hours to run. I don’t personally spend 8 hours on running that automation; I can do other things while the computer does that work. In that sense, investing more time into automation can actually decreases the automation-to-exploratory ratio. If automated tests are already quire thorough and robust, those tests alone could be enough to release and the exploratory work is gravy. At the other extreme, the automated tests might still be too immature to tell me much useful yet and the exploratory tests are the crucial part.
You could, perhaps, get into questions of how much time automated testing saves you compared to doing all of that manually. (Let’s hope that you would never actually try such a thing.) But then, we’re starting to talk more about automated vs scripted manual, and on how you might measure ROI of automation. Both are totally different beasts.
In reality, I expect most people would answer the question with some ratio of “human time” spent, and I’m thinking about it too much. Nobody asked for units.
The question is really asking about strategy: How do you balance these two activities to be most effective? There’s no one way to objectively measure that, as it depends on what you care about. Are you asking how much time should be spent on automation? Are you asking about relative importance? Are you asking about what can be automated or what should be explored? Are you just trying to plan how many people you need on a project? Your priorities will determine which way to quantify these things.
Top comments (0)