If you are curious or bewildered by the discussion about “testing vs. checking”, this post is for you. I read the article about testing versus checking by James Bach and Michael Bolton. At first I thought “so what?”. Then I thought it was some instance of dualism. Now I know better. And it even makes sense.
If you are in the testing community, you probably know the discussion about testing versus checking. Roughly speaking, the idea behind “testing vs. checking is that checking is a purely mechanical task, where in contrast, testing can only be performed by humans. Since the distinction was propagated by James Bach and Michael Bolton, it got pretty famous in the testing community.
When I first read about the distinction, I couldn’t wrap my head around it. I thought “so what, we still have to do both". Like, what is the benefit of the distinction, other than hairsplitting. What value does it add to the discussion? Calling things differently doesn’t change facts in the real world, so why would anyone bother? And I was not the only one who thought so:
Then I recognized that the distinction is all about what a machine can’t do and never will be able to do. Well, I know a few things about AI, and I don’t think there is anything that a machine will never be able to do. So I thought, this is some weird instance of dualism. Again, I was not the first one to think that. Nor the second one.
This is the point where I lost interest and shrugged it off as some weirdo tester eccentric thing. But when I was at TestBash, people I respected made a point about that distinction. So I went back to that article and thought “what did I miss”?
What lies at the heart of the discussion
Turns out I did indeed miss something.
In most companies where I have been, testing was regarded as “low value”. Testers were mostly students, or even worse, people that had failed as developers. They “clicked through the application”, usually following a given script, and manually checked what they were told to check. There is nothing challenging or respectable in the task the way it is performed by most companies. This usually also reflects the stand of those companies on the topic of software quality: as something that is not that important, something that can be added later if the time and budget allows it. These “testers are (mis-)used as “clicking machines”. And this is the sort of “manual automation that can easily be outsourced–and often is.
But there are people and companies that put the quality of the software at the center of their effort. There is a whole community of people who take pride in testing. People that take testing as a craft. If you do testing well, it is an original, creative, challenging task. A task that tries to understand the purpose of the product and tries to find ways in which the product can be misused or improved. These people are about as far from “clicking machines as you can get. Yet they are lumped together in the same bucket. As testers. That do testing. Like, you know, low value clicking and checking.
It took me some time to understand, but at the heart of this discussion of “testing vs. checking is the esteem of real testing. And the self-esteem of people who take pride in contributing to a high-quality end-product. So if you ever meet someone who is emotional about the distinction of testing vs. checking, that person probably sees himself as a real tester, someone that honors the craft. Pay him respect by making that distinction.
But it is even more: It also is about marketing. If you want to “sell testing and promote it, you need to make that distinction. (It is no coincidence the term was coined by consultants.) To make an analogy: The situation is that the current notion of testing is a bag full of sand and diamonds. And when people need to “sell testing (e.g. to management or customers), they have to sell testing as such. Of course nobody pays the full diamond price, if there is a lot of sand in the mix. So if you want to sell the diamonds, it is important to separate them from the sand. By differentiating, you make clear that testing is important and valuable and needs expertise.
Essentially, this is what this distinction is all about. Testing is diamonds – a valuable and important task. Checking on the other hand is a mere mechanical task with low value that you better automate.
Checking is just sand.
If you liked this post, I would be thankful if you clapped (as much as you want), and twittered or shared. Thank you!
Top comments (8)
Testing vs Checking is, imo, a false dichotomy.
I think Cem Kaner put it best in his writing called "The Insapience of Anti-Automationism"
"A test is checking to the extent that it is designed to produce results that can be compared, without ambiguity, to results anticipated by the test designer.
A test is not-checking to the extent that it can produce a result that is not anticipated by the test designer but is noticeable and informative to the person who interprets the test results."
I highly recommend reading the full article.
Roesslerj I think wrote a good explanation why there is value in the distinction.
Testing vs. Checking — so what?
roesslerj
Seriously it sounded as though the argument was that the distinction does not exist because both are equally valuable and checking is most common.
But I find that even in his examples of where checking is occurring misses the fact that his planning and executing those initial checks falls into testing and the continued checking fall outside testing.
James isn't against tools, they are important to testing. But creating a distinction is valuable as there is a different mindset to accomplishing both, along with an ability to identifying what each bring to the table and what is lost if one is ignored.
Great, thank's a lot for the pointer—will do!
Being still a somewhat green tester, this distinction was an eye opener for me when I first learned about it. I had thought that the low value clicking and checking was testing and that test engineering was the activity that required brainpower.
I've been trying to weave calling each activity correctly given this separation. Hopefully with enough folks doing it, we can make it commonplace. I had a discussion about this recently and a person in the conversation basically said that they felt like it's an unwinnable uphill battle. :(
Anyway, thank you for writing this! It was very well done and I enjoyed the window into your journey!
I also think the differentiation as such is an unwinnable uphill battle. And more so, because what ultimately convinced me why this is a sensible differentiation isn't even mentioned in the original article.
But I think test automation (yes, I know) actually has a potential to end that discussion, simply by automating all checks and leaving only high-value testing tasks (not checking) to the human.
Out of curiosity, how does one get involved in "checking"?
Should we rename Test Driven Development to Check Driven Development?
To make things even more complicated, a test can be automated using checks, but testing as an activity cannot be automated. So if you automate tests with checks to drive your development, that term still fits :-D