Every person that has worked as a professional quality assurance engineer has, at one point, had a conversation about test cases. How to organize them, what they should entail, who should write them, etc.
Test cases seem to be one of those things that every company knows they should be doing, but in the end, they have no idea why. When asked, the de facto answer is something along the lines of:
“We need to document test cases so that our future automation efforts know what steps to take.”
While this sounds like a good response, it really isn’t. It’s a response that, in reality, says two (bad) things. The first is: “We’re not doing automation now, and have no idea if we will, or what we want from it, so these test cases will probably get relegated to documentation hell.” The second is: “We heard a qa professional on youtube talk about test case management, so we need to have test cases.”
This might sound harsh, but further examination should make the point clear. To start, what are test cases?
A test case is a series of steps that a user can follow to complete a desired outcome. In other words, it’s a set of step by step instructions for a user to follow, where each step has the expected outcome. If the user encounters an outcome that doesn’t match the prescribed results, the test case fails, and a bug report is written up. It sounds simple. Any quality assurance engineer that has spent time writing test cases will tell you, it really isn’t that straight forward.
Preconditions:
Preconditions are the bane of any test case writer. When writing test cases, there has to be an agreed upon starting point:
Does the user already have an account?
Has the user already logged in?
What type of user is this test for?
It might look like a simple problem, but this choice will dictate how every test case is written, and any exceptions to the agreed upon starting point will need to be explicitly written out. It’s also just the tip of the precondition iceberg. The real issues happen when talking about dependent test cases.
In other words: Test X has to pass to even try to validate tests Y and Z. So now whatever software is being used to track the test cases must be able to handle preconditions as an option on the tests. Again, it sounds simple, but what about tests A, B and C that depend on Y? Or test D which depends on X and B? What about when the first 5 steps of C are the same as all of test A? Should A be a precondition of C? Or should C just have 5 more steps?
Did you follow all of that? Most test case management software can’t either, so don’t feel too bad.
Step breakdown:
If all of the issues of preconditions are solved, teams have to decide how to break out each step? Is a test a single step? Is each step its own action with its own result? How many steps in a test case is too many?
This is a pretty small problem, like a toddler building a block tower, every new block added threatens to bring the whole tower down.
Test Case Language:
This might seem completely unrelated, but it’s not, so bear with me.
Test cases are a form of documentation, and just like any documentation, it’s not written for the purpose of one that wrote it. The overarching idea is that somebody, somewhere, down the line is going to need to figure out exactly what the software they’re working on should do, and the test cases (ideally) will be the go-to documentation for getting that answer.
In order to maximize the efficiency of that discovery, each test case should use language that describes each component the same way throughout every test case.
If a test case uses the word ‘table’ and the word ‘list’, every other test case should refer to those components by the same name. Switching them, or introducing a new word will only lead to confusion and frustration (always assume the documentation writer won’t be around to explain things).
This could just be me being a stickler, but I’ve read enough documentation that throws out synonyms willy-nilly to experience an unhealthy amount of frustration when trying to decipher these kinds of ‘helpful’ instructions.
Types of tests:
This is probably the worst part of test cases. The question of: Which test cases should you write?
It’s easy to just focus on happy path testing, you want to know how the application works after all, not how it doesn’t work. Any good qa engineer, however, will tell you that how the application handles errors is just as important as how it handles success. To illustrate, here are some test cases for a single, simple number field:
Can I enter a number into the field? (Happy path)
What’s the smallest number the number field allows? (.000001 is a number)
What’s the largest number the field allows?
Is there an error for numbers that are too large?
Is there an error for numbers that are too small?
Can letters be added to the input?
Do letters in the input show an error?
There’s a quick seven test cases for a single number input. Now think about the number of inputs your application has. How many test cases do you need to properly document how it works?
Is your head spinning yet?
The point
Again, each of these seems small on their own, but like the block tower mentioned above, they all build on each other and make it that much harder to balance. Sustaining this pattern for a few dozen use cases isn’t too bad, but your application probably has more than a dozen workflows. When there are ten qa engineers all writing test cases, whose job is it to make sure every one of them has stacked the blocks correctly?
At the end of the day the real question to ask from all of this though, is what’s the ROI? Test cases, and test case management is an expensive endeavor, and any business should be looking at the value a full set of test cases adds. So if the answer is: “someday we’ll turn these into automated tests,” it’s probably better to forgo the headache until “someday” turns into an actual day.
Conclusion
It might sound like I have no faith in test cases, and think of them as valueless, and you wouldn’t be blamed for thinking that. I have seen companies that manage to successfully balance all of these problems, and make the difficulties worth the effort. It is very tough, but when it works it’s enormously helpful. That issue I see is that the companies that make it work well don’t seem to be in the majority, because they fail to ask themselves the right questions.
Top comments (0)