I couldn't have said it better myself: "That's what most people get wrong".
It's the subtle difference between NAIVE code coverage (that tools show you) and REAL code coverage (that you can only deduce by using intuition... or very strong AI).
And that's why code coverage is not a reliable metric.
I fully agree with your observations about code coverage. Though, the testing approach behind it is a bit more precise and useful when being applied correctly.
What you mean with "naive" code coverage is called "statement coverage". It means counting how much code statements (lines of code) you have covered.
There are more precise metrics available like "decision coverage" which checks if you cover all possible outcomes of boolean decisions (true or false) of statements like "if" and "while". Or "condition coverage", which even looks into the boolean subexpressions. Etc.
The problem is, that most coverage tools are only able to check for statement coverage.
And, furthermore, for your email regex all this would not help since you do not use any boolean decisions here. But for that example you are simply facing the basic problem of test case selection. This has nothing to do with unit testing especially, you have that problem in all kinds of tests.
There is a "real" coverage which would mean to test all string (and non-string) inputs to your methods. But testing all cases is usually not feasible, so you need to make a selection.
Also here there are techniques available which at least help you to make a good selection of cases to test, like "equivalence partitioning", "boundary value analysis", "decision table testing", etc.
So you do not need to rely only on intuition to have good unit tests. And you are right, just writing some tests to achieve high statement coverage is definitely a very weak approach.
I agree with everything you said.
Especially the last sentence which is essentially the point I was trying to make when I wrote this article.
Thanks for sharing your thoughts.
P.S. "Intuition" was probably not the best choice of words. What I was trying to say is that it requires some degree of non-trivial thinking and is, therefore, not easy to write tools that help you in these situations.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I couldn't have said it better myself: "That's what most people get wrong".
It's the subtle difference between NAIVE code coverage (that tools show you) and REAL code coverage (that you can only deduce by using intuition... or very strong AI).
And that's why code coverage is not a reliable metric.
I fully agree with your observations about code coverage. Though, the testing approach behind it is a bit more precise and useful when being applied correctly.
What you mean with "naive" code coverage is called "statement coverage". It means counting how much code statements (lines of code) you have covered.
There are more precise metrics available like "decision coverage" which checks if you cover all possible outcomes of boolean decisions (true or false) of statements like "if" and "while". Or "condition coverage", which even looks into the boolean subexpressions. Etc.
The problem is, that most coverage tools are only able to check for statement coverage.
And, furthermore, for your email regex all this would not help since you do not use any boolean decisions here. But for that example you are simply facing the basic problem of test case selection. This has nothing to do with unit testing especially, you have that problem in all kinds of tests.
There is a "real" coverage which would mean to test all string (and non-string) inputs to your methods. But testing all cases is usually not feasible, so you need to make a selection.
Also here there are techniques available which at least help you to make a good selection of cases to test, like "equivalence partitioning", "boundary value analysis", "decision table testing", etc.
So you do not need to rely only on intuition to have good unit tests. And you are right, just writing some tests to achieve high statement coverage is definitely a very weak approach.
I agree with everything you said.
Especially the last sentence which is essentially the point I was trying to make when I wrote this article.
Thanks for sharing your thoughts.
P.S. "Intuition" was probably not the best choice of words. What I was trying to say is that it requires some degree of non-trivial thinking and is, therefore, not easy to write tools that help you in these situations.