"The terms 'unit test' and 'integration test' have always been rather murky, even by the slippery standards of most software terminology."
-- Mart...
For further actions, you may consider blocking this person and/or reporting abuse
This is an interesting article, that covers a lot of ground -- which is great for people who are new to the pros and cons of unit (and other) testing.
But it also buys into several myths that can blind us to the value of testing.
MYTH #1. Tests should only call publicly available methods (functions, whatever). A la Kent Beck's "only entry points to externally-visible system behaviours define units".
Balderdash! The whole private/public/visible debate is arguing about how many angels dance on the head of a pin! The meaning of private or public depends ENTIRELY on the context. Just because some ancient languages (e.g. Java, which I love) have crude granularity for privacy levels, doesn't mean that's how reality works!
I used to build robots to inspect jet aircraft engines. We don't test aircraft engines by flying the entire plane! And yet the engines should certainly not be "public" to, say, the passengers! We remove the engine and put it in a special test-bed. The engine is "private" to the plane, but "public" to the test-bed.
E.g. Michael Feathers noted that in, in the public-vs-private debate, often a particular set of "private" methods (say that we really, really, want to be able to test) is really a class looking to be extracted into a library of 'public' methods.
We build software in layers... like onions (tears included). The innermost layers need be visible to the next layer out, and so on and so forth. Someday our language implementors will realize this, and we'll have more things like C++'s "friend" classes.
"We build software in layers... like onions (tears included)."
I think that's a reasonably good approach.
The product I work on is not written like that.
It's written like a big ball of mud. (The mud is probably an admixture that includes tears.) That is, in part, a byproduct of having a code base that is almost 4 decades old. And which (at least in great part) has evolved organically.
My programming language is C++.
It does not have unit tests as part of the core language. So there are many testing frameworks to choose from, like Google Test, Boost Test, Catch2, (my favorite) Doctest, and many others. I wish C++ had unit tests as part of the core language. Using a test framework causes lock-in, which is unfortunate.
My product does not use a testing framework. It uses a dozen testing frameworks. Each of those testing frameworks are not redundant. Each are used for a particular aspect of the product. Some of those testing frameworks are focused on testing the system, some are focused on testing functionality, and some are for testing APIs. A couple are used for performance testing in a very narrow domain. The ones that could be focused on unit testing have been co-opted to be used for integration testing.
I also wish that C++ had design by contract as part of the core language. If the C++ language had contracts, that would obviate the a good part of the need for unit tests. Not all unit tests, just the where unit tests are used to express contracts.
Alas and alack. C++ core language lack unit tests. And C++ core language lacks contracts.
(The context of Coplien's "most unit testing is waste" is about maintaining and running unit tests for code that is pretty much settled. It isn't against writing unit tests in the first place. TDD-style unit tests serve as a forcing function upon the programmer, which in turn makes their code abide by OO principles such as SOLID, or DRY/WET, or YAGNI, or GRASP, or KISS, et al. All of those OO principles were discovered in the OO era to shore up deficiencies in OO. Maybe if/when we move from OO era to an FP era or a DLS era, we can dispense with those principles. But we'll probably discover FP or DLS deficiencies, and discover new principles to shore up FP deficiencies, or DSL deficiencies. To be determined. There is no silver bullet.)
Eljay, could you say more about what you mean by "unit tests as part of the core language"?
I'm not aware of ANY language that has unit-tests as part of the "core". E.g. Java, where sometimes we have to fight with the poor granularity of public/private, in order to make useful unit-tests.
But I would sincerely love to hear more about what this means, or might mean, to you (and others).
The D programming language has unit testing as part of the core language.
It also has design by contract (precondition, postcondition, invariant) as part of the core language.
Fascinating. Thank you!
Great analogy! Thanks for your insight, Charles.
That's a great article.
By the way, when someone submits a regular expression in a PR, to avoid the "now you have two problems" issue, I have a rule that you must have a (fast blackbox) unit test for the regexp.
Excellent job on this article! Sets the record straight on doing a lot of irrelevant work in the name diligently "unit" testing all the things, and I love the premise that we should classify tests as merely white box or black box instead, and that definitely black box tests should be preferred. Good job. 🙂👍
There is only one thing I want to contest here.
This is often repeated by people who insist on TDD meaning "test first", and I don't buy it.
Writing the test first only "proves that the test would fail" in the same sense that running your program and looking at the output proves that the program works - you are essentially arguing in favor of "hands-on testing" this fact about your test suite.
Conducting a hands-on test by running your program only proves that this momentary version of the program worked - similarly, running your test (which is running a program) only proves that your test can fail for this exact momentary version of your program.
Just as running a program does not guarantee it will keep working, running your test and watching it fail does not guarantee that the test can still catch bugs in future versions of the program and/or test.
This is after all why we write automated tests in the first place.
The red/green approach can give you a sense of confidence, and probably increases the odds of writing a test that can catch bugs, but hands-on testing a test provides no guarantee of that. (If you want that guarantee, look to mutation testing.)
This feels hostile towards developers who don't practice TDD.
Personally, I never write tests to satisfy a code coverage metric - I only use code coverage to highlight potential areas for testing, and I will never write a low value test to increase a metric.
I reject the assumption that tests "aren't written to catch bugs" because someone typed in the code in a different order than you did. That's silly.
Tests can definitely help me think through difficult implementations - as I'm writing the code, I am always thinking about whether this code will be easy to test. As the code takes shape, I am always concerned with the structure and form as it relates to testing.
My added/updated code and tests are always in the same commit, as I'm sure are yours - my process is different from yours, but you can't look at my commits and see any difference.
I have tried the test-first approach, and it simply does not work for me. For one, I prefer strongly typed languages, and often, if I were to write a test first, it wouldn't even compile or run. Fighting an IDE the whole way because it can't complete code or statically check anything, is simply not a productive use of time for me.
The assumption you make about tests "not written to catch bugs" or "not written to help a developer think" because I chose to write the code first and test second, it sounds a lot like the kind of doctrine you're actually trying to dispel with this article.
I think you should reconsider that position. It's not true.
Writing tests first or code first, or writing bits of code and bits of tests, or changing or refining those bits as you progress -- whatever your approach -- it's simply a matter of difference of thought process. Our minds don't all work the same, and this kind of doctrine isn't helpful to those of us for whom this approach does not work.
I definitely encourage people to try the test-first approach. It works great for some people, and that's wonderful for them. But if something else works better for you, and if the outcome is quality code and high value tests, you do you. ✌️
That's fair! Thanks for sharing your viewpoint! I am constantly trying to remember that there is no "right" or "wrong" way of doing just about anything when coding. It's just that my opinions have been shaped by my experiences, good or bad, as have everyone else's. Thanks for reminding me of that :)
Whenever we talk about testing strategies we always talk too little about code that is being tested. Different code architecture and different software peaces needs different approaches to testing. The part that some developers don't agree on what "unit" is thus isn't problem at all as long as we inform reader what "unit of code" means and looks like in our testing strategy that suggests certain % of unit tests.
If there is any single definition of a “unit” of code, it’s what Kent Beck described above. But, like you say, there are differently-sized units for different kinds of programs. But most discourse on the internet treats it as a standard chunk of code (usually a single function). I think this notion is harmful.
MYTH #2. Tests should (or should not)... insert your opinion here.
Also balderdash! The primary purpose of tests is to help write (and maintain) WORKING CODE. Everything else is about context... or religion.
And that's why Reflection mechanism exists, for example PHP Reflection allows to "unlock" the method so it can be called even if private/protected.
So the method visibility is not a blocker for Unit Testing
But I think this tells you that you are testing at too fine-grained a level. If a method is invisible from “the outside”, it is not a “unit”, according to Beck’s definition. I think tests like this will make it harder for you to refactor that code.
i don't agree, the visibility of a method is a security mean not a level of granularity.
Assuming we are running unit tests on our own code : there could be places where methods are private, still they need to be tested.
MYTH #3. There's a right answer to all of these questions.
Wrong. EVERY engineering decision has pros and cons. If you can't articulate them, then you're not being honest. All of the issues described in the original article occur on a spectrum. For any particular context (size of project, size of team, expected lifetime of project, etc., etc.) there is a "sweet spot" on the spectrum.
E.g. To do TDD or "development-informed" ("DI")? It's not an either-or question! Which approach, in context, produces better code? I was the unit-test evangelist for a project with 1M LOC and 20 developers... and under my nudging/wrist-slapping/rewarding, we went from 40% coverage to 75% converage. We used TDD, DI, and what I call TND (Test Near Development: sometimes before, sometimes after, sometimes during)... and we had 20 THOUSAND tests.
They saved our bacon many, many, times.
I think this is the most important thing to impress on new developers: "it depends". I read somewhere recently that newbies tend to have a mindset that there is a single "correct" implementation, but as you gain experience, you learn that everything is negotiable. This is what I was trying (poorly, apparently) to get across in my original article: there is no such thing as a "unit test", a single definition which can be used to categorise tests into "unit" vs. "not unit". Rather, we should look at different aspects of tests, like their intention, what kind of testing style they use, etc. You've raised lots of great points, particularly that the things in my blog post are not to be taken as gospel, either.
Question everything!
In my casse I test php application that storing data correctly upon db is crucial. What I do is I mock and REST/XMLRPC API calls and any 3rd party service EXCEPT db.
For that upon each test I make a dedicated test db.
Hi @awwsmm ,
Thank you for your insightful article on the complexities of unit testing! Your perspective on the definitions and implications of unit tests versus integration tests is refreshing and thought-provoking. I particularly appreciated your emphasis on the importance of black-box testing and the distinction between development-informed and development-informing tests.
This has given me a lot to consider regarding my own testing practices and how to improve the maintainability and effectiveness of my test suites. I look forward to applying these concepts in my work!
Good work, great article!
I also dislike the "traditional" definitions of unit/integration/whatever tests.
I mean if you're completely new to testing or software development at all, in theory that probably makes sense. But as soon as you get more experienced writing tests, you'll quickly realize that you cannot simply split your tests into those categories. (Maybe you can, but it wouldn't always make sense).
The fewest components are "units" and can be tested isolated. And just because some component has a dependency, it doesn't make the test for it automatically an integration test. When replacing the dependency with a fake implementation I can still run the test as part of my (unit) test suite without any issues.
I struggled in the past to find good names for these different kind of tests, but your idea of the new testing pyramid is really good and it makes perfect sense - thank you for that!
Flagging this as a High quality article because it most certainly is!
Damn, really cool article. Categorizing things is always a problem in software engineering, but, cool to see we're revisiting points that may not make sense anymore.
Great insights
Really amazing article.. I cannot be more agree. thanks!
I thought I saw the title of the article like this: "There is No Such Thing as a Unit Test in my Project"
😂
tes