When I was in university, I had a lecturer who didn't like unit tests. He was an eldery man who worked at IBM and gave lectures about mainframes. He always said, the hype about unit tests would simply double the amout of code written and do nothing for its safety.
When I did my first big project in 2009, a HTTP API, nobody in the company I worked for (the company was founded in 2001) had written any unit tests. They had huge C/C++ and PHP code-bases. They did integration tests, but the project I had been given was the first that used unit-tests.
I heard about it at university and wanted to make my first project look good right from the start. So I wrote a bunch of unit tests for every class, ending up with about 200 tests after the first version was released. Trying to hit that famous 100% coverage. Only a few months later the architecture of the API changed and somebody new at the project hat to rewrite more than 100 tests.
In the lifetime of the project, the unit tests didn't prevent any major bugs, just the stuff I had in mind while testing the code, but they slowed down the development progress tremendously. Also, they forced some style on the code that was mostly there to ease the writing of the tests and not the resulting application.
So what is your opinion about this? Did I do unit tests wrong? Is there an alternative? Are integration tests (black- or grey-box) enough when automated? Is TDD a placebo? Are type-systems the way to go?
Lets discuss! :)
Oldest comments (47)
(I'm a unit-test-addict :) )
Did I do unit tests wrong?
In my opinion unit-tests are documentation, so if your product change, your unit-tests must be rewrited. If you had to rewrite to many tests for a little change so maybe you should make your tests more flexible, or use them only to test the "freezed part of your code" (utils functions and algorithms).
Is there an alternative?
In case of API (constantly evolving) some tools create test directly from spec (Swagger maybe).
Are integration tests enough?
It's difficult to test only a function with integration test, the scope is not the same. But testing "GET .../user/1" return the good object it could be ok. I higly recommand to use unit-test to deal with user-inputs (POST requests) because you can test the endpoint with a lot of bad entry (and check for security, malformed, bad type, ...)
Is TDD a placebo?
Personnaly it's a security I love to have :)
> maybe you should make your tests more flexible
How? :)
> or use them only to test the "freezed part of your code"
Isn't this against the TDD philosophy?
> I higly recommand to use unit-test to deal with user-inputs
How does this eliminate the problem that I only test what I had in mind anyway when writing the functionality in the first place?
Like, when I test my software I find fewer bugs than when someone else tests it. etc.
More flexibility:
It's recommanded to test only one case per test, I get the bad practice to put all my testing case into an array:
for (arg1, arg2, result) in [(1,2,3),(-1,-3,-4)]:
assert(my_sum_function(arg1, arg2) == 3)
It's bad but you can make a lot of case and change function name easily.
Maintains few tests is always better to have no test at all. To encourage your team adding tests it should be easy ;). So test the freezed functions is a good start.
I'ld love to write an article about "unexpected testing cases", I have this list of error cases:
The main thing with TDD from my understanding is that tests are the requirements, so anything that falls outside of the tests is by definition irrelevant. Most of the "test everything" recommendations come from the TDD mindset, so if you try to apply that outside of the TDD framework it can get messy.
This perspective helps limit the scope and coupling of your tests, since there is typically an astronomical number of tests that you could do, but a very finite number of testable requirements. Refactoring should not generally break tests, but if refactoring occurs across/between several modules then you will probably have some rework, but I would argue that that is more of a "redesign" than a "refactor".
One good reason to test every module/class is to reduce the scope of any bugs you do come across. If I have a suite of tests that demonstrate my module's behavior then I know where not to look for the bug. With integration/system tests alone you will have some searching to do.
I always have the feeling that is still a problem.
I get rather high leven requirements, but they are implemented by many parts of the code. So simply writing a "Req1 passes" would require to implement many many thigns till the requirement is met.
One of the important things that unit tests will do is to get you focused on SOLID, most notably single responsibility. It reduces the temptation to write "Swiss army knife" functions or massive blocks of if..else or switch..case code. When you work in short blocks of testable code it makes debugging so much easier. Likewise, if you find tests becoming elaborate, maybe some refactoring is needed.
When you're working on a team, having the unit test gives other developers a guide as to how a particular function should work. If they come up with use cases you didn't anticipate, it provides an easy way for them to communicate it. When you're primarily working on the backend, it gives you something to demo in the sprint retrospective/demo.
When debugging issues unit tests make it easier to locate problem areas both in integration testing and in production. Without having this testing you can spin your wheels trying to find bugs.
Alternatives to unit tests? I've had to do these when working with legacy code where there were no tests originally written. Usually, these tests were in the form of one-off sandbox applications that would exercise a particular function or set of functions, trying to track down a bug. I've found this to be more inefficient than writing tests to begin with, particularly when trying to deal with critical production problems.
Probably yes (sorry!)
It's a classic issue of writing tests that are too coupled to implementation detail. People then get frustrated at tests because they can no longer refactor without changing everything
I'm going to speak in terms of TDD; and that does not prescribe writing tests for every class/function/whatever. It prescribes it for every behaviour. So you may write a thing that does X, internally it may have a few collaborators; don't make the mistake of writing tests for implementation detail. These are still unit tests.
Ask yourself. If I were to refactor this code, would I have to change lots of tests? The very definition of refactoring is changing the code without changing behaviour. So in theory you should be able to refactor without changing tests.
I would suggest looking into Kent Beck's book on test driven development. It's an easy read and quite short. Or if you like Go and dont want to pay any money have a look at my book. This video covers some of the main issues you talked about and probably explains what i've typed a lot better infoq.com/presentations/tdd-original
Writing tests effectively takes a while to get proficient at, but the fastest way to get there is to study and retrospect the effect tests had on your codebase
Thanks, I'm going to read this book :D
Adding to test the code's behavior, test that the code implements requirements: those things the end user, legal, marketing has to have. Then you get into tracing requirements to exact lines of code, and anything else can get deleted.
My 2¢ about this discussion.
First of all, I think that this quote is fundamental to understand why we test our code:
“Testing shows the presence, not the absence of bugs” ~ E. W. Dijkstra
It means that our tests can't prove the correctness of our code, they can only prove that our code is safe against the bugs that we are looking for.
Having 100% code coverage doesn't guarantee that our code is 100% correct and bug-free.
It only means that our code is 100% safe against the bugs that we are looking for.
There may be bugs we aren't looking for even with a 100% code coverage passing tests.
Tests show the presence, not the absence of bugs.
Chris James says: "the very definition of refactoring is changing the code without changing behavior."
The behavior refactoring refers to is external behavior, that is, the expected outcome of a piece of code, not how the code behaves internally.
When we write a test, we can make assertions about internal behavior but it can change without modifying the expected output.
That's the very definition of refactoring.
When we make assertions about the internal behavior, we are coupling our test to an implementation: internal behavior changes will likely bring to change the test.
That's why I like what Michał T. says: "code that is perfectly suited for unit tests are things that have predictable inputs and outputs, and don't have dependencies or global effects."
The assertions about the behavior of our code will likely depend on the behavior of our dependencies.
Indeed, we mock external dependencies because we don't want our code being affected by their potentially bugged outcome.
Thus, we set up our environment to have a predictable output.
That's why even if external dependencies have bugs, our unit tests can pass. And that's why unit tests aren't enough to save us from having issues.
Reducing external dependencies will make our code easier to test and less prone to side effects coming from the outside.
My last thought, starting with this quote from connectionist: "code changes happen all the time and unit tests have to change with them. It's unpleasant but necessary."
Software, by definition, is soft to adapt to changes.
Otherwise, it would have been "hard" ware.
We have to deal with it. It should not be unpleasant but the opposite: it's its the ability to change that proves the real value of software.
The frustration that we feel when we have to change our software comes from the fact that as long as we add code we tend to reduce the flexibility of our software (we add accidental complication).
Thus, adapting to changes becomes frustrating.
But it's not software's fault.
It's not our customers' fault.
It's our fault.
It's only by making our code better over time that we can reduce that frustration.
And we can make it better by performing refactoring on a regular basis.
Everything that encourages refactoring should be welcome.
I warmly recommend watching this: vimeo.com/78898380
Cheers
I haven't written any unit tests for Web APIs yet but here's my take on TDD:
In my part, I don't recommend writing unit tests for every class. Only for classes that changes behavior based on various arguments and conditions.
Writing unit tests helps me in various ways:
It validates my understanding of the requirement. There's a tendency for us developers to jump right into coding without fully grasping the requirement. Writing unit tests forces us to think and ask questions even before the actual coding. Which eventually saves us more time than rewriting code from previous assumptions.
It helps me make design decisions. That is, if a class is hard to test, it may still be broken down into smaller testable classes. Therefore, enforcing SRP (Single Responsibility Principle)
Acts as harness after refactoring and bug fixing. Tests should still be green after code changes. It's a quality layer that signals me that I didn't break anything.
Like @JeffD said, also a documentation. I've written and deleted a lot of unit tests. Requirements may or may not change in the future. You don't know when or if it will but for this time that it's true, its better to write unit tests than to write none in anticipation that it will just be deleted or changed in the future.
Hopefully, these insights helped you.
You're probably right.
I often read unit tests of libraries I used to understand them, but on the other hand I don't write libraries myself. They feel like they would lend themselves rather well to unit-testing, like APIs and such. UIs feel different somehow.
From my experience unit tests are incredibly useful when developing code that is perfectly suited for unit tests, generally things that have predictable inputs and outputs, and don't have dependencies or global effects. On the other hand if you're testing boilerplate code with a lot of complex dependencies (i.e. an MVC controller) it's probably better to cover it with integration or acceptance tests.
You should move as much code as reasonably possible into unit-testable blocks, but going out of your way for 100% unit-test coverage leads to tests that aren't worth writing and updating.
Then there are tricks, like mocking outside services (so that you don't have to actually hit remote services when running acceptance tests) and comparision testing, i.e. not testing the contents of an XML document but just storing it and comparing output directly to it. When testing APIs I also automatically test inputs and outputs on endpoints against a specification, which is a pretty good way of testing both the endpoints and the specification.
I also think that unit tests are a great way to force yourself to write easily testable code, which is usually better structured than non-testable code :)
But in general code needs to be tested if you care about it working. Any endpoint you don't test will eventually be broken.
Using TDD does change the way I write my code but I feel like it improves the code in terms of readability and maintainability (probably performance too but I've not tested this myself) by ensuring I use pure functions and ensuring there are as few side effects as possible.
For projects that are going to be maintained long term (more than a couple of months) I find unit tests to be super useful. If it's a very simple or short lifespan project I agree that they add needless complexity to a project.
Interesting point with regards to only being able test things you plan for, I guess this comes back to "devs shouldn't test their own code" and I'm not sure how we could improve this situation other than let QAs write some test cases too which is obviously not suitable for every business.
Not sure on alternatives really, I guess it depends on your situation and the project at hand but I don't think every project should use unit tests for the sake of using unit tests.
Did I do unit tests wrong?
I can't say for sure, but what I can say if "Trying to hit that famous 100% coverage" is a nothing but a wild goose chase. To find out why, see this article: dev.to/conectionist/why-code-cover...
Is there an alternative?
Code changes happen all the time and unit tests have to change with them.
It's unpleasant but necessary.
However, if a large part of your architecture has to change (and this happens quickly/frequently) then the problem is not with your unit tests.
It's with the architects and the faulty/rushed decisions they make when deciding upon an unstable/unreliable architecture.
Are integration tests (black- or grey-box) enough when automated?
NO!
Unit tests and integration tests serve different purposes. They are complementary. They are not meant to be a substitute for one another.
Unit tests are meant to test code. They are like a defense mechanism against yourself (or, more specifically, against accidental mistakes you might make).
The idea is the following:
Because it's possible that changes you make in some places, have undesired effects in other places. That's where unit tests come in. They tell you "No, no! If you continue with these changes, you will break something that was working well. Back to the drawing board!"
Integration tests on the other hand test functionality. They check if everything works ok when it's all put together.
Is TDD a placebo?
Certainly not. But like all things, it works only if used properly.
As a side note, don't be discouraged if your unit tests didn't catch any major bugs. That's very good! That means your a good programmer who writes very good code.
If your unit tests failed every time you ran them, it would mean you're very careless (or in love and with your head somewhere else :)) )
Think of it this way:
If you hire a security guard and you have no break-ins, are you upset that you have no break-ins?
You're probably feel that you're paying the security guard for nothing.
But trust me, if he wasn't there, you'd have more break-ins that you'd like.
Yes, I guess that's the problem.
After a few years of practice you write code that is pretty robust and the tests you write basically do nothing until the first changes to the software happen :)
I've been a developer for over 20 years and have never once written tests in a real project
Tell me your secret!
No secret. I just don't use automated tests
What language are you programming in?
In a professional capacity, I've worked with Visual Basic (back in the day for some Desktop apps), PHP, JavaScript, and Ruby
Nice.
How do you assure code quality?
If by code quality, you mean functioning code... manual testing. It's always worked for me
My take on unit tests is to avoid them. Write your software ina way it could be tested easily, as this will keep your code decoupled, will force you to explicitly inject external stuff and more.
But if you have a decent type system and a bunch of integration/end-to-end tests, unit tests are not worth the hazzle.
After all you dont care about implementation details as long as your module/component/insert-similar-here does the correct thing
To me, you should treat tests like features and features like wizards. As Gandalf has said, "A wizard is never late, nor is he early, he arrives precisely when he means to." To me, 100% test is always too early. By writing tests to 100% completion, you are saying that your features are 100% done, and that your product, in turn, is 100% complete. So, when that requirement came in, there was no room for it, thus forcing a rewrite of the system to accommodate it. Tests aren't just an assertion that everything is complete, but a measure of how much work the product needs.
Another way to think about it, is using the same quote, but focus on the last bit. "...he arrives precisely when he means to." Rather than testing what you say (code), you test what you mean (intent/behavior/requirement). Sometimes, we developers only know code, we don't know the requirements. If that happens, then any tests that we create may be worthless, as they do not express what was intended. Due to lack of communication, you did not anticipate a new feature, thus creating new work. Some may argue that dependency injection would have solved this, but unless you apply that to littlest model, there will be some way that this will get you. This is why agile was about building smaller and communicating faster.
I like to think about tests in a different context than TDD. Rather than testing to mean asserting, I like testing to mean trying it on, sort of like shoes. If I like the result, I will lock it in. This idea harkens back to when we started programming. Code a bit, complain about a missing semicolon, compile it, play with it to see if it works, repeat. With this same idea, we just gain two things: it is automated and no CLI required to input. This will bring a different mindset into testing. Rather than building the test first or last, it is with the feature. Code a bit of the implementation. Code a bit of the test. Code a bit more of the implementation, code a bit more of the test. Rather than the test being something to assert against, it becomes an explanation of your intent. This is what it means for a test to become your documentation. Of course, that should not be your only form of documentation. Just because it passed the unit test, does not mean it is correct behavior.