How I failed at Test-Driven Development and what it took to get it right

For more content like this subscribe to the ShiftMag newsletter.

I’ve always liked the idea of test-driven development, and it’s three simple steps – write a test, make it pass, and refactor – but it took some time before I could say that TDD really works (for me).

I first heard about it over 20 years ago, probably when somebody talked about Kent Beck’s Extreme Programming:

In XP, when possible, tests are written in advance of implementation.

If I am programming test-first, I write one failing test, make it work, and then write the next failing test.

In Growing Object-Oriented Software, Guided by Tests, Steve Freeman and Nat Pryce further define it as follows:

The cycle at the heart of TDD is: write a test; write some code to get it working; refactor the code to be as simple an implementation of the tested features as possible. Repeat.

Using TDD should result in significantly fewer production bugs by having code that works as expected (in tests) and with better design and higher quality of the code (“easier to change”). The code should be modular, loosely coupled, cohesive, separate concerns well , and provide good encapsulation.

Dave Farley made it almost a haiku:

The goal of TDD is:

Well Designed Code,

That We Know Works!

A luke-warm introduction to TDD

Initially, I listened to and learned about TDD through conference sessions, online tutorials, and books, as well as trying it out in my (test) projects. I always got the “green bar,” but I didn’t think my code was significantly better than before , with tests written after the code.

One notable difference was that the code was testable (or at least more testable than before). But I didn’t see all the benefits I’ve read about.

Nevertheless, many authors, presenters, and software engineers I liked and learned other things from, and whose frameworks and libraries I used, kept talking about TDD and the many benefits they reaped from it.

So I kept at it, but every time, it didn’t feel like I was doing it correctly, and I definitely didn’t think I got the “advertised” results.

My mistake: Taking too big of a bite

Later, I started working on complex information systems, implementing complex business rules with a relatively small and stable team, where the developers were domain experts.

The ever-growing and ever-changing business rules made me more appreciative of a codebase that was easy to change , but that too often wasn’t the case with (my) code. At the time, we were writing some unit tests, more functional tests (actual use-case tests), and doing a decent continuous integration on most applications.

I refactored code daily to make it more readable and understandable. I mostly used rename/extract variable/method/class, and the code was better each time, but again, it wasn’t as easy to change as I would have liked it. The problem was that my object design wasn’t the best , and I tried to find a way to improve it. I also often tested implementation instead of functionality.

What helped bring me closer to TDD and up my object design Domain-Driven Design (DDD). For me, the largest breakthrough came from internalizing the concepts of bounded contexts and ubiquitous language, but Entities, Value Objects, and Service Objects, including Aggregates, helped me with object design.

When I was writing code (in Java), influenced by DDD, the code was usually easier to understand , but the tests I wrote after the code were too hard to write and maintain. So, again, I tried TDD. Once again, my code was better, not to the full extent TDD enthusiasts raved about.

I later understood that my biggest mistake was that I wasn’t working in small steps, starting with the simplest (“boring”) use cases and building up to the more complex ones. The other mistake was that I didn’t refactor the code to “Well Designed Code,” so the methods and classes were too complex and coupled. The tests were simply too complex to set up and maintain.

And I’m not the only one who made that mistake, according to Dave Farley:

Over-complicating the solution is one of the commonest mistakes that I see TDD beginners make. They try to capture too much in one step. They prefer to have fewer more complex tests than many, small, simple tests that prod and probe at the behaviour of their system. The small steps, in thinking and in code, help a lot. Don’t be afraid of many small simple tests.

Go TDD or go home

In the end, the circumstances pushed me all the way to TDD. I found myself in a situation where I had to maintain and further develop an application that processed billions of messages daily, had only a few tests, and had no access to developers who initially built it. It had complex and large classes and methods, scalability and reliability issues, and high hardware demands.

The only way to survive that was implementing much better code than I was doing before.

So, once again, I turned to books, videos, and courses about TDD, refactoring, creating APIs, and code quality to improve my object design and its implementation.

This time, I implemented many (small?) changes in my designing and coding way of doing TDD:

1. Working in small, really small, steps with a commit after each step (many 10s per day)
2. Stopping the primitive obsession
3. Refactoring until the API design is good (“You are the first user of the API”) and tests are easy to write, understand, and maintain (“Tests are trying to tell you something”)
4. Doing pair programming

I noticed significant changes in my code in a few months. It was simpler to understand, and the methods and classes were much shorter (but there were more classes). Adding new features wasn’t a challenge anymore, and other developers could help me in hours or, later, in minutes, without lengthy onboarding.

With Golden Master tests and using TDD for new or changed and refactored code, soon there were enough tests and, as a result, confidence that changes to the code won’t carry any significant risk.

In that short time, using TDD daily, I got very predictive and almost constant progress in the right direction with my coding, and I could provide value to my users better and faster.

It wasn’t TDD; it was me

In retrospect, I think that my problem with getting better code with TDD was in my design and coding skills, not in TDD. Not that they were particularly bad – I did manage to create and maintain large, complex, and successful applications/systems.

But I missed some (or many?) details that make TDD something people rave about. If you have the chance, find someone good at TDD and pair program with them for a few days to see if it works for you.

What was your turning point in using TDD and getting good results? Please share your experience with me on Twitter or LinkedIn!

The post How I failed at Test-Driven Development and what it took to get it right appeared first on ShiftMag.