How to get functional testing right

There is no topic that has kept me learning more new things in software development than automated testing. Every team that I've worked with had discussions about testing:

• How much should testing influence the tested code itself?
• What should be tested?
• Is high test coverage important?
• How much mocking should be done?

I could easily continue this list.

As testing is a relevant topic for all software development, there should be recommended practices that many (most?) agree on. This seems to be the case with TDD for example, which is at least considered good practice by many (although not followed often). However, TDD doesn't tell you which kind of tests to start with.

For example, let's say you want to extend a todo list management web app so that users can choose an icon for their todo list. Do you start with an end-to-end (e2e) test? If so, do you need extra tests for the api, as this would be covered by the e2e tests? How would you test for http(s) error codes? Would you mock the business logic in such tests? Would you test the business logic behind the todo list api as part http(s) tests, or would you write extra unit tests?

A guideline that I have seen recommended quite a few times is called the "testing pyramid" (https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html), referring to a thin-on-top and thick-on-bottom pyramid-like distribution of tests. It is recommended to have few e2e tests, somewhat more integration tests, and lots of unit tests.

Then, there is the "testing trophy" (https://kentcdodds.com/blog/write-tests), proposing to rely mostly on integration tests instead of unit tests, as these tests would deliver more value for the work invested.

I find this advice problematic. For one, the reasons given for why these distributions would be considered good are very weak. While it is stated that e2e tests typically run slower than unit tests (as they cover much more code), there is no argument made for why having many more unit tests than integration and/or e2e tests is a good thing. What if the app you are building has very little business logic and lots of UI flow? Why have many unit tests then?

More importantly though, neither the "testing pyramid" nor the "testing trophy" tell you which tests to use when you implement a feature. After all, they are vizualization of metrics (e.g. "number of unit tests" vs. "number of integration tests") and can thus be only applied after a feature has been built. They aren't actionable at the level of detail that you need to implement a feature.

Therefore neither a testing pyramid or testing trophy answer the question: When I implement a feature, which functional tests should I write?

Instead of relying on gut feeling or "best practices" (of which in my experience there aren't many in testing), I have found that there are some guidelines that are actionable and easy to understand. I try to follow them (for web apps and http(s) apis), and they have been making deciding about what to test considerably easier for me.

Here they are:

• When you implement business logic or common helper functions, write tests that run a specific function of method in isolation. Mock everything else. Test that the result of the tested function or method conforms with your expectations. Use any unit test library here.
• When you implement a http(s) api, write tests that execute http(s) client requests. Mock everything else you need to make the tests robust and fast. Test that the http(s) responses you get conforms with your expectations. Use any unit test library here together with a http(s) request library.
• When you implement a client for a 3rd party service, write tests that execute calls to that 3rd party service. Check that the results match your expectations. Use any unit test library. Consider running these tests only nightly as they depend on the stability of the 3rd party service and may fail sometimes.
• When you implement a flow in a UI (the "happy path"), write a test that execute that exact flow, often called the "happy path". Mock 3rd party services and your own api when needed. Use a UI testing library. With Cypress and TestCafe there are finally some good ones available for web developers.
• When you implement new controls or forms in a UI, write tests that modify these controls or forms. Mock everything else (e.g. http(s) requests, navigation, 3rd party services) to reduce flake as much as possible. Use a UI testing library here as well.

Besides being actionable, these guidelines have some other advantages:

1. They can be used as a check list for code reviews. When you know which feature was implemented, you can check each item in the list against the code written. When you see code that falls into the categories above but is missing tests expected for that category, you can state explicitly what's missing.
2. They ensure that tests are written at the lowest abstraction level applicable to each category. This keeps the number of dependencies that are used in each test as low as possible, which helps keeping the tests fast and robust.
3. They go well with TDD-style development. When you are implementing a features and have decided on the architecture (which apis need to be changed, which 3rd party services to use etc.), then you can use the guidelines to decide which test to start with. The feature itself then follows naturally.
4. They avoid having to stick to the vague and overloaded terminology of "unit tests", "integration tests" and "e2e tests" and similar. Instead, you could use the terminology above like "business-logic-tests", "http-api-tests" or similar which would be much more clear regarding what is tested there. The fact that the test framework was about "unit tests" would then be an implementation detail.
5. They ensure a high coverage that doesn't miss out any large part of the code base. My experience is that they facilitate refactoring by reducing the amount of mocking and focusing on testing results instead of implementations.

I would be interested in how this kind of categorization of code applies to domains other than the web apps and http(s) apis used here. Also, this post focuses purely on functional testing, which from my experience is the most important type of testing for most apps. It would be interesting to extend this list to other kinds of testing such as performance and security.