If you haven't already, you should read Joel Spolsky's excelent article Five Worlds. To sum it up - great programmers sometimes come up with tips and best practices that make perfect sense in the area they work in, but are not very useful and maybe even harmful in other areas.
I believe unit testing is one of these best practices. When it comes to library development, for example, unit testing are great. In other areas their RoI is too low to be useful, and other kind of tests should be preferred.
In my opinion, automated tests should be an automation of manual tests. It usually easy to decide how to test something manually. For example:
These workflows are intuitive:
Since the manual testing strategy is so clear, the automated testing strategy should mimic it. Use unit tests for the library function and integration tests for the feature. Some people will insist on unit tests for the feature, but that has the exact same drawbacks of manually testing it with a custom main.
I second this!
When working with a mature framework, or using a good library, features should usually come in the form of extensions. The purest extensions are those that are almost entirely declarative. i.e. you are just picking what functionality offered by the framework to compose into your new feature. When a piece of code simply composes, or declares constants, there is nothing to unit test. There's no such thing (at a unit level) as declaring the wrong constant or composing the wrong functionality. The declarations should trivially match your requirements, and (though we may have our opinions) there are no wrong or right requirements. If you write unit tests to re-assert declarative requirements, you will just have to change those tests as the requirements change without ever really protecting the "correctness" of anything. Also, these extensions are usually the most sensitive thing to API changes, and can double your clean-up effort if you have a framework API update.
Of course there are usually logical utilities and functional bits added with feature extensions, but those can usually be tested in isolation of the declarative bits. Their functional bits can always be made into a local mini-library, which is again just composed into the final feature, locally testable, and ideally not sensitive to changes to the API that the feature is extending.
High level integration tests are what you need to guarantee that you've composed these features properly to produce the desired effect.
My guess from the OP stating that there were hundreds of tests to change on an API change is that he was either testing declarative bits, or didn't have declarative bits properly isolated.
We’re a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.