You Might Not Need Integration Tests

NOTE: This is an advanced topic and assumes knowledge of CI/CD, integration testing, types and a bit of devOps.

Why I questioned the existence of integration tests

Ah the classic integration test, giving you confidence that all the parts of your app work together in a way unit tests never can. They can take a long time to run and require complex tooling like localstack (in the case of AWS), but that doesn't matter right? All good things take time and the confidence of everything going green is worth it.

Maybe Not.

My idea on testing was challenged when I read this tweet:

What is going on here? Are my integration tests a waste of time???

How did we get here?

After having a good think for few days I finally understood the simple idea behind Brian's tweet:

Your testing strategy should include your operational strategy.

I had failed to take into account how operations could affect my testing, it was a shock and when I rounded up my fellow engineers it turns out none of them had either.

But why does it matter? Ops is Ops and testing is testing surely they have minimal impact on each other? Right?

Nope.

The success of your application is dependent not only on what you do in code and how it runs on your machine but also how it will work in production.

Testing strategies have remained solid for decades now, integration tests have existed since the dawn of programming and still continue to be useful. But Ops has been evolving, ideas like CI/CD have only recently taken off and not taking advantage of modern Ops would be a mistake when deciding on a testing strategy.

So what do we do?

Canary Deployments

... Prefer to invest in monitoring (and partial rollouts/rollbacks) when possible. - Brian

Modern Ops tooling gives you the option of doing canary releases which is an extension of red green deployments where you always deploy your changes onto a new server.

Canary deployments divert a small % (usually 10%) of users for a small amount of time (usually 15-60 min) to the new deployment.

During this switch your deployment service is receiving events from your monitoring tools and will "cancel" the switch if an error event is seen and all users will continue to use the currently deployed application.

It can be difficult to setup but luckily managed services like AWS give you canary deployments for free and can generally be configured in less than a day.

The result of canary deployments is that any issues with the new deployment are contained to only a few users with automatic rollback preventing any extra damage.

The cost of making mistakes in production has been greatly reduced with tools like canary deployments.

Type to safety

So making a mistake in production is much less of an issue than it used to be, but still how do you know your code works if you aren't writing any integration tests?

Types. If it compiles it works.

Let's look at some common examples of integrations and how types are able to give us enough confidence that our code will work in production.

3rd Party Api's

Chances are these APIs come with SDK's that are fully typed, meaning that if you give the right input and it passes type checking then it is safe to assume that it works.

There is actually nothing to test, you have called the SDK correctly and the rest is up to the 3rd party provider.

The response will also be typed so you know the exact shape of data that you get back from the API. If you try to use this data incorrectly you will have a compile time error.

Database interactions

Assuming your database methods are fully typed and tested in an internal or external module you should have no trouble just using the methods and types provided by that module.

If your database requires certain data to create a record then your types will catch that. If you can only update certain fields your types will catch that. If you are using the response from the database to do some logic and you are missing some data the types will also catch that.

I would say that for almost every database interaction besides complex data migrations and table creation types are more than sufficient to have confidence that it will work in production.

What about stuff that isn't CRUD?

So from above you end up with 0 unit and integration tests for CRUD and basic API usage. But what about business logic?

...Testing of functions pays for itself (when using property testing and no mocks)... - Brian

You should be able to write your business logic as functions that you can test separately without the need to mock anything.

Doing this you get access to more advanced testing options like property testing where you can generate a wide range of inputs (usually 100) and check that your business logic handles them in a fraction of a second. Your tests will alert you to any edge cases that you may have missed in your code providing extra value to traditional deep assertion unit tests. (I recommend a combination of both)

Why does this matter? Integration tests have worked for decades why should we change?

Think of it like this, you either spend time writing integration tests and running them on every deployment or just skip them and rely on your Ops and types.

There is a point where the time spent fixing that one prod error that occasionally makes it through is less than adding on 40, 60, 90+ minutes of waiting for your
integration tests to run before going to production on each commit.

Lets do some Math:

Traditional Workflow per commit:

Write code (10 min)
Write unit tests (10 min)
Write integration tests (10 min)

Deploy to a test environment (5 min)
Run your integration test (2N min)

Deploy to production (5 min)

Total: 40 + 2N mins per commit

My Suggested Workflow:

Write code (10 min)
Write unit and generative tests (15 min) (extra value here)

Deploy to production (5 min)

Total: 25 mins per commit

Let's say you end up with 100 commits, that would take 2,500 mins to develop without integration tests compared to a whopping 14,000 mins using integration tests. That's a factor of 5.6x.

Is the extra fraction of a % more confidence worth losing out on 5.6x speed?

Integration tests can also be difficult to setup and maintain, time that could be better spent on better monitoring in production.

Integration tests are still useful and maybe occasionally necessary, but consider their use carefully because you might not need integration testing for your webapp.

Thanks for reading,

Please feel free to comment, tweet or DM me with any thoughts or questions.

Thanks to @paulgrizzay, @someivorytower, @nhnhatquan, @puffnfresh and others for reviewing.