Kick-start a Testing Strategy

Dylan Broadbridge — Tue, 16 Apr 2019 10:28:33 +0000

A common hurdle I see among developers is deciding on a testing strategy for the system they are working on.

Thankfully finding a good testing strategy involves the entire team, not just developers as it is everyone's responsibility that the system works as intended.

Kick it off

Gather everyone who has a stake in your technology together for a meeting, the purpose of which is to discover and understand the risks involved in your technology operations.

Get everyone to list off features of your technology onto post it notes, the more diverse the people present the more features you will discover as certain features are more relevant to different areas.

Once you have a list of features, draw the following chart on a whiteboard.

Get everyone to place the features where they think is appropriate. Since I have some experience in insurance I will use that as an example.

Here I have four features;

User can submit a claim: Users rarely submit claims and are able to call us if there is an issue with the online system.
User can buy a policy: Need this to make money but a user will only ever do it once and it is not the end of the world if it malfunctions.
User can be declined: Every user will interact with this feature before buying a policy, and the cost to the business for one malfunction of this feature can be catastrophic to the business.
User gets the correct price: Users need to get quotes that are reasonably priced, but this feature is about getting an accurate price which doesn't carry as much risk as a decline.

Once the features are understood by everyone draw two or more lines from top left to bottom right. These define how much risk it is to the business if something goes wrong with a feature.

Once everyone has agreed on where features sit within the zones, conclude the meeting.

Develop and Communicate the Technical Implementation

Gather all technical staff and develop a rough guide to testing different parts of the system based on the risk (risk-based testing). Following on from the previous example you may end up with something like this:

Feel free to have additional rows like trivial and critical to better classify the varying risks in your business.

In the example we have extra monitoring for success on the declines to let us know that the service was functioning as expected. We also manually test that the service was working before each release on pre-production to be 100% sure it is working before releasing.

Once the technical team has decided on a rough testing strategy they can present back to the rest of the team. Developers should take onboard any feedback and adapt.

Learn and Improve

It is highly unlikely the strategy is perfect and it is important to keep improving.

A couple ideas to improve your testing strategy are:

Whenever there is a bug in production assess its impact and if its significant adjust your testing strategy to prevent it from happening again.
Set metrics to measure the outcome of your testing strategies, an interesting metric I've heard of is escape defects which are issues in production that were not identified before reaching customers. If you are able to measure something like that accurately and weigh it by impact of the defect it can provide insight into how your testing is performing.

Thanks for reading.

Inspired by:

https://www.youtube.com/watch?v=RuGFSSUYGXQ

https://www.youtube.com/watch?v=rSY-zqDfc_s

Reviewed by: @nhnhatquan, @WickyNilliams

You Might Not Need Integration Tests

Dylan Broadbridge — Sun, 07 Apr 2019 08:22:50 +0000

NOTE: This is an advanced topic and assumes knowledge of CI/CD, integration testing, types and a bit of devOps.

Why I questioned the existence of integration tests

Ah the classic integration test, giving you confidence that all the parts of your app work together in a way unit tests never can. They can take a long time to run and require complex tooling like localstack (in the case of AWS), but that doesn't matter right? All good things take time and the confidence of everything going green is worth it.

Maybe Not.

My idea on testing was challenged when I read this tweet:

// Detect dark theme var iframe = document.getElementById('tweet-1110021204272971776-860'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1110021204272971776&theme=dark" }

What is going on here? Are my integration tests a waste of time???

How did we get here?

After having a good think for few days I finally understood the simple idea behind Brian's tweet:

Your testing strategy should include your operational strategy.

I had failed to take into account how operations could affect my testing, it was a shock and when I rounded up my fellow engineers it turns out none of them had either.

But why does it matter? Ops is Ops and testing is testing surely they have minimal impact on each other? Right?

Nope.

The success of your application is dependent not only on what you do in code and how it runs on your machine but also how it will work in production.

Testing strategies have remained solid for decades now, integration tests have existed since the dawn of programming and still continue to be useful. But Ops has been evolving, ideas like CI/CD have only recently taken off and not taking advantage of modern Ops would be a mistake when deciding on a testing strategy.

So what do we do?

Canary Deployments

... Prefer to invest in monitoring (and partial rollouts/rollbacks) when possible. - Brian

Modern Ops tooling gives you the option of doing canary releases which is an extension of red green deployments where you always deploy your changes onto a new server.

Canary deployments divert a small % (usually 10%) of users for a small amount of time (usually 15-60 min) to the new deployment.

During this switch your deployment service is receiving events from your monitoring tools and will "cancel" the switch if an error event is seen and all users will continue to use the currently deployed application.

It can be difficult to setup but luckily managed services like AWS give you canary deployments for free and can generally be configured in less than a day.

The result of canary deployments is that any issues with the new deployment are contained to only a few users with automatic rollback preventing any extra damage.

The cost of making mistakes in production has been greatly reduced with tools like canary deployments.

Type to safety

So making a mistake in production is much less of an issue than it used to be, but still how do you know your code works if you aren't writing any integration tests?

Types. If it compiles it works.

Let's look at some common examples of integrations and how types are able to give us enough confidence that our code will work in production.

3rd Party Api's

Chances are these APIs come with SDK's that are fully typed, meaning that if you give the right input and it passes type checking then it is safe to assume that it works.

There is actually nothing to test, you have called the SDK correctly and the rest is up to the 3rd party provider.

The response will also be typed so you know the exact shape of data that you get back from the API. If you try to use this data incorrectly you will have a compile time error.

Database interactions

Assuming your database methods are fully typed and tested in an internal or external module you should have no trouble just using the methods and types provided by that module.

If your database requires certain data to create a record then your types will catch that. If you can only update certain fields your types will catch that. If you are using the response from the database to do some logic and you are missing some data the types will also catch that.

I would say that for almost every database interaction besides complex data migrations and table creation types are more than sufficient to have confidence that it will work in production.

What about stuff that isn't CRUD?

So from above you end up with 0 unit and integration tests for CRUD and basic API usage. But what about business logic?

...Testing of functions pays for itself (when using property testing and no mocks)... - Brian

You should be able to write your business logic as functions that you can test separately without the need to mock anything.

Doing this you get access to more advanced testing options like property testing where you can generate a wide range of inputs (usually 100) and check that your business logic handles them in a fraction of a second. Your tests will alert you to any edge cases that you may have missed in your code providing extra value to traditional deep assertion unit tests. (I recommend a combination of both)

Why does this matter? Integration tests have worked for decades why should we change?

Think of it like this, you either spend time writing integration tests and running them on every deployment or just skip them and rely on your Ops and types.

There is a point where the time spent fixing that one prod error that occasionally makes it through is less than adding on 40, 60, 90+ minutes of waiting for your
integration tests to run before going to production on each commit.

Lets do some Math:

Traditional Workflow per commit:

Write code (10 min)
Write unit tests (10 min)
Write integration tests (10 min)

Deploy to a test environment (5 min)
Run your integration test (2N min)

Deploy to production (5 min)

Total: 40 + 2N mins per commit

My Suggested Workflow:

Write code (10 min)
Write unit and generative tests (15 min) (extra value here)

Deploy to production (5 min)

Total: 25 mins per commit

Let's say you end up with 100 commits, that would take 2,500 mins to develop without integration tests compared to a whopping 14,000 mins using integration tests. That's a factor of 5.6x.

Is the extra fraction of a % more confidence worth losing out on 5.6x speed?

Integration tests can also be difficult to setup and maintain, time that could be better spent on better monitoring in production.

Integration tests are still useful and maybe occasionally necessary, but consider their use carefully because you might not need integration testing for your webapp.

Thanks for reading,

Please feel free to comment, tweet or DM me with any thoughts or questions.

Thanks to @paulgrizzay, @someivorytower, @nhnhatquan, @puffnfresh and others for reviewing.

DEV Community: Dylan Broadbridge