Serverless Test Strategy

I was chatting with some colleagues today and the subject of test strategy for Serverless applications came up. I've worked with a few clients on Serverless applications over the last few years and this is the test strategy I tend to come back to.

AWS say best practice is to prioritise testing in the cloud. It's hard to argue with them, they've helped more clients with Serverless than I'll ever dream of helping, but I'll give it a shot anyway.

 Unit Tests

The feedback cycle of testing in the cloud is just too slow for me. We can use CDK + hot swap deployments to make the deployment as fast as possible but, for me it still doesn't have the same pace as doing TDD and running tests on save.

This is why the bulk of my testing is still unit tests.

 Outside In Tests

Naming things is hard so it's no surprise that I don't have a name for the style of unit testing I prefer. Some people call it "outside in" testing. Martin Fowler calls them https://martinfowler.com/articles/2021-test-shapes.html. I've tended to call it "integration style unit testing" because the modules remain integrated rather than testing each unit separately.

My mocking approach stems from the principle "test the behaviour, not the implementation". Which to me means, verify that the right data eventually gets stored in the database or sent to a downstream API, rather than test that module A calls function X in module B.

Testing the behaviour, not the implementation means we can refactor the system under test to our hearts content and be confident that it's still correct. Testing the implementation means we have to re-write the tests when we refactor, that's more work than I'd prefer and raises the risk of introducing bugs due to incorrect tests.

With outside in testing the tests wouldn't need to change to verify this refactor to add an additional module.

This doesn't mean there's no place for focused unit tests which isolate individual modules. I still use TDD at the unit scale to implement complex business logic. These focused tests may even make up the largest number of test cases but that's usually because they're data driven to cover the full range of possible inputs.

test.each([
  [1, 1, 2],
  [1, 2, 3],
  [2, 1, 3],
])('.add(%i, %i)', (a, b, expected) => {
  expect(a + b).toBe(expected);
});

 Integration Testing

Now we've got a solid foundation of unit tests we can get to the AWS recommendation, testing in the cloud.

The aim here is to be able to deploy "ephemeral" environments. Every developer and every merge request should be able to create a stack that is isolated from the other environments to run their own tests. I've used Terraform Workspaces for this most recently but it's supported by CDK, Cloudfromation, Serverless Framework and presumably all the other IAC frameworks.

Ideally these environments be isolated from 3rd party downstream dependencies and other components of the wider system. This prevents unnecessary cost of 3rd party licenses, PAYG consumption, etc and it prevents flakyness caused by downstream collaborators outside your control. Use additional CloudFormation/CDK/Terraform stacks to deploy fake or alternative stand ins.

What about data?

The Burning Monk has some articles about monolithic deployments which I won't reproduce (although a lot of this test strategy aligns with his approach too). I totally agree with this approach. It simplifies and de-risks the deployment but I've had issues with taking this approach too far.

For example, when deploying an API with a WAF included in the monolithic deployment we quickly ran into limits with the number of WAFs allowed in a region. I've also ran into issues with secrets in the monolithic deployment. Secrets are created but they are empty and need to be populated each time we created a new ephemeral stack.

The "fix" for us was to create some "shared" infrastructure outside the monolithic deployment. We called it shared but it was only really shared between stacks in the dev account. In higher environments they were used by the single instance of the workload.

Simple scripts deployed as Lambdas can be used to populate standing data. These serve double purpose for setting up production without us having to do "click-ops" through the console or connect to prod from developer machines.

 A Necessary Evil

I can't stand e2e tests, they're the most brittle tests, particularly in a Serverless environment where we're using asynchronous patterns a lot.

Annoyingly I also can't feel totally confident without a thin veneer of them. It's such a thin veneer that it'd be quick enough to run manually but then how would we do continuous delivery?

 Test in production

Each layer of the test pyramid should provide more confidence than the last that we can release, but what provides confidence that the release to production is actually working? Just because we've got the config right for the integration environment, our prod config might be trash.

We absolutely have to have some level of testing in production, ideally the e2e tests will run after deployment to prod.

What else?

This is a fairly traditional test strategy and it's really not complete for modern engineering.

I've not talked about static code analysis and linting which you could describe as testing.

We should also be including consumer driven contract testing, mutation testing, performance testing, operational acceptance testing, accessibility testing and probably more but I'm going to save all those for another blog.

Comments

[Ryan Breidenbach]

The aim here is to be able to deploy "ephemeral" environments.

This is easier said than done. How have you done this with Terraform workspaces. Are the ephemeral environment provisioned to an isolated AWS account? Or do you use a convention to generate unique names for resources that are specific to a merge request? Do the Terraform configuration support this?

[Ben Crinion]

Hi @twoqubed, thanks for your comment.

Yeah, it has it's challenges. The way I've done it in the past is, all engineers shared sandbox account and resources have unique names based on the workspace name.

We used a local to give us a reusable conventional prefix depending on whether it was dev, test or prod or a workspace.

You can get the name of the workspace using ${terraform.workspace}.

developer.hashicorp.com/terraform/...