DEV Community: Ben Crinion

Serverless Test Strategy

Ben Crinion — Tue, 07 May 2024 23:08:53 +0000

I was chatting with some colleagues today and the subject of test strategy for Serverless applications came up. I've worked with a few clients on Serverless applications over the last few years and this is the test strategy I tend to come back to.

AWS say best practice is to prioritise testing in the cloud. It's hard to argue with them, they've helped more clients with Serverless than I'll ever dream of helping, but I'll give it a shot anyway.

Unit Tests

The feedback cycle of testing in the cloud is just too slow for me. We can use CDK + hot swap deployments to make the deployment as fast as possible but, for me it still doesn't have the same pace as doing TDD and running tests on save.

This is why the bulk of my testing is still unit tests.

Outside In Tests

Naming things is hard so it's no surprise that I don't have a name for the style of unit testing I prefer. Some people call it "outside in" testing. Martin Fowler calls them https://martinfowler.com/articles/2021-test-shapes.html. I've tended to call it "integration style unit testing" because the modules remain integrated rather than testing each unit separately.

My mocking approach stems from the principle "test the behaviour, not the implementation". Which to me means, verify that the right data eventually gets stored in the database or sent to a downstream API, rather than test that module A calls function X in module B.

Testing the behaviour, not the implementation means we can refactor the system under test to our hearts content and be confident that it's still correct. Testing the implementation means we have to re-write the tests when we refactor, that's more work than I'd prefer and raises the risk of introducing bugs due to incorrect tests.

With outside in testing the tests wouldn't need to change to verify this refactor to add an additional module.

This doesn't mean there's no place for focused unit tests which isolate individual modules. I still use TDD at the unit scale to implement complex business logic. These focused tests may even make up the largest number of test cases but that's usually because they're data driven to cover the full range of possible inputs.

test.each([
  [1, 1, 2],
  [1, 2, 3],
  [2, 1, 3],
])('.add(%i, %i)', (a, b, expected) => {
  expect(a + b).toBe(expected);
});

Integration Testing

Now we've got a solid foundation of unit tests we can get to the AWS recommendation, testing in the cloud.

The aim here is to be able to deploy "ephemeral" environments. Every developer and every merge request should be able to create a stack that is isolated from the other environments to run their own tests. I've used Terraform Workspaces for this most recently but it's supported by CDK, Cloudfromation, Serverless Framework and presumably all the other IAC frameworks.

Ideally these environments be isolated from 3rd party downstream dependencies and other components of the wider system. This prevents unnecessary cost of 3rd party licenses, PAYG consumption, etc and it prevents flakyness caused by downstream collaborators outside your control. Use additional CloudFormation/CDK/Terraform stacks to deploy fake or alternative stand ins.

What about data?

The Burning Monk has some articles about monolithic deployments which I won't reproduce (although a lot of this test strategy aligns with his approach too). I totally agree with this approach. It simplifies and de-risks the deployment but I've had issues with taking this approach too far.

For example, when deploying an API with a WAF included in the monolithic deployment we quickly ran into limits with the number of WAFs allowed in a region. I've also ran into issues with secrets in the monolithic deployment. Secrets are created but they are empty and need to be populated each time we created a new ephemeral stack.

The "fix" for us was to create some "shared" infrastructure outside the monolithic deployment. We called it shared but it was only really shared between stacks in the dev account. In higher environments they were used by the single instance of the workload.

Simple scripts deployed as Lambdas can be used to populate standing data. These serve double purpose for setting up production without us having to do "click-ops" through the console or connect to prod from developer machines.

A Necessary Evil

I can't stand e2e tests, they're the most brittle tests, particularly in a Serverless environment where we're using asynchronous patterns a lot.

Annoyingly I also can't feel totally confident without a thin veneer of them. It's such a thin veneer that it'd be quick enough to run manually but then how would we do continuous delivery?

Test in production

Each layer of the test pyramid should provide more confidence than the last that we can release, but what provides confidence that the release to production is actually working? Just because we've got the config right for the integration environment, our prod config might be trash.

We absolutely have to have some level of testing in production, ideally the e2e tests will run after deployment to prod.

What else?

This is a fairly traditional test strategy and it's really not complete for modern engineering.

I've not talked about static code analysis and linting which you could describe as testing.

We should also be including consumer driven contract testing, mutation testing, performance testing, operational acceptance testing, accessibility testing and probably more but I'm going to save all those for another blog.

Is consultancy right for you?

Ben Crinion — Tue, 03 Jan 2023 00:06:20 +0000

I wanted to reflect on 2022 and with January usually being a popular time for people to move jobs hopefully my reflections can help someone with their next move.
At the start of 2022 I was wrapping up my first year as a consultant at Infinity Works and a year of working with Cinch. What an incredible experience that was, my first exposure to serverless (with a small 's' and a big 'S'). Cinch have an incredible engineering culture. They play a big part in the local meet-up scene and featured in Werner Vogels' reInvent keynote talk. Check out Toli's talk at GOTO EDA Day from September https://www.youtube.com/watch?v=wM-dTroS0FA.
What's my point?
I imagine you're thinking "that sounds great, how is that helpful to me?". I've done a fair bit of interviewing at IW of the last couple of years. One of the worries people have about joining consulting is that people don't like the sound of short projects. The start of 2023 finds me almost a year in with my second client. I've definitely had shorter tenures in previous non consultancy roles. Some of my colleagues have spent over 2 years at a single client.
Don't let short engagements put you off consultancy. They exist but there are lots of opportunities for longer stints if you want them.
In February I rotated to a different client as a tech lead for a mixed team of onshore and offshore colleagues. The tech stack was similar but the culture is very different. This client is in the health space and as such have a very different approach to governance and risk. I found it difficult to accept the change of pace. Going from multiple deployments to production a day, to maybe one a week, felt like exchanging a sports car for a people carrier (probably, I've never driven either). Also, considering the reduced appetite for risk the team thought that bundling a weeks worth of changes into each release wasn't the ideal strategy. With that in mind we started making a plan for introducing continuous deployment.
What's my point?
Another of the common questions we get in interviews is about the impact consultants can have at a client. It's a great question to ask. Lots of people are familiar with Daniel Pink and the intrinsic motivators: autonomy, mastery and purpose. Some candidates worry about being a "bum on a seat" without much input into technology decisions or how a company runs and that leaves them without that autonomy and purpose. That's not been my, admittedly short, experience. I think the continuous delivery working group is an example of this, when it's complete it'll be a massive change from the current way of working (although we'll get there through evolution, not revolution). It's not the only example, we've helped with technology strategy, company culture, even recruitment (which might seem unusual for a consultancy to help a client with something that'll end with fewer billable hours but that's what we did). If you're thinking about consultancy check out some of the things we've worked on over at our website. https://www.infinityworks.com/work/
In April I became a certified architect and a certified Kanban professional and in December I completed a 2 week data engineering course. I know there are mixed opinions about certification. For what it's worth I learned a tonne on both courses and found them valuable. I've also been to a couple of conferences: GOTO EDA day and AWS community summit.
What's my point?
My point is that I've received as much training in 2 years of consultancy as probably the last 10 years working for product companies. There's more to it than this but the fact is that the more we know, the more useful we are to our clients and the more billable we can be. There's a clear return on investment for the training we receive and most of my colleagues have been given time and money to work on something whether it's cloud certification, agile or leadership training, books or something else.
I don't want to give the impression that consultants are just billable resources to our leadership. We're absolutely not, our leadership are great and show a genuine interest in us. There's no pressure to train on a path set out for us. Our career journeys are in our own hands.
One final quick thing to mention. I ruled out consultancy in the past because I didn't like the idea of lots of travel. I think I've spent approximately 6 nights away from home in the last 2 years. Take this with a pinch of salt because there's been this unheard of thing called COVID that's somewhat limited travel but Infinity Works have regional offices and aim to place our consultants with local clients. It's not always possible, but considering that everyone is much more remote friendly these days and there are increasingly more companies and government agencies setting up outside London, being away from home might not be the deal breaker that it used to be.
I work with some amazing colleagues at IW, consultancy can be a career fast track. I wish I'd made the leap years ago.
Best of luck with whatever you decide to do in 2023!

Make onboarding simple using VS Code Remote Containers

Ben Crinion — Sun, 28 Nov 2021 12:12:41 +0000

Note: this article was written before the Docker Desktop license change but I still think it's a valuable technique. I believe that the Docker Desktop license will still be good value for money compared with the time it takes to setup a dev environment.

Over the last few weeks our team has grown rapidly. Each time a new engineer joins the team (or an existing engineer gets a new machine) we dig out the laptop onboarding guide and spend a chunk of time installing the right frameworks and tools to get our teammate up and running. This can be fairly painful: the onboarding doc isn't always updated, links die and toolchains evolve. To add to this we have a mix of Apple, Windows and Linux users which means we might be trying to support someone using a platform we're not familiar with.

Another issue we have is that our squad is responsible for multiple services. These have slightly different dependencies. Different versions of NodeJS, Python, Serverless Framework or CDK, different test runners etc. Add consultancy into the mix and we might have people working on several services at multiple clients and managing the dependency mix gets difficult.

Wouldn't it be useful if we had some light weight, isolated operating systems? Something we could run on any machine and that we can configure separately without them impacting each other?

Luckily for us Docker exists and can do exactly this. Even better, Microsoft have created the Visual Studio Code Remote - Containers extension which lets you use a Docker container as a full-featured development environment within VS Code.

This is how we solved some of the problems we came up against using Dev Container and Serverless framework.

Not using dev containers

The first problem we have is that not everyone on our team wants to use VS Code. Because of this, everything we change to enable dev containers needs to also work natively and with our CI/CD pipeline. This baiscally boils down to replacing localhost with the container hostname which is available by default in a Docker container.

const hostname: process.env.HOSTNAME || 'localhost'

Using Docker

We use LocalStack for integration testing so we need to be able to run containers from within our dev container.

It's possible to install a container engine within a container and create "child" containers but it's complex and there's a simpler solution.

We can use Docker on the host machine to create "sibling" containers by installing the Docker CLI and mounting /var/run/docker.sock. The devcontainer.json settings file has a mounts property which can be used to have some control over the dev container file system.

  "mounts": [
    "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind",
  ],

Docker Sock Permissions

If you're using a non-root user inside your dev container (and you probably should) then you need to give that user permissions to use docker.sock.

You could run this as sudo and it will persist until you rebuild the container or it can be automated using a post run command in the devcontainer.json file which means no-one has to remember to do it.

  "postCreateCommand": "sudo chown vscode:vscode /var/run/docker.sock",

Using AWS and Git

We need to use the AWS CLI and Github. We could duplicate the credentials and keys in our dev container file system but they would not persist if we had to rebuild the container and aren't reusable between different projects.

We can share the host's ssh keys and AWS credentials by mounting the host file system in the container (again using the mounts property in devcontainer.json).

  "mounts": [
    ...
    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.aws,target=/home/vscode/.aws,type=bind",
    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.ssh,target=/home/vscode/.ssh,type=bind"
  ],

Filesystem Performance Issues

We're using the serverless-webpack plugin but we were getting errors during packaging.

Serverless: Packing external modules: .....

 Error ---------------------------------------------------

  Error: npm install failed with code 1
      at ChildProcess.<anonymous> (/workspace/node_modules/serverless-webpack/lib/utils.js:91:16)
      at ChildProcess.emit (events.js:314:20)
      at ChildProcess.EventEmitter.emit (domain.js:483:12)
      at maybeClose (internal/child_process.js:1022:16)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

The error message doesn't give any pointers to what's going wrong but there were some clues when we tried to clean up the .webpack folder. Running ls from inside the container showed it to be enpty but it wouldn't allow us to delete it because it wasn't empty on the host.

This is because the default source code mount uses the cached consistency model. The cached consistency model is more appropriate for files which the host modifies. There's a good description of the different modes in this StackOverflow answer.

Our solution was to use a volume for the webpack and node_modules folders as "volumes are the preferred mechanism for persisting data generated by and used by Docker containers". mounts property to the rescue again.

  "mounts": [
    ...
    "source=node_modules,target=${containerWorkspaceFolder}/node_modules,type=volume",
    "source=webpack,target=${containerWorkspaceFolder}/.webpack,type=volume",
  ],

These folders will be owned by root so we'll use the postCreateCommand again to change their ownership back to vscode.

  "postCreateCommand": "sudo chown vscode:vscode node_modules && sudo chown vscode:vscode .webpack",

Finally we need to modify the webpack config slightly. It's not possible for the container to delete the volume so we've set the webpack output path to a sub folder in the webpack.config.js.

  ...
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack/build'),
    filename: '[name].js',
  },
  ...

Another option would be to use a delegated mount which are more appropriate when the container's view of the filesystem is authoritive or clone the whole repo into a container volume.

Docker Networking

As I mentioned earlier, we're using LocalStack for integration testing and we have a bash script which uses docker-compose to manage that container. Docker compose creates a network for the workload, this allows all the containers in the workload to communicate easily but it isolates them from other workloads and individual containers. This meant that Serverless offline and the tests which were running in the dev container couldn't access the database running in LocalStack.

Docker containers can be attached to more than one network at a time so we've solved this by creating a dedicated network and attaching the dev-container and LocalStack container to it. There are another couple of properties in the settings file which can help us with this. We can ensure the network exists before we start the dev container using the initializeCommand property, and use runArgs to provide additional arguments to the dev container (we append || true to the initializeCommand to ensure the command succeeds if the network already exists.).

  "initializeCommand": "docker network create payment_network || true",
  "runArgs": ["--network=payment_network"],

This is only half the job. We also need to attach the LocalStack container to the network and we still can't use localhost for addressing. This is another area where we've had to consider the CI/CD pipeline and users who don't want to use VS Code.

In our test setup shell script we inspect an environment variable which will only be present in our dev container and combine settings from more than one YAML file by using the -f parameter. We can set environment variables in the dev container using the containerEnv property in devcontainer.json.

if [ -z "$LOCALSTACK_HOST" ]
then
    docker-compose -f docker-compose.yml up -d localstack
else
    docker-compose -f docker-compose.yml -f docker-compose.devcontainer.yml  up -d localstack
fi

# docker-compose.yml
version: '3.5'
services:
  localstack:
    image: localstack/localstack:0.12.15
    environment:
      - DEFAULT_REGION=eu-west-1
      - DEBUG=true
      - LAMBDA_EXECUTOR=docker
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
    ports:
      - '4567:4566'

# docker-compose.devcontainer.yml
version: '3.5'
services:
  localstack:
    container_name: paymentslocalstack
    environment:
      - HOSTNAME_EXTERNAL=paymentslocalstack
networks:
  default:
    external:
      name: payment_network

  "containerEnv": { "LOCALSTACK_HOST": "paymentslocalstack", "LOCALSTACK_PORT": "4566" },

Specifying the container_name in the devcontainer compose file means we've got a consistent hostname we can use to address the LocalStack container and we expose that inside the dev container using an environment variable.Another thing to remember about container networking is that containers on the same network don't need to use the mapped external port. That's only required for the host to container communication. We've also added this as an environment variable so we can use it in our tests.

The final issue we had with networking was LocalStack specific. Many AWS services publish metadata which includes the host name i.e. SQS queue URLs. This metadata is fundamental to how they operate. We need to tell LocalStack the new hostname by setting the HOSTNAME_EXTERNAL environment variable in that container which you can see in the second docker-compose yaml file.

Summary

Now we've got a repeatable way to onboard new team members, no one should ever install the wrong version of Python again.

Instead of taking hours or even days to get their system setup, possibly being guided by someone else on the squad, new team members can get themselves up and running in minutes.

Hopefully some of these fixes will be useful for you when you setup a dev container for your project.

The next step for us is to investigate how we can use this with GitHub Code Spaces.

Date handling bites again

Ben Crinion — Wed, 23 Sep 2020 08:11:17 +0000

One of my team mates found an interesting issue in our code-base so I thought I'd do a write-up.

TLDR: Don't compare a DateTimeOffset to DateTime.MinValue, compare it with DateTimeOffset.MinValue.

Technically you could compare DateTimeOffset with DateTime.MinValue if you can guarantee that the system timezone will always be UTC or UTC minus some offset.

Into The Detail

Every request my teammate mate to our API was failing with the same exception, but despite running the master branch and the exact same request, I couldn't reproduce the error.

The exception was being thrown by the constructor of an AbstractValidator from the FluentValidation package.

public class TestValidator : AbstractValidator<ThingWithDate>
{
    public TestValidator()
    {
        RuleFor(x => x.Date)
            .GreaterThan(default(DateTime));
    }
}

This validator was injected into nearly every controller in our API via a BaseController (👍).

It turns out that to reproduce the issue I needed to run the service using the same system timezone as my colleague in the Balkans (UTC+3).

You can see from the stack trace that:

We're initialising the TestValidator
That is hitting the DateTimeOffset implicit cast operator to convert the DateTime to a DateTimeOffset.
That calls the DateTimeOffset constructor which accepts one DateTime parameter.
The constructor calls ValidateDate(DateTime, Timespan). The TimeSpan is the difference between UTC and the local system timezone.

Internally a DateTimeOffset is a DateTime and an offset in minutes. ValidateDate, unsurprisingly, ensures that the DateTime is valid when the offset is applied. When you apply a negative offset to DateTime.MinValue you get a date that can't be represented by the DateTime type and you get an ArgumentOutOfRangeException instead.

Here is the code for the validate method lifted from here. Comments are mine...

private static DateTime ValidateDate(DateTime dateTime, TimeSpan offset)
{
    // +2 is 72000000000 ticks
    // utcTicks is converting the "local" dateTime to UTC by subtracting the offset
    Int64 utcTicks = dateTime.Ticks - offset.Ticks;

    // This section validates that utcTicks is a valid time that can be represented

    // utcTicks = -72000000000
    // DateTime.MinTicks = 0
    // -72000000000 < 0 == true => Kablamo
    if (utcTicks < DateTime.MinTicks || utcTicks > DateTime.MaxTicks)
    {
        throw new ArgumentOutOfRangeException("offset", Environment.GetResourceString("Argument_UTCOutOfRange"));
    }
    return new DateTime(utcTicks, DateTimeKind.Unspecified);
}