DEV Community: Paul Swail

The Simple Guide to Testing within your Serverless CI/CD Pipelines

Paul Swail — Mon, 29 Mar 2021 19:49:29 +0000

Designing a testing strategy and a CI/CD pipeline for your serverless application go hand-in-hand. You can't do one without the other. And if you don't have a dedicated DevOps expert in your team, it can be hard to know what your pipeline should look like.

You'll need to answer questions such as:

What environments do I need?
What types of tests/checks do I need to run?
For each type of test:
- What purpose does it serve? What type of failures is this meant to detect?
- What are its dependencies/pre-requisites to running? (e.g. code libraries, config settings, deployed resources, third-party APIs)
- When should it run and against which environment?

In this article, I'll help you answer these questions by walking through two relatively straightforward CI/CD workflows that I've used with my clients. For each workflow, we'll look at the different types of tests or checks that run at different stages.

Workflows overview

Here are the two workflows I use, along with associated triggers:

Pull Request Workflow — A new pull request (PR) is created, or an existing PR branch has new commits pushed to it
Mainline Workflow — A PR is merged to the main branch (or a commit is pushed directly to the main branch, if your settings allow this)

Let's look at each workflow in turn.

Pull Request Workflow (CI only)

This simple workflow is triggered whenever a developer creates or updates a PR in GitHub.

The goal of this workflow is to check the quality of the code changes a developer wishes to merge before any human code review is performed, and feedback any violations to the developer as quickly and precisely as possible.

Aside: The steps I run here are CI only with no CD. This decision is a trade-off on the complexity involved in setting up dynamic ephemeral AWS environments for each Pull Request as opposed to the long-living environments of test, staging and prod. That said, you may very well find that it's worth investing this effort for your team in order to bring forward the feedback on failing integration/E2E tests to pre-merge.

Here's what gets run within this workflow:

npm ci — A variation of npm install that performs a clean install of node_modules specified in the package-lock.json file
Static analysis
1. ESLint — Ensure the code matches agreed-upon standards to ensure consistency across the team (others have bikeshed so you don't have to!)
2. Run TypeScript tsc to check types
Run unit tests using Jest. Unit tests here must run totally in memory (or filesystem) and not have any dependency on deployed resources or third-party APIs (which would make them integration/E2E tests, which we'll cover later).

I have recently started using GitHub Actions to run this workflow:

It's really quick to configure and get working if you're already using GitHub to host your code.
It delivers extremely fast feedback to developers directly within the GitHub UI where they created their PR (checks usually complete in under 30 seconds).
Code reviewers can see that the automated checks have passed right within GitHub and don't need to check another CI system before starting their review.
Given that I'm only running unit tests here, I don't need to worry about connecting my workflow to AWS accounts or other third party services.

Here's the code for configuring the workflow:

# ./.github/workflows/ci.yml
name: CI

# Triggers the workflow on new/updated pull requests
on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x] # set this to configured AWS Lambda Node.js runtime

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: Cache dependencies
      id: cache
      uses: actions/cache@v2
      with:
        path: ./node_modules
        key: modules-${{ hashFiles('package-lock.json') }}
    - name: Install dependencies
      if: steps.cache.outputs.cache-hit != 'true' # only install if package-lock has changed
      run: npm ci --ignore-scripts
    - run: npm run lint
    - run: npm run compile
    - run: npm test

(Check out this article for details on the technique I used here to cache the NPM install node_modules between executions, which was the single slowest step in my workflow, often 20–30 seconds).

Mainline Workflow (CI + CD)

The mainline workflow is responsible for getting a developer's change (which has passed automated CI and human code review checks) through further stages and eventually into production.

To implement this, I use AWS CodePipeline for orchestrating the flow and AWS CodeBuild for running the deployment and test tasks. Since these services are already tightly integrated into the AWS ecosystem, there is less effort and security risk in performing the deployment (e.g. I don't need to give an external service access to my production AWS account).

Here's what my flow looks like:

You'll notice that there are 3 stages: test, staging and prod. I typically deploy each of these stages to their own AWS account. While this isn't a must-have for the pre-production environments, the account boundary provides the best isolation between environments so you can be more confident that deployments and tests run in each environment don't interfere with each other. And you should always isolate prod in its own AWS account.

There are a few assumptions my pipeline is making here that you should be aware of:

The dev team is following a release-from-mainline trunk-based development Git branching model. If this isn't the case for your team, most of what I'm recommending here could be adapted for a branch-for-release model.
The app is being deployed monolithically from a monorepo, i.e. all resources within the system are deployed at the same time. I find monolithic deployments are the easiest to manage within the client teams and products I work with. If your system has multiple microservices AND you wish to deploy these independently based on what code changed, then you'll need multiple pipelines which will be significantly more complex than what I'm proposing here (check out Four Theorem's SLIC Starter for a great example of CI/CD pipeline for independently deployed serverless microservices).

In the next sections, I'll walk through each stage in more detail.

Build stage

The build stage repeats the static analysis and unit test tasks performed in the GitHub Actions Pull Request Workflow. This is to double check that the merge hasn't introduced any issues.

You might be surprised to notice that there is no packaging step here. The reason for this is because unfortunately Serverless Framework (which I use to define and deploy the infrastructure resources) doesn't support building an environment-independent deployment artifact which can then flow through each stage in the pipeline. This means that the "Deploy stacks" task that you see in the later stages creates an environment-specific package which it deploys to the target account. (General consensus among experienced serverless devs is that this limitation isn't anything to worry about).

Test stage

The purpose of the test stage is to provide an isolated environment that only this CI/CD pipeline has access to for automated E2E testing. No human testers or other systems have access to this environment.

The first step at this stage is a configuration test. While the vast majority of the configuration will be contained in the version-controlled source code (because every resource is defined in Infrastructure-as-Code), the one thing that's not in Git is secrets. Secrets such as passwords, API keys, etc, that are required by Lambda functions should be stored in SSM Parameter Store (or Secrets Manager) and fetched at runtime. However, the deployment of the values for these secrets needs to be performed by a human engineer ahead of time, and this is something which they could forget to do! This check uses a secrets.sample.env file stored in the Git repo and checks that each key defined in it has an associated parameter set within SSM Parameter Store for the target environment. If this check fails, deployment does not proceed.

After the config check comes the deployment itself, using the sls deploy command. I usually have 2 stacks to deploy (an infra stack containing stateful resources such as DynamoDB tables and S3 buckets and an api stack containing APIGW or AppSync endpoints along with Lambda functions), which are deployed in series.

Once the deployment completes, the E2E tests are run using Jest. Before running these tests, the serverless-export-env plugin is used to generate a .env file with all the URLs, etc from the deployment so that the tests know which endpoints to hit.

Staging stage

The purpose of the staging stage is to act as an environment where human testers can perform manual testing before releasing changes to production. If you're building an API and you have separate front-end (web or mobile) teams, they can point their pre-production apps at this environment as it will be relatively stable, since code that gets this far has passed full E2E testing in the test stage.

We still do need to run a few tests at this stage though. Before deployment, we need to again perform a config check to ensure that secrets have been deployed correctly.

Post-deployment, we run a set of smoke tests. The main purpose of these smoke tests is to ensure that the system is available and that environment-specific configuration is correct.

For example, if a Lambda function talks to a third party API, we could write a smoke test that invokes the function and verifies that it communicates correctly with this service. This would verify that the API URL and API keys we're using are correct.

An important aspect of smoke tests (which differentiates them from E2E tests) is that they should have minimal side effects. Ideally they would be fully readonly but at the very least they should not introduce any test data into the system which could become visible to human users.

Manual approval

Once the staging deployment and tests complete, a notification is sent to the team to manually approve. This is an opportunity to perform any manual testing before proceeding to production.

Once satisfied with the system, you can use the CodePipeline console to give approval and the pipeline will move onto the prod stage.

Production stage

The prod stage is the live environment that your users will access. The steps at this stage are the same as those in the staging stage.

Summary of test types

So now we've reach the end of the pipeline, let's summarise the different types of tests we've used and where they were employed:

Test	Description	When to run
Static analysis	Lint and compile (`tsc`)	At start of PR and mainline pipelines
Unit	Verifies complex business logic. Has no out-of-process runtime dependencies	After static analysis at start of PR and mainline pipelines
Config	Verifies presence of non-version controlled configuration (e.g. secrets)	Pre-deployment, in `test`, `staging` and `prod` stages
E2E	Verifies behaviour of deployed system, e.g. by invoking API endpoints, deployed Lambda functions, etc	Post-deployment, in `test` stage
Smoke	Verifies system availability and correctness of environment-specific config	Post-deployment, in `staging` and `prod` stages

Building out your own pipeline

This may seem like a lot of checks to create your own pipeline and maybe you don't have the time to build all this upfront. But don't let this put you off as having ANY tests running in a CI/CD pipeline is better than none. I've seen many teams that do no CI/CD at all and just deploy directly from developer workstations (and I'm not the only one!).

To get you started, the Pull Request Workflow is much easier to configure as it has less moving parts. The limitation is that you can only do CI and not CD.

Once you move on to the mainline CI+CD pipeline, consider adding each type of test incrementally to your pipeline. Here's my recommended order for building out each one:

Static analysis — Start with these as they're quick to run, catch a lot of silly mistakes early in the cycle, and are easy to implement as they just have a single config file at the repo-level.
E2E tests — These are the biggest confidence drivers for serverless apps, so once these run successfully you can have much greater assurance doing Continuous Delivery
Unit tests — Unit tests are usually fast to run and a good fit for testing complex business logic inside a Lambda function. I've ranked them behind E2E tests as I generally find my use cases don't require many of them.
Smoke tests — If you're doing manual testing, these are probably lower priority
Config tests — These checks would be lowest priority as your E2E or smoke tests should capture any missing/bad configuration. The major benefit config tests bring is not increased confidence but instead a faster failure since they can be run before the (usually slow) deployment step.

If you're interested in getting hands-on practice in creating the types of tests discussed in this article, check out my 4-week Serverless Testing Workshop. The workshop is a mixture of self-paced video lessons alongside weekly live group sessions where you will join me and other engineers to discuss and work through different testing scenarios.

Integration and E2E tests are the primary confidence drivers for serverless apps

Paul Swail — Tue, 27 Oct 2020 08:47:12 +0000

This article is part of a series on “Testing Serverless Applications” based on lessons I’m teaching in the Serverless Testing Workshop.

In the last article, we covered the three core goals driving why we write automated tests and the trade-offs that we need to make in order to reach satisfactory levels of confidence, maintainability and feedback loop speed.

You may have been left wondering "how do I get to a sufficient level of confidence?". A typical answer to this could be along the lines of: "write a load of unit tests and aim for as close to 100% code coverage as possible".

And while unit tests and automated code coverage measurement can be part of building up confidence in your serverless system, in my experience they are a small part.

In this article, I'll argue that you should favour integration and end-to-end (E2E) tests for maximising the confidence delivered by your automated test suite.

What can go wrong?

Before deciding what tests to write for a given use case, we need to first understand our failure modes. A failure mode is a specific way in which a system-under-test can fail, and it can have any number of root causes. If we know what these modes are, then we can write tests to "cover" as many of the failure modes as is feasible.

Aside: There are many failure modes for which we shouldn't write automated tests that run in pre-production environments as they're too costly/difficult to simulate in a test. For these we can instead employ testing in production techniques to detect.

Let's illustrate this with an example...

Consider the Simple Web Service pattern from Jeremy Daly's excellent collection of Serverless microservice patterns (that I must've now referenced about 5,932 times in previous articles):

This pattern is by far the most common in serverless applications that I've built and involves a synchronous request-response cycle from API Gateway thru a single-purpose Lambda function thru DynamoDB and back again.

Let's make our example more concrete and say that we're adding a new REST API endpoint for a sports club management app that allows managers to create a new club: POST /clubs.

So what could go wrong here? What tests should we write? Let's look at the deployment landscape for this use case to help answer these:

The light red boxes depict source areas made up of code or configuration that we (as the developer) write and which are then deployed to a target environment.

Each source area can be a cause of several failure modes. Let's look through each in turn:

API GW Method config:
- Path or method configured incorrectly (e.g. wrong case for path parameter)
- Missing or incorrect CORS
- Missing or incorrect Authorizer
API GW to Lambda integration config:
- IAM permission for APIGW to call Lambda function
- Misconfigured reference to Lambda function
Lambda execution config:
- Insufficient memory allocated
- Timeout too low
Handler function code:
- Business logic bug, e.g. client request validated incorrectly or user incorrectly authorized
- Incorrect mapping of in-memory object fields to DynamoDB item attributes (e.g. compound fields used for GSI index fields in single table designs)
Lambda to DDB integration config:
- Incorrect IAM permissions for Lambda function role to make request to DynamoDB
- Incorrect name/ARN of DynamoDB table
DynamoDB table config
- Incorrect field name or type given for hash or sort keys of a table or GSI when table was provisioned

Covering these failure modes with tests

If you look through each failure listed above, very few of them can be covered by standard in-memory unit tests. They're predominantly integration configuration concerns.

The diagram below shows the coverage scope and System-under-test (SUT) entrypoint for each of the three test granularities (unit, integration and E2E):

An important point for this particular use case is that each test level covers everything that the level below covers, i.e. integration tests will cover everything that unit tests do and E2E tests will cover everything that integration tests do.

So then why not only write E2E tests? Let's look at the key properties of each type of test for our example use case to see what they have to offer:

Unit tests:
- Test and Lambda handler code execute on local machine
- SUT entrypoint could either be the Lambda handler function or modules used within the handler
- No deployment required — uses test doubles to stub out calls to DynamoDB
- Can use automated code coverage tools
Integration tests:
- Test and Lambda handler code execute on local machine
- SUT entrypoint is typically the Lambda handler function
- Requires partially deployed environment (DynamoDB table)
- Can use automated code coverage tools
E2E tests:
- Only test code runs on local machine
- SUT entrypoint is the APIGW endpoint in the cloud which receives a HTTP request from our test
- Requires fully deployed environment (APIGW resources, Lambda function, IAM roles and DynamoDB table)
- Cannot use automated code coverage tools

The key benefit that integration tests give over E2E tests is that they allow for much faster iterative development. Their deployment overhead is low (one-off provisioning of a DynamoDB table) whereas E2E tests require deploying the most frequently changing areas (the handler code) every time.

And it's also possible to craft different event inputs for an integration test in order to test the business logic failure modes that you might otherwise consider writing a unit test for. For single-purpose Lambda functions with minimal business logic such as this, I often don't bother writing any unit tests and just cover off the business logic inside my integration tests.

Aside: Given the similarity between the entrypoint input events to the E2E and integration tests (a HTTP request and an APIGatewayProxyEvent), it's possible to get 2-for-1 when writing your test cases whereby a single test case can be run in one of two modes by switching an environment variable. I cover this technique in my workshop and I'll try to write more on it soon.

Conclusion

We've looked at the Simple Web Service pattern here but I've found value in leading with integration and E2E tests for almost any serverless use case that involves the triggering of a single-purpose Lambda function that calls on to a single downstream AWS service, e.g. S3-> Lambda-> DynamoDB, or SNS -> Lambda -> SES.

You definitely still can write unit tests in addition to integration and E2E tests (and there are many valid reasons why you might still want to do so). But my point here is that you may not have to for many serverless use cases where business logic is often simple or non-existent.

Unit tests are no longer the significant drivers of confidence from your test suite that they once were in server-based monolithic apps. In serverless apps, integration and E2E tests are king.

If you're interested in learning more about testing serverless apps, check out my 4-week Serverless Testing Workshop, starting on November 2nd, 2020. The workshop will be a mixture of self-paced video lessons alongside weekly live group sessions where you will join me and other engineers to discuss and work through different testing scenarios. If you sign up by October 28th you'll get a 25% earlybird discount.

The testing trade-off triangle

Paul Swail — Wed, 14 Oct 2020 19:39:35 +0000

Writing automated tests for serverless applications, indeed for any type of software application, can be somewhat of an art. Answers to the following commonly asked questions will be highly contextual and unique to an individual project or even a component/use case within a project:

How many tests should I write?
What type of tests (unit, integration, E2E) should I write?
How can I make these tests run faster?
Can/should I run these tests on my local machine?
What test coverage do we need? And how do I measure it?

I have observed a few different schools of thought recently on the best way to approach testing serverless applications amongst respected practitioners. These schools of thought are roughly split across two dimensions:

Local vs cloud-based testing
The weights given to each test granularity (unit, integration and end-to-end) within your test portfolio (e.g. the traditional test pyramid vs the microservices test honeycomb)

I have some thoughts on these based on my own experiences but I'll save those for a future article.

Get back to the basics of testing

What I do find useful to help answer the above questions for a specific context is to go up a level and first remind myself of why we write automated tests in the first place.

The first and foremost objective is confidence.

The tests should give sufficient confidence to the relevant stakeholders (engineers, product managers, business folks, end users) that the system is behaving as expected. This allows for decisions (automated or otherwise) to be made about the releasability of the system (or subcomponents of it). To do this well, the engineer authoring the tests needs a good knowledge of both the functional requirements and the deployment landscape (e.g. the AWS cloud services) and its failure modes.

Following on from this core objective of confidence, I see two further objectives for an automated test suite:

Feedback loop: the faster we learn about a failure and are able to identify its root cause, the faster a fix can be put in place. This feedback loop is relevant both in the context of a developer's iterative feature dev process inside their own individual environment and also within shared environments (test, staging, prod) as part of a CI/CD pipeline. Both the speed and quality of the feedback related to a test failure are important here.
Maintainability: How long does it take to write the tests in the first place? Are our tests making the system easier or more difficult to maintain in the long term? Do they require a lot of hand-holding/patching due to transient issues? Do the tests help new developers understand the system better or do they just add to their overwhelm?

The tension between each objective

So we've reflected on the three high-level objectives for why we write tests in the first place. These objectives aren't boolean but are instead on a sliding scale—a test needs to have "enough" of each level to be of net positive value and therefore be worth keeping (or writing in the first place).

Secondly, and crucially, each objective is often in conflict with at least one of the other two.

This is what I call the testing trade-off triangle.

The general principle is that when deciding upon your testing approach for a new system (or a new feature/change to an existing system), you need to make a compromise between confidence level, feedback loop and maintainability.

I believe this trade-off holds true for all categories of software development projects, not just serverless systems.

Example 1: Optimise for maximum confidence

Say that for every system use case, you write an exhaustive suite of tests covering every eventuality, both functional and environmental.

By doing this you're stealing from the maintainability pot as the time taken to write all these tests will mean you can't meet any deadlines that most businesses inevitably have.

Example 2: Optimise solely for fast feedback

Take another example where you optimise for the feedback loop objective by writing all your tests as lightning fast, in-memory unit tests, swapping out integrations with cloud services with a form of test double.

The loser here is the confidence objective. While we gain confidence that each individual component works as intended (and can iterate really quickly locally without making cloud round-trips), we don't get any confidence that the components will work together when deployed to a full environment (you've seen the memes!). And in serverless applications—where these components and their associated integration points are more numerous than ever—this is even more problematic.

(You could also make a case that the maintainability aspect would suffer in this example, since identifying, injecting and maintaining a suitable test double can take time to ensure it always matches the behaviour of the real system.)

Work out how to define "just enough" for your context

The next step in deciding how to approach testing for your specific context is to go through your use cases and for each one attempt to define what constitutes "just enough" for the three high-level testing objectives. This means asking yourself questions like:

What behaviours or integration points do we need to test?
What behaviours or integration points can we get away without testing?
Where is a fast feedback loop critical? And exactly how fast does this need to be?
Where can we tolerate a slower feedback loop?
Can we cover this test case with an integration (or even E2E) test or should we also spend time write unit tests for it?

Many of these answers will be specific to your organisational and functional requirements. But the good news with serverless applications on AWS is that there are general patterns for testing common use cases based on the AWS services being used that you can use to inform your decisions.

If you're interested in learning more about these patterns, I will be covering several of them in the 4-week Serverless Testing Workshop, starting on November 2nd, 2020. The workshop will be a mixture of self-paced video lessons alongside weekly live group sessions where you will join me and other engineers to discuss and work through different testing scenarios. If you sign up before October 26th you'll get a 25% earlybird discount.

Add type definitions to your Lambda functions

Paul Swail — Fri, 20 Mar 2020 08:36:48 +0000

Every Friday, I will share a small tip with you on something Lambda/FaaS-related. Because Fridays are fun, and so are functions. 🥳

Early last year, I tried out TypeScript in place of JavaScript after several years of holding off. Now I couldn’t go back to plain JS! I’ll save going into all the reasons why I think you should strongly consider TypeScript for Lambda-based apps for a future article, but for today’s tip I want to show you the @types/aws-lambda type definitions library.

What problem does this solve?

Lambda functions can have loads of different trigger sources. Each source has its own unique event parameter and response payload schema. Having to look up the docs for each one can be a PITA.

The @types/aws-lambda library gives you handler, event, context and response definitions for most of the major services that can trigger a Lambda function invocation.

By using type definitions, you get autocomplete and type checking built into your IDE. As well as making your initial authoring faster, this also helps you uncover stupid mistakes as you type them instead of having to wait until your code is run.

Some examples

Let’s look at a few different event trigger handlers to see how we can add type definitions to them.

Before we do, install the NPM package:

npm install @types/aws-lambda --save-dev

Here’s how you add type defs to your API Gateway proxy handler function:

import { APIGatewayProxyHandler } from 'aws-lambda';

export const handler: APIGatewayProxyHandler = async (event) => {
  console.log('Received event', event);
  return {
    statusCode: 200,
    body: JSON.stringify({ message: 'Success' }),
  };
};

No longer will you forget to JSON.stringify the body field in your API Gateway proxy response as you’ll now get a compiler error if you don’t assign a string to it.

Wiring up type definitions to an SNS handler is similar:

import { SNSHandler } from 'aws-lambda';

export const handler: SNSHandler = async (event) => {
  const message = JSON.stringify(event.Records[0].Sns.Message);
  console.log('received message', message);
};

No longer will you have to remember or google for the deep selector path to get at the body of your SNS or SQS message as you can easily navigate the event object inside your IDE:

It’s important to note that this is a purely compile-time library and emits no run-time Javascript after transpilation. For example, unlike fully static typed languages like C# and Java, you won’t get a casting error if you happen to specify the wrong type on your event parameter.

💌 If you enjoyed this article, you can sign up to my newsletter. I send emails every weekday where I share my guides and deep dives on building serverless solutions on AWS with hundreds of developers and architects.

Originally published at winterwindsoftware.com.

Optimise your Lambda functions using Webpack

Paul Swail — Fri, 13 Mar 2020 13:54:51 +0000

Every Friday, I will share a small tip with you on something Lambda/FaaS-related. Because Fridays are fun, and so are functions. 🥳

Today we'll cover why and how to package your Node.js Lambda functions for deployment using Webpack and the Serverless Framework. This is an approach I take for all my Lambda function development.

What problem does this solve?

The primary goal of using Webpack is to reduce the amount of code contained in the zip artifact that is uploaded when your Lambda function is being deployed. This has the benefit of reducing cold start times whenever your function code is loaded into memory.

It also has some secondary benefits:

Lower security risk as only the required parts of third party modules are deployed rather than the entire contents of the node_modules folder.
Transpilers such as TypeScript and Babel can be easily hooked into the build process via Webpack loaders.

How does it work?

If you've used Webpack in the past for front-end development, you might already know that (amongst other things) it's used to bundle multiple client-side JavaScript modules into a single file. You may not know that it can be used to do the same thing with server-side Node.js modules (it's just JavaScript after all).

It works by first configuring an entrypoint function, which in our case will be the Lambda handler function. Starting with this function, it proceeds with a static analysis checking for require and import statements, following each path to other files as needed. It bundles up each file into its own module within a single file. Webpack uses a technique called treeshaking to eliminate dead code and only import the specific functions from a module that are referenced by your application code.

You might also know that Webpack can be pretty complex to configure! Don't worry though, our configuration will be simple and we'll be using the serverless-webpack plugin to help us.

This plugin allows us to create optimised individual bundles for each Lambda function in our service.

Setting it up

You can follow the detailed instructions on the serverless-webpack plugin README, but here's a quick run-through of my standard setup. I'm assuming you already have the Serverless Framework installed and an existing serverless.yml file in place.

Install the plugin:

npm install serverless-webpack --save-dev

Add the following sections to your serverless.yml file:

# serverless.yml

custom:
  webpack:
    includeModules: false

package:
    individually: true

plugins:
  - serverless-webpack

This specifies that separate zip files for each individual function should be created rather than one for the whole service. It also tells the plugin to not package the node_modules folder in the zip but instead to trust Webpack to discover all the required modules itself and bundle them into a single .js file.

Now create a file called webpack.config.js in the same folder as your serverless.yml file. Here's what mine typically looks like for plain JavaScript projects (Typescript requires a bit more config):

// webpack.config.js
const path = require('path');
const slsw = require('serverless-webpack');

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  stats: 'minimal',
  devtool: 'nosources-source-map',
  performance: {
    hints: false,
  },
  resolve: {
    extensions: ['.js', '.jsx', '.json'],
  },
  output: {
    libraryTarget: 'commonjs2',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js',
    sourceMapFilename: '[file].map',
  },
};

That's all the config done.

To view the results, you can package your service without deploying by running serverless package in your terminal. Then open the ./.serverless folder and look at the zip files that have been created.

Handling edge cases

You may need to stray from the above configuration if your Lambda function references a module that is required at runtime but Webpack cannot discover it during its analysis. The most common cause of this is when the module contains dynamic requires, whereby the path string passed into the require statement is composed at runtime. If this is the case, you can configure serverless-webpack to use forced inclusion to always include specific modules in its bundle.

Originally published at winterwindsoftware.com.

Async Initialisation of a Lambda Handler

Paul Swail — Fri, 06 Mar 2020 20:32:58 +0000

Each Friday, I will share a small tip with you on something Lambda/FaaS-related. Because Fridays are fun, and so are functions. 🥳

Today we'll cover how to perform some asynchronous initialisation outside of your Lambda handler in Node.js.

For example, you may need to fetch configuration data from SSM Parameter Store or S3 that the main body of your function depends upon.

There are a few points to consider here before we start coding:

Our initialisation code should only be executed once — on the first "cold start" execution.
Our initialisation data may not be loaded by the time the execution of the handler function body starts.
JavaScript does not allow await calls to be defined at the root level of a module. They must happen inside a function marked as async.
If our Lambda function has Provisioned Currency enabled, you want this initialisation to be performed during the background warming phase and not when the function is serving an actual request.
If our initialisation code fails, it should be re-attempted on subsequent invocation as the first failure could be due to a transient issue.

Let's jump to the code:


const init = async () => {
  // Perform any async calls here to fetch config data.
  // We'll just dummy up a fake promise as a simulation.
  return new Promise((resolve, reject) => {
    console.log('fetching config data...');
    resolve({ myVar1: 'abc', myVar2: 'xyz' });
  });
};

const initPromise = init();

exports.handler = async (event) => {
  // Ensure init has completed before proceeding
  const functionConfig = await initPromise;
  // Start your main handler logic...
  console.log('functionConfig is set:', functionConfig);
};

The init function is responsible for asynchronously fetching an object containing all configuration data required for the function. Note that it is triggered as soon as the module is loaded and not inside the handler function. This ensures that the config is fetched as early as possible. It also should ensure that this initialisation processing will happen in the warming phase of a function with Provisioned Concurrency enabled.

The second key point here is that a promise returned from the init function is stored at the module scope and then awaited upon inside the handler. This ensures that your function can safely continue. Subsequent invocations will proceed immediately as they will be awaiting on an already resolved promise.

So far we've covered off requirements 1–4 from our list above. But what about #5?

What if an error occurs when loading the config data due to some transient issue and the init function rejects? That would mean that all subsequent executions will keep failing and you'd have a dead Lambda function container hanging around until it's eventually garbage collected.

Actually no! The Lambda runtime manages this case for you. If any errors occur in the initialisation code outside your handler, the function container is terminated and a new one is started up in a fresh state. If the transient issue has passed, your init function will resolve successfully. 😃

Thanks to Tow Kowalski, Jeremy Daly and particularly Michael Hart whose suggestions in this Twitter thread prompted this tip.

Originally published at winterwindsoftware.com.

Give developers their own AWS account

Paul Swail — Fri, 07 Feb 2020 09:31:19 +0000

If you’re in charge of a team of developers building a serverless application and your number one goal is to have them deliver quality software to users as fast as possible, then you should do whatever’s in your power to get them their own individual AWS account.

I’ve discussed approaches for managing shared accounts or projects in the past, but in this post I want to talk about sandboxed AWS accounts that are paid for by the company but are for use only by an individual developer. Here’s why I think they are a good idea…

Fully local development workflows are suboptimal or even impossible in serverless stacks

In traditional server-based development projects, developers would typically run the full stack on their local development machine. Once a feature is ready and merged into the main branch, it would be deployed (either via a CI/CD process or manually by an engineer) to a shared environment for further testing.

In serverless stacks however, while local emulators do exist for some cloud-native services (e.g. DynamoDB Local), I almost always want to use the real cloud services for a few reasons:

It’s faster and less error prone to consistently setup across the team (baseline cloud environments are more homogeneous than individual developer machines)
It reduces integration bugs that “worked on my machine” (e.g. often around IAM permissions or infra config)
I only need to configure a resource once (e.g. using CloudFormation YAML) rather than first manually creating and configuring a local emulator at dev time and having to separately configure the cloud equivalent once my feature is ready to integrate.

Given that Lambda functions are probably the resource developers will be iterating upon most frequently, we don’t want one developer deploying changes over the top of another one. You could introduce a naming convention to prevent such collisions, but that adds unnecessary complexity to your configuration management.

You give developers more exposure to infrastructure management

One of the big benefits of serverless architectures is that in many (most?) cases you should no longer need an engineer 100% dedicated to managing and operating the infrastructure of your system. But a corollary of this is that your developers will need to improve their DevOps skills, in particular around Infrastructure-as-Code and configuration management . By entrusting each developer with their own cloud account, you automatically expose them to these concepts at an early stage from within the safety of their own sandbox.

The good news is that this learning curve should be quite shallow as serverless cloud services are generally much simpler to configure than server-based systems that involve EC2 instances and VPCs.

You give less experienced developers more confidence to experiment

Often a large part of the development process involves experimentation and trial and error. If a developer is new to software development in general or just new to serverless, then this is even more the case.

I was working on a project for a client recently where the CTO wanted to jump in and fix a bug in an API to help out his team who were busy working on other projects. He is a highly experienced developer and architect but was new to serverless. At the time, he hadn’t his own personal AWS development account set up. He was easily able to identify and make the necessary changes to the codebase to fix the bug but couldn’t deploy and test his changes. There was a shared DEV account available but this was being used as the backend for a mobile app that was due to be demoed to a client the following day and so was nervous about breaking it. Basically he needed a sandbox to safely experiment in before being confident enough to deploy to this shared account.

A much more egregious example I witnessed was a large enterprise who used a single AWS account for a whole department of developers who were responsible for delivery of a range of products. This one account contained a mishmash of resources for different projects/products, personal resources for individual developers and resources for all shared environments (including production 😱). Being productive as a developer in those conditions was a real challenge, never mind all the security concerns!

Common objections

Isn’t this a big admin overhead to have to set this up?

There is a small overhead but this only needs to be done once whenever a new developer joins your team. Each developer can re-use the same personal AWS account for different projects. You can use AWS Organizations to provision the account from a master account without needing to separately enter credit card details, etc. If you are doing this very frequently, you can create a script using the AWS CLI to automate the entire account provisioning process.

How will we control costs?

I have 2 recommendations on this point:

Give developers read-only access to the Billing Dashboard for their own account (this isn’t enabled by default). Not only does this treats them like responsible adults who can manage their own money (!) but also encourages them to be curious about the costs of the cloud services they are consuming.
Developers aren’t always responsible adults! Be sure to set up a billing alert with a sensible threshold so a senior person can be notified if a rogue developer starts mining bitcoin (or more likely, accidentally triggers an infinitely recursive loop of Lambda invocations).

Another mitigation to the cost objection (that’s specific to fully serverless systems) is that serverless cloud resources have pay-per-use pricing. So you won’t be getting billed for an EC2 instance that a developer who infrequently helps out your team forgot to turn off. And you’ll often find that each developer’s usage of a service won’t exceed the free tier quota, so will be costing you next to nothing.

What about security?

Developer accounts will be completely isolated from other AWS accounts within your organisation so cannot interfere with important resources. If you’re concerned about the actions developers take within their personal account, you can use AWS Organizations to set a Service Control Policy on all child accounts to limit the specific AWS services they have access to.

Our IT department simply won’t allow this

Big enterprise red tape can be a real PITA for dev teams looking to ship quality software fast to their users. But it is what it is and if this is you, you have my sympathies. I don’t have a great recommendation for you here other than to use whatever influence you do have to lobby whoever is making this decision and educate them on the overall benefits of building serverless apps for your organisation as a whole.

In your place of work, how are AWS accounts allocated? Do you have your own personal one? Let me know in the comments...

💌 If you enjoyed this article, you can sign up to my newsletter. I send emails a few times a week where I share my guides and deep dives on building serverless solutions on AWS with hundreds of developers and architects.

Originally published at winterwindsoftware.com.

How to access VPC and internet resources from Lambda without paying for a NAT Gateway

Paul Swail — Fri, 29 Nov 2019 12:23:56 +0000

AWS recommend you don't connect Lambda functions to a VPC unless absolutely necessary. This is solid advice because doing so brings several limitations that a standard function doesn’t suffer from.

One of these limitations is that your Lambda function can no longer access the internet. What many people don’t realise is that communicating with other AWS resources inside your account from your function also requires internet access (think S3, SNS, SQS, etc). This makes many common VPC use cases tricky to implement with Lambda.

Let’s take the example of a scheduled Lambda function that runs a daily report by performing a SQL query against an RDS database. Based on this query result, it then adds messages to an SQS queue for processing. VPC access is required to access the RDS database and internet access is required to post to SQS. How can you do both?

The typically recommended solution is to set up a NAT Gateway which allows your VPC-enabled Lambda running in private subnets to connect to a public subnet that has an internet gateway set up.

Eugh! 😩 The last thing I want to be doing is network configuration. Never mind the extra billing cost that I’ll incur since NAT Gateways are billed both by the hour and per GB of data processed. Pretty far from the serverless way.

Using a VPC proxy Lambda function

An alternative solution I’ve used when faced with this problem is to create what I call a “VPC proxy Lambda function”. Instead of having one Lambda function that does all your work, you have two. The first Lambda function, let’s call it runDailyReport (per our earlier example), is the entrypoint. It's triggered by an event (e.g. a CloudWatch schedule rule) and it's NOT configured to run inside the VPC. Its job is to orchestrate all I/O calls that need to be performed.

The second Lambda function is our VPC proxy, let’s call it dbGetReportResults. It’s configured to run inside the VPC and its sole responsibility is to connect to the RDS cluster, perform a query and return the result. It has no triggers configured.

The key thing here is that the runDailyReport function uses the AWS Lambda API to synchronously invoke the dbGetReportResults function, with InvocationType: "RequestResponse". This means that it will wait for the response to be returned before proceeding. Once it gets the result back, it can then parse it and post jobs to SQS.

Despite executing inside the VPC, the dbGetReportResults function is still accessible to functions outside the VPC because the calling function (runDailyReport) interacts with the AWS Lambda service API, which, like all the other AWS service APIs, is internet facing. It does not need to connect directly to the underlying container inside the VPC where the target function is executed.

Limitations

There are a few limitations to be aware of with this approach.

Firstly, since you’re now using 2 Lambdas instead of 1, you will be paying for the execution time of both. While your VPC proxy function is executing and waiting on the database query to return, your entrypoint function will also still be executing (albeit it will be idle while waiting for the proxy function to return). This is why you might hear people saying that one Lambda function directly invoking another Lambda synchronously is an anti-pattern. I’d usually agree with that, but this use case is a valid exception IMO.

Another minor limitation is that there will be a small additional latency as you need to account for the delay in invoking 2 functions in series (cold or warm start) instead of 1. This should not be an issue if your use case is not user facing.

💌 If you enjoyed this article, you can sign up to my weekly newsletter on building serverless apps in AWS.
Originally published at winterwindsoftware.com.

Comparing multi and single table approaches to designing a DynamoDB data model

Paul Swail — Fri, 22 Nov 2019 09:03:25 +0000

DynamoDB is the predominant general purpose database in the AWS serverless ecosystem. Its low operational overhead, simple provisioning and configuration, streaming capability, pay-per-usage pricing and promise of near-infinite scaling make it a popular choice amongst developers building apps using Lambda and API Gateway as opposed to taking the more traditional RDBMS route.

When it comes to designing your data model in DynamoDB, there are two distinct design approaches you can take: multi-table or
single-table. In this article, I will explore how both design approaches can impact the Total Cost of Ownership of your application over the lifecycle of its delivery
and hopefully help you decide which approach is right for your needs.

What are the key differences between each approach?

Let’s start with an overview of what each involves:

Multi-table — One table per each type of entity. Each item (row) maps to a single instance of that entity and attributes (columns) are consistent across every item. This is the way most people are used to thinking about data models and, in my anecdotal experience, the most common approach used.
Single-table — One table serves the entire application or service and holds multiple types of entities within it. Each item has different attributes set on it depending on its entity type. I find this approach to be less common (at least in terms of articles and code examples on the internet) and is definitely a harder concept to grasp for most newcomers to DynamoDB. But crucially, this approach is the one that the AWS DynamoDB team espouses (somewhat without qualification) in their official docs:

You should maintain as few tables as possible in a DynamoDB application. Most well designed applications require only one table.

The primary benefits of single-table design are faster read and write performance at scale and lower cloud bill costs. At the core of its design pattern is the concept of “index overloading”. This means that a single index (both Global Secondary and Local Secondary) on your one table can be used to support several different query patterns.
This enables SQL-like JOIN queries to be performed, whereby multiple related entities are fetched in a single round trip to the database. This pattern is not possible in a one entity per table model.
Secondly, since indexes are multi-purpose, less indexes are needed in total. This means there are fewer indexes to update whenever a write is performed, resulting in both faster writes and a lower billing cost.

I appreciate this has been a very brief introduction to single-table design, so if you’re totally new to it and are still wondering “how can you squeeze different entities into the same database table?”, please check out the links in the resources section below.

My experience with both approaches

Up until mid-2019, I had only ever used a multi-table approach to data modelling in DynamoDB and more generally in NoSQL databases as a whole (I previously used MongoDB regularly). Since then, I’ve worked on several greenfield projects that use a single-table data model to underpin transaction-oriented apps.

In the remaining sections, I’ll walk through each phase involved in a typical project delivery as it relates to your application’s database.

Big design up front

Before any database tables are provisioned or a single line of code is written, the first step is to design your data model. The official DynamoDB docs state the following general guideline for any type of NoSQL design:

… you shouldn’t start designing your schema until you know the questions it will need to answer. Understanding the business problems and the application use cases up front is essential.

This second sentence struck me when I first read it. I work almost exclusively on agile-delivered projects where changes related to client feedback are the norm. Does this then rule out DynamoDB (and NoSQL in general) for me altogether on these projects?
The short answer to this is “no” and there are strategies for managing changes (which I’ll get to later), but there’s no getting away from the fact that there is more Big Design Up Front with DynamoDB versus using a SQL database. But the “serverlessness” benefits of DynamoDB over an RDBMS that I described in my opening paragraph above outweigh the impact of this upfront design effort IMHO.

In terms of tools, I use a spreadsheet to define my design and have seen many DynamoDB experts doing the same. AWS have recently released a new tool DynamoDB NoSQL Workbench that as of this writing is in early preview, but will hopefully provide a bit more structure to the data modelling design process.

The design process

So what is the process for creating your data model? Jeremy Daly has a great list of 20 steps for designing a DynamoDB model using a single-table approach that I recommend you check out as it’s a quick read.
Steps 11–14 in particular should give you a flavour of the level of rigour required:

Deciding on the composition of your index fields is core to the whole design process and will involve many iterations. You need to consider the entirety of your access patterns across all entities in order to come up with your final design.

The main schema difference you will see between single and multi-table models is that single-table will have generically named attributes that are used to form the table’s partition and sort key. This is required because different entity types will likely have differently named primary key fields. A common convention is to use attributes named pk and sk that map to the table’s partition and sort keys respectively. Similar generically named attributes may be used for composite keys that make up GSIs or LSIs.

Provisioning and configuration management

So now we have our data models designed, it’s now time to provision our tables. This is probably the easiest step of the whole development process.
DynamoDB has good CloudFormation support which makes Infrastructure-as-Code a breeze. With a few lines of YAML and a CLI deploy command you can quickly provision your DynamoDB tables and indexes along with associated IAM access control privileges in less than a minute. I use the Serverless Framework which allows raw CloudFormation to be embedded in the resources section.

This area is one where the single-table approach wins out in terms of less configuration to manage and faster provisioning — I only need to define one table and pass its name into my Lambda functions as an environment variable. In the multi-table approach, I have config and environment variables for each individual table. A minor benefit in the entire scheme of things, but still nice.

Implementing data access in the codebase

Your database is now deployed and it’s time to start talking to it from your application.
Chances are you will have domain entity objects that you pass around in your code (e.g. in API request/response payloads or SNS / SQS messages). If you need to persist these entities to your database, you can use one of the higher-level AWS DynamoDB SDKs (such as the DocumentClient for Node.js) to do so. But there are a few key differences between multi and single table designs here…

Writing objects to the database

In a multi-table design, you can often just write your in-memory domain object directly to the database as-is without any mapping. Fields on your object will become attributes in your DynamoDB item. Occasionally you may need to create a concatenated composite field that’s used in an index in order to support a particular filtering or sorting requirement.
In a single-table design however, there will always be some mapping you need to do at write-time. Specifically, you will need to add 2 new fields pk and sk to your domain object before persisting it to DynamoDB. If you’re using other generic composite index fields, then you’ll also need to do the same for each of them. The values of these fields need to match the formats defined in your data model spreadsheet. I find that I usually need to concatenate a static prefix (that uniquely identifies the entity type and prevents collisions) to one or more fields from my domain object that I need to filter or sort on.

Partial item updates are more complex again. Given that in single-table design there is data duplication within each item, if you are using the DynamoDB UpdateItem API to update a single field, you need to check whether that field is also used within a composite indexed field, and if so, also update the value of the composite field. I’ve forgot about this several times and it can be quite difficult to remedy.

Reading objects from the database

When you fetch items back from DynamoDB (via GetItem or Query API calls), you will almost always want to strip off the composite indexed fields before, say, returning the entity to the client who is calling your API. Unfortunately, the DynamoDB API calls do not allow you to blacklist attributes that you don’t want to return. So instead, you either have to whitelist all the other fields that you do wish to return (by using a ProjectionExpression) or you do the blacklisting in your application code after the query returns. I usually take the latter option as it’s less code to maintain, despite being slightly less performant (as more data is being returned than what I need).

Strategies for controlling code complexity in a single-table design

For both reads and writes, you will find yourself doing a lot of string concatenations using the same prefixes and separator characters. This can be quite error prone. For this reason, I recommend you keep all data-access code for each entity type within a single module/file so you can quickly reference how an entity was created when you are writing a function to query or update it.

In serverless apps, I usually structure my code such that a Lambda handler would handoff to a model / service module which would then be responsible for doing data access as well as talking to any other downstream services (SNS, etc). If your data access code becomes sufficiently complex (which it easily can once composite fields are introduced), there is a case for using the repository pattern whereby you create modules whose sole responsibility is to perform DynamoDB operations for a particular entity type.

Another recommendation for increasing the maintainability of your data access code is to keep your data model design spreadsheet up-to-date and have it reviewed alongside the code as part of your pull request process. I’ve found it helpful to have an “Implementation Status” flag column or colour code in my design spreadsheet as part of each query pattern showing whether it’s been implemented yet.

Schema migrations

This is the scenario the official AWS docs warned you about. Those business use cases that you fully understood at the project outset have changed! You need to make changes to your existing access patterns — maybe change a sort order or filter on a different field.
Broadly, the solution to this will involve either or both of the following:

Creating a new GSI/LSI index pointing at the new fields (optionally also dropping an existing index that’s no longer needed)
Writing a migration script that scans a table and performs per-item updates, such as amending the value of a composite indexed field.

Once the new indexes/composite indexed fields are in place, then the application code updates can be deployed. You may want to then run a cleanup script to remove the old composite fields / indexes.

Such a migration script can be difficult to manage, especially in a single-table design which is highly dependent on composite indexed fields.
Full table scan operations can take a long time to complete. So while it’s running, your database will be in an inconsistent state with some items patched and some not. You may need to write your script such that it can operate on smaller batches/partitions and ensure that it’s idempotent (e.g. by storing state somewhere to show what migrations have been applied or what items were already patched) . Also, I could not find any well-known tools that currently help with this in the same way as the likes of Rails ActiveRecord Migrations works with SQL databases.

I haven’t yet hit this issue in a post go-live production environment so I haven’t explored in-depth what solutions are currently available to this. If you have a good strategy for managing schema migrations, then please let me know in the comments.

Integrating with other data stores

Another concern that affects single-table designs is that some managed services that have built-in integrations for exporting data out of DynamoDB (for analytics) expect each table to map to a single domain entity. Exporting a single table that contains entities of varying shapes just won’t work without some custom-built intermediate step to perform a transform. An example of this is the DynamoDB to Redshift integration . This might be something you need to consider when choosing your design approach.

When should you use either approach?

We’ve covered the impact of multi-table vs single-table design approach on each stage of the delivery lifecycle from design right through to post-go-live change management.
The big question that now remains is when should you choose one approach over the other?

The overly simplistic AWS official line of “Most well designed applications require only one table” doesn’t do the nuance of this decision justice IMHO.
It implies that if you don’t use the single table approach that your application is not well designed. But their definition of “well designed” only considers performance, scaling and billing costs and neglects the other considerations that go into the Total Cost of Ownership of an application.

So for me, it comes down to answering this question — what do you want to optimise for:

time to market and flexibility of requirements; or:
performance, scalability and efficient billing cost?

One of the core tenets of the serverless movement is that it allows developers to focus more on the business problem at hand and much less on the technical and operational concerns that they’ve had to spend time on in the past working in server-based architectures. Another core tenet is near unlimited scalability. Often both of these are in tandem, but in this debate they are pulling against each other.

The multi-table approach is an easier on-ramp for developers coming from an RDBMS background (which is the majority of developers).
Adding the single table design approach on top of that cranks up the steepness of the learning curve. Add in the rigidity enforced by designing overloaded indexes and the overhead if any migrations need to be performed, then I think it’s fair to say that for most teams, they would ship an app faster using the multi-table approach

With a multi-table model, I would argue that your team will be less dependent on the presence of a resident “DynamoDB modelling expert” in order to implement or approve any changes to the application’s data access.
I’m sure most of you have experienced part of an application architecture or codebase that you're afraid to touch because you don’t really understand it and seems a bit like magic.

All that said, once you do get the hang of the single-table approach and learn new strategies for creating composite indexes to support new query patterns, it’s undoubtedly very powerful.
Your code only needs to make one fast database round-trip to fetch a batch of related entities. And you get that warm fuzzy feeling of confidence that your app performance and billing costs are as optimised as they can be. (But remember the cost of your engineer’s time usually trumps the cost of your cloud service bill).

Learn more

If you’d like to learn more about data modelling in DynamoDB, here’s a list of resources that have helped me:

Thanks to Darren Gibney for providing review on this post.

💌 If you enjoyed this article, you can sign up to my weekly newsletter on building serverless apps in AWS.
Originally published at winterwindsoftware.com.

How to build a serverless photo upload service with API Gateway

Paul Swail — Fri, 25 Oct 2019 15:17:42 +0000

So you’re building a REST API and you need to add support for uploading files from a web or mobile app. You also need to add a reference to these uploaded files against entities in your database, along with metadata supplied by the client.

In this article, I'll show you how to do this using AWS API Gateway, Lambda and S3. We'll use the example of an event management web app where attendees can login and upload photos associated with a specific event along with a title and description. We will use S3 to store the photos and an API Gateway API to handle the upload request. The requirements are:

User can login to the app and view a list of photos for a specific event, along with each photo's metadata (date, title, description, etc).
User can only upload photos for the event if they are registered as having attended that event.
Use Infrastructure-as-Code for all cloud resources to make it easy to roll this out to multiple environments. (No using the AWS Console for mutable operations here 🚫🤠)

Considering implementation options

Having built similar functionality in the past using non-serverless technologies (e.g. in Express.js), my initial approach was to investigate how to use a Lambda-backed API Gateway endpoint that would handle everything: authentication, authorization, file upload and finally writing the S3 location and metadata to the database.
While this approach is valid and achievable, it does have a few limitations:

You need to write code inside your Lambda to manage the multipart file upload and the edge cases around this, whereas the existing S3 SDKs are already optimized for this.
Lambda pricing is duration-based so for larger files your function will take longer to complete, costing you more.
API Gateway has a payload size hard limit of 10MB. Contrast that to the S3 file size limit of 5GB.

Using S3 presigned URLs for upload

After further research, I found a better solution involving uploading objects to S3 using presigned URLs as a means of both providing a pre-upload authorization check and also pre-tagging the uploaded photo with structured metadata.

The diagram below shows the request flow from a web app.

The main thing to notice is that from the web client’s point of view, it’s a 2-step process:

Initiate the upload request, sending metadata related to the photo (e.g. eventId, title, description, etc). The API then does an auth check, executes business logic (e.g. restricting access only to users who have attended the event) and finally generates and responds with a secure presigned URL.
Upload the file itself using the presigned URL.

I’m using Cognito as my user store here but you could easily swap this out for a custom Lambda Authorizer if your API uses a different auth mechanism.

Let's dive in...

Step 1: Create the S3 bucket

I use the Serverless Framework to manage configuration and deployment of all my cloud resources. For this app, I use 2 separate "services" (or stacks), that can be independently deployed:

infra service: this contains the S3 bucket, CloudFront distribution, DynamoDB table and Cognito User Pool resources.
photos-api service: this contains the API Gateway and Lambda functions.

You can view the full configuration of each stack in the Github repo, but we’ll cover the key points below.

The S3 bucket is defined as follows:

resources:
  Resources:
    PhotosBucket:
        Type: AWS::S3::Bucket
        Properties:
            BucketName: !Sub ‘${self:custom.photosBucketName}’
            AccessControl: Private
            CorsConfiguration:
                CorsRules:
                -   AllowedHeaders: [‘*’]
                    AllowedMethods: [‘PUT’]
                    AllowedOrigins: [‘*’]

The CORS configuration is important here as without it your web client won’t be able to perform the PUT request after acquiring the signed URL.
I’m also using CloudFront as the CDN in order to minimize latency for users downloading the photos. You can view the config for the CloudFront distribution here. However, this is an optional component and if you’d rather clients read photos directly from S3 then you can change the AccessControl property above to be PublicRead.

Step 2: Create "Initiate Upload" API Gateway endpoint

Our next step is to add a new API path that the client endpoint can call to request the signed URL. Requests to this will look like so:

POST /events/{eventId}/photos/initiate-upload
{
    "title": "Keynote Speech",
    "description": "Steve walking out on stage",
    "contentType": "image/png"
}

Responses will contain an object with a single s3PutObjectUrl field that the client can use to upload to S3. This URL looks like so:

https://s3.eu-west-1.amazonaws.com/eventsapp-photos-dev.sampleapps.winterwindsoftware.com/uploads/event_1234/1d80868b-b05b-4ac7-ae52-bdb2dfb9b637.png?AWSAccessKeyId=XXXXXXXXXXXXXXX&Cache-Control=max-age%3D31557600&Content-Type=image%2Fpng&Expires=1571396945&Signature=F5eRZQOgJyxSdsAS9ukeMoFGPEA%3D&x-amz-meta-contenttype=image%2Fpng&x-amz-meta-description=Steve%20walking%20out%20on%20stage&x-amz-meta-eventid=1234&x-amz-meta-photoid=1d80868b-b05b-4ac7-ae52-bdb2dfb9b637&x-amz-meta-title=Keynote%20Speech&x-amz-security-token=XXXXXXXXXX

Notice in particular these fields embedded in the query string:

x-amz-meta-XXX — These fields contain the metadata values that our initiateUpload Lambda function will set.
x-amz-security-token — this contains the temporary security token used for authenticating with S3
Signature — this ensures that the PUT request cannot be altered by the client (e.g. by changing metadata values)

The following extract from serverless.yml shows the function configuration:

# serverless.yml
service: eventsapp-photos-api
…
custom:
    appName: eventsapp
    infraStack: ${self:custom.appName}-infra-${self:provider.stage}
    awsAccountId: ${cf:${self:custom.infraStack}.AWSAccountId}
    apiAuthorizer:
        arn: arn:aws:cognito-idp:${self:provider.region}:${self:custom.awsAccountId}:userpool/${cf:${self:custom.infraStack}.UserPoolId}
    corsConfig: true

functions:
…
    httpInitiateUpload:
        handler: src/http/initiate-upload.handler
        iamRoleStatements:
        -   Effect: Allow
            Action:
                - s3:PutObject
            Resource: arn:aws:s3:::${cf:${self:custom.infraStack}.PhotosBucket}*
        events:
        - http:
            path: events/{eventId}/photos/initiate-upload
            method: post
            authorizer: ${self:custom.apiAuthorizer}
            cors: ${self:custom.corsConfig}

A few things to note here:

The httpInitiateUpload Lambda function will handle POST requests to the specified path.
The Cognito user pool (output from the infra stack) is referenced in the function’s authorizer property. This makes sure requests without a valid token in the Authorization HTTP header are rejected by API Gateway.
CORS is enabled for all API endpoints
Finally, the iamRoleStatements property creates an IAM role that this function will run as. This role allows PutObject actions against the S3 photos bucket. It is especially important that this permission set follows the least privilege principle as the signed URL returned to the client contains a temporary access token that allows the token holder to assume all the permissions of the IAM role that generated the signed URL.

Now let's look at the handler code:

import S3 from 'aws-sdk/clients/s3';
import uuid from 'uuid/v4';
import { InitiateEventPhotoUploadResponse, PhotoMetadata } from '@common/schemas/photos-api';
import { isValidImageContentType, getSupportedContentTypes, getFileSuffixForContentType } from '@svc-utils/image-mime-types';
import { s3 as s3Config } from '@svc-config';
import { wrap } from '@common/middleware/apigw';
import { StatusCodeError } from '@common/utils/errors';

const s3 = new S3();

export const handler = wrap(async (event) => {
    // Read metadata from path/body and validate
  const eventId = event.pathParameters!.eventId;
  const body = JSON.parse(event.body || '{}');
  const photoMetadata: PhotoMetadata = {
    contentType: body.contentType,
    title: body.title,
    description: body.description,
  };
  if (!isValidImageContentType(photoMetadata.contentType)) {
    throw new StatusCodeError(400, `Invalid contentType for image. Valid values are: ${getSupportedContentTypes().join(',')}`);
  }
  // TODO: Add any further business logic validation here (e.g. that current user has write access to eventId)

  // Create the PutObjectRequest that will be embedded in the signed URL
  const photoId = uuid();
  const req: S3.Types.PutObjectRequest = {
    Bucket: s3Config.photosBucket,
    Key: `uploads/event_${eventId}/${photoId}.${getFileSuffixForContentType(photoMetadata.contentType)!}` ,
    ContentType: photoMetadata.contentType,
    CacheControl: 'max-age=31557600',  // instructs CloudFront to cache for 1 year
    // Set Metadata fields to be retrieved post-upload and stored in DynamoDB
    Metadata: {
      ...(photoMetadata as any),
      photoId,
      eventId,
    },
  };
  // Get the signed URL from S3 and return to client
  const s3PutObjectUrl = await s3.getSignedUrlPromise('putObject', req);
  const result: InitiateEventPhotoUploadResponse = {
    photoId,
    s3PutObjectUrl,
  };
  return {
    statusCode: 201,
    body: JSON.stringify(result),
  };
});

The s3.getSignedUrlPromise is the main line of interest here. It serializes a PutObject request into a signed URL.

I'm using a wrap middleware function in order to handle cross-cutting API concerns such as adding CORS headers and uncaught error logging.

Step 3: Uploading file from the web app

Now to implement the client logic. I've created a very basic (read: ugly) create-react-app example (code here). I used Amplify's Auth library to manage the Cognito authentication and then created a PhotoUploader React component which makes use of the React Dropzone library:

// components/Photos/PhotoUploader.tsx
import React, { useCallback } from 'react';
import { useDropzone } from 'react-dropzone';
import { uploadPhoto } from '../../utils/photos-api-client';

const PhotoUploader: React.FC<{ eventId: string }> = ({ eventId }) => {
  const onDrop = useCallback(async (files: File[]) => {
    console.log('starting upload', { files });
    const file = files[0];
    try {
      const uploadResult = await uploadPhoto(eventId, file, {
        // should enhance this to read title and description from text input fields.
        title: 'my title',
        description: 'my description',
        contentType: file.type,
      });
      console.log('upload complete!', uploadResult);
      return uploadResult;
    } catch (error) {
      console.error('Error uploading', error);
      throw error;
    }
  }, [eventId]);
  const { getRootProps, getInputProps, isDragActive } = useDropzone({ onDrop });

  return (
    <div {...getRootProps()}>
      <input {...getInputProps()} />
      {
        isDragActive
          ? <p>Drop the files here ...</p>
          : <p>Drag and drop some files here, or click to select files</p>
      }
    </div>
  );
};

export default PhotoUploader;

// utils/photos-api-client.ts
import { API, Auth } from 'aws-amplify';
import axios, { AxiosResponse } from 'axios';
import config from '../config';
import { PhotoMetadata, InitiateEventPhotoUploadResponse, EventPhoto } from '../../../../services/common/schemas/photos-api';

API.configure(config.amplify.API);

const API_NAME = 'PhotosAPI';

async function getHeaders(): Promise<any> {
  // Set auth token headers to be passed in all API requests
  const headers: any = { };
  const session = await Auth.currentSession();
  if (session) {
    headers.Authorization = `${session.getIdToken().getJwtToken()}`;
  }
  return headers;
}

export async function getPhotos(eventId: string): Promise<EventPhoto[]> {
  return API.get(API_NAME, `/events/${eventId}/photos`, { headers: await getHeaders() });
}

export async function uploadPhoto(
  eventId: string, photoFile: any, metadata: PhotoMetadata,
): Promise<AxiosResponse> {
  const initiateResult: InitiateEventPhotoUploadResponse = await API.post(
    API_NAME, `/events/${eventId}/photos/initiate-upload`, { body: metadata, headers: await getHeaders() },
  );
  return axios.put(initiateResult.s3PutObjectUrl, photoFile, {
    headers: {
      'Content-Type': metadata.contentType,
    },
  });
}

The uploadPhoto function in the photos-api-client.ts file is the key here. It performs the 2-step process we mentioned earlier by first calling our initiate-upload API Gateway endpoint and then making a PUT request to the s3PutObjectUrl it returned. Make sure that you set the Content-Type header in your S3 put request, otherwise it will be rejected as not matching the signature.

Step 4: Pushing photo data into database

Now that the photo has been uploaded, the web app will need a way of listing all photos uploaded for an event (using the getPhotos function above).

To close this loop and make this query possible, we need to record the photo data in our database. We do this by creating a second Lambda function processUploadedPhoto that is triggered whenever a new object is added to our S3 bucket.

Let's look at its config:


# serverless.yml
service: eventsapp-photos-api
…

functions:
…
    s3ProcessUploadedPhoto:
        handler: src/s3/process-uploaded-photo.handler
        iamRoleStatements:
            -   Effect: Allow
                Action:
                    - dynamodb:Query
                    - dynamodb:Scan
                    - dynamodb:GetItem
                    - dynamodb:PutItem
                    - dynamodb:UpdateItem
                Resource: arn:aws:dynamodb:${self:provider.region}:${self:custom.awsAccountId}:table/${cf:${self:custom.infraStack}.DynamoDBTablePrefix}*
            -   Effect: Allow
                Action:
                    - s3:GetObject
                    - s3:HeadObject
                Resource: arn:aws:s3:::${cf:${self:custom.infraStack}.PhotosBucket}*
        events:
            - s3:
                bucket: ${cf:${self:custom.infraStack}.PhotosBucket}
                event: s3:ObjectCreated:*
                rules:
                    - prefix: uploads/
                existing: true

It's triggered off the s3:ObjectCreated event and will only fire for files added beneath the uploads/ top-level folder.
In the iamRoleStatements section, we are allowing the function to write to our DynamoDB table and read from the S3 Bucket.

Now let's look at the function code:

import { S3Event } from 'aws-lambda';
import S3 from 'aws-sdk/clients/s3';
import log from '@common/utils/log';
import { EventPhotoCreate } from '@common/schemas/photos-api';
import { cloudfront } from '@svc-config';
import { savePhoto } from '@svc-models/event-photos';

const s3 = new S3();

export const handler = async (event: S3Event): Promise<void> => {
  const s3Record = event.Records[0].s3;

  // First fetch metadata from S3
  const s3Object = await s3.headObject({ Bucket: s3Record.bucket.name, Key: s3Record.object.key }).promise();
  if (!s3Object.Metadata) {
    // Shouldn't get here
    const errorMessage = 'Cannot process photo as no metadata is set for it';
    log.error(errorMessage, { s3Object, event });
    throw new Error(errorMessage);
  }
  // S3 metadata field names are converted to lowercase, so need to map them out carefully
  const photoDetails: EventPhotoCreate = {
    eventId: s3Object.Metadata.eventid,
    description: s3Object.Metadata.description,
    title: s3Object.Metadata.title,
    id: s3Object.Metadata.photoid,
    contentType: s3Object.Metadata.contenttype,
    // Map the S3 bucket key to a CloudFront URL to be stored in the DB
    url: `https://${cloudfront.photosDistributionDomainName}/${s3Record.object.key}`,
  };
  // Now write to DDB
  await savePhoto(photoDetails);
};

The event object passed to the Lambda handler function only contains the bucket name and key of the object that triggered it. So in order to fetch the metadata, we need to use the headObject S3 API call.
Once we've extracted the required metadata fields, we then construct a CloudFront URL for the photo (using the CloudFront distribution's domain name passed in via an environment variable) and save to DynamoDB.

Future enhancements

A potential enhancement that could be made to the upload flow is to add in an image optimization step before saving it to the database. This would involve a having a Lambda function listen for S3:ObjectCreated events beneath the upload/ key prefix which then reads the image file, resizes and optimizes it accordingly and then saves the new copy to the same bucket but under a new optimized/ key prefix. The config of our Lambda function that saves to the database should then be updated to be triggered off this new prefix instead.

💌 If you enjoyed this article, you can sign up to my weekly newsletter on building serverless apps in AWS.
Originally published at winterwindsoftware.com.

Migrating authentication from Express.js to API Gateway using a Lambda Authorizer

Paul Swail — Wed, 17 Apr 2019 20:19:20 +0000

This is part 6 in the series Migrating a Monolithic SaaS App to Serverless — A Decision Journal.

Before I can migrate any of the routes from my Express.js API to API Gateway + Lambda, I first need to implement an authentication and authorization mechanism such that the API Gateway endpoints respect the same auth logic as their legacy API counterparts.

My constraints for this are as follows:

Keep the same back-end MongoDB user and session store that the legacy app is using as I want to avoid/minimise code changes to the legacy app. This rules out using dedicated auth services such as AWS Cognito or Auth0 which would be my first stops for auth in a greenfield app.
Clients authenticate to the existing API by first obtaining a session token via a call to a login endpoint and then by providing this token in subsequent requests either in the Cookie or Authorization HTTP headers. This behaviour needs to be reproduced in my API Gateway implementation.
The login endpoint itself (i.e. how the token is obtained in the first place) is out of scope for this task, and the legacy login endpoint will continue to be used for now.
This will be an interim solution as my longer-term goal for this migration process is to replace MongoDB as my back-end data store.

Using a Lambda Authorizer to authenticate API requests

API Gateway allows you to define a Lambda Authorizer to execute custom authentication and authorization logic before allowing a client access to the actual API route they have requested. A Lambda Authorizer function is somewhat similar to a middleware in Express.js in that it gets called before the main route handler function, it can reject a request outright, or if it allows the request to proceed, it can enhance the request event with extra data that the main route handler can then reference (e.g. user and role information).

Authentication vs Authorization

Before we dive into the implementation detail, I want to make clear the distinction between these related “auth” concepts as they are often conflated and the AWS naming of “Lambda Authorizer” does not help here:

Authentication is the process of verifying who you are. When you log on to a computer or app with a username and password you are authenticating.
Authorization is the process of verifying that you have access to something. Gaining access to a resource because the permissions configured on it allow you access is authorization.

(What is the difference between authentication and authorization? - Server Fault)

If you are implementing a Lambda Authorizer, your function will always need to perform authentication (i.e. ensure you are who you say you are) but it does not necessarily need to perform authorization (i.e. check that you have permissions to access the resource you are requesting).

In my case, I decided (for now) that my Lambda Authorizer would only perform authentication and that the authorization logic will reside in the route handler functions as the necessary permissions vary across different routes. As I start migrating more routes over to Lambda, I may then decide to move common authorization logic to the shared Lambda Authorizer.

For an in-depth look at different strategies for using Lambda Authorizers, check out The Complete Guide to Custom Authorizers with AWS Lambda and API Gateway.

Reverse engineering the Express authentication logic

My legacy API uses the Passport.js and express-session middlewares.
I could potentially just import these modules into my Lambda Authorizer function. However, I decided against this for a few reasons:

These modules were built specifically for use with Express so I would end up having to hack a way of invoking them in a non-standard way from a Lambda.
I don’t want to add a raft of new dependencies to my Lambda and incur the extra coldstart overhead and increased security threat that this would bring.

So I decided to inspect the code on Github for these modules and extract the necessary logic into my Lambda function. I’ll not share the full implementation code here, but it follows these steps to process a request:

Fetch token from HTTP request header (either the Cookie or the Authorization header).
Use session secret to decrypt token and extract SessionID from it.
Using SessionID, fetch session object from MongoDB and get user data stored inside it.
Add user data to the request context.

Allowing and denying requests

If a request is successfully authenticated, in order to tell API Gateway it can proceed with invoking the handler for the requested route, the Lambda Authorizer function needs to return a response which contains an IAM policy document that allows the caller invoke access to the handler.

Here’s an example of a response the Lambda Authorizer function returns for an allowed request:

{
    "principalId": "my_user_id",
    "policyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "execute-api:Invoke",
                "Effect": "Allow",
                "Resource": "*"
            }
        ]
    },
    "context": {
        "userId": "my_user_id",
        "customerAccountId": "my_customer_account_id",
        "fullName": "John Smith",
        "roles": "[]"
    }
}

Notice here the context object where I provide further information that is stored against the user record in MongoDB. API Gateway makes this context data available to the handler function (which we’ll cover below).

That’s the happy path covered, but there are several reasons why a request could be rejected, e.g.

No token provided
Invalid token provided
Session expired

In each of these cases, I want to send back a HTTP 401 Unauthorized status code to the client, but it wasn’t immediately obvious from reading the AWS docs how I could do this.

In normal API Gateway Lambda handlers, there is a statusCode field in the response that you can set, but Lambda Authorizer responses don’t work that way. The examples show throwing an error (or if you’re using legacy Node, passing an Error in the callback). However, when I tested this, API Gateway returned a 403 error. I couldn’t work out what was going on until I realised that the actual string in the error message needs to match one of API Gateway’s built-in message -> status code mappings. I hadn’t realised this significance and had been using my own custom error strings and API Gateway didn’t know what to do with those so it just defaulted to returning a 403.

import { CustomAuthorizerEvent, AuthResponse } from 'aws-lambda';

/** Built-in error messages that API Gateway auto-maps to HTTP status codes */
export enum APIGatewayErrorMessage {
    /** 401 */
    Unauthorized = 'Unauthorized',
    /** 403 */
    AccessDenied = 'Access Denied',
}

/** Lambda Authorizer handler */
export const handler = async (event: CustomAuthorizerEvent): Promise<AuthResponse> => {
    if (!event.headers) {
        // No token provided
        throw new Error(APIGatewayErrorMessage.Unauthorized);
    }
    // first check Authorization bearer header
    if (event.headers.Authorization) {
        const [key, val] = event.headers.Authorization.split(' ');
        if (key && key.toLowerCase() === 'bearer' && val) {
            return authenticateToken(val);
        }
        // Badly formed header
        throw new Error(APIGatewayErrorMessage.Unauthorized);
    }
    // ... rest of auth logic
};

Wiring up auth logic to a private endpoint

So far I’ve covered the implementation of the Lambda Authorizer but not shown how you connect it to the endpoints that you want to protect. As I don’t yet have a real endpoint in my service, I created a test private-endpoint. This endpoint simply returns the user context data passed onto it from the Lambda Authorizer back to authenticated clients.

Here are the relevant parts of my serverless.yml file:

custom:
    vpcSettings:
      securityGroupIds:
        - !Ref MLabSecurityGroup
      subnetIds:
        - ${cf:vpc.SubnetAPrivate}
        - ${cf:vpc.SubnetBPrivate}
    lambda_authorizer:
        name: authorizer
        resultTtlInSeconds: 0
        identitySource: ''
        type: request

functions:
    # Lambda Authorizer function
    authorizer:
        handler: src/functions/authorizer.handler
        vpc: ${self:custom.vpcSettings}
        environment:
            SESSION_SECRET: ${ssm:/autochart/${self:provider.stage}/session-secret~true}
    private-endpoint:
        handler: src/functions/private-endpoint.handler
        vpc: ${self:custom.vpcSettings}
        events:
        - http:
            path: ${self:custom.apiRoot}/private
            method: get
            authorizer: ${self:custom.lambda_authorizer}

Firstly, you’ll notice that my functions need to be inside a VPC in order to access my MongoDB database. I also pass a SESSION_SECRET environment variable (fetched from SSM Parameter Store) to my authorizer function. This is the same session secret that the legacy API uses to sign session keys.
The http.authorizer attribute of the private-endpoint function is where the connection is made between the endpoint handler and the authorizer function.

The private-endpoint handler function can then access the custom user data via the event.requestContext.authorizer field:

// src/functions/private-endpoint.ts
import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';

export const handler = wrap(async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
    const response = {
        authContext: event.requestContext.authorizer,
    };
    return {
        statusCode: 200,
        body: JSON.stringify(response),
    };
});

To cache or not to cache

API Gateway allows you to cache the responses of Lambda Authorizers for a period of time. This can be useful as it avoids the extra latency incurred on each request by calling an extra function and the roundtrip to MongoDB to fetch the session data.
While this seems like it would be prudent, I decided against implementing this at this stage for a few reasons:

The existing legacy API currently has no auth caching, so the roundtrip to MongoDB will not add additional latency.
Caching could introduce strange behaviour and require complex invalidation logic across both new and legacy APIs (e.g. if user logs out).
I couldn’t work out if my use case of allowing the auth token to be in EITHER the cookie OR the authorization header is supported. API Gateway allows you to specify zero or more “Identity Sources” which stipulate the HTTP request parameters that are required in the auth logic. If this is specified, the parameter is used to form a cache key. However, from my testing it seemed that if you provide more than 1 source that API Gateway then ANDs each parameter, which has the effect of requiring that the client supply all the headers. This wouldn't work for my use case.

I will review this decision to skip auth caching after I observe the real-world latency of my migrated endpoints.

Next Steps

Now that I have my auth logic in place, I can begin migrating the "Event Metrics" service from the legacy API. I'll be covering this in my next post.

✉️ If you enjoyed this article and would like to get future updates from me on migrating to serverless, you can subscribe to my weekly newsletter on building serverless apps in AWS.

You also might enjoy:

Originally published at winterwindsoftware.com.

Building CICD pipelines for serverless microservices using the AWS CDK

Paul Swail — Tue, 09 Apr 2019 12:15:00 +0000

This is part 5 in the series Migrating a Monolithic SaaS App to Serverless — A Decision Journal.

In order to make sure that each new route I migrate over from the legacy Express.js API to Lambda is thoroughly tested before releasing to production, I have started to put a CICD process in place.

My main goals for the CICD process are:

Each Serverless service will have its own pipeline as each will be deployed independently.
Merges to master should trigger the main pipeline which will perform tests and deployments through dev to staging to production.
The CICD pipelines will be hosted in the DEV AWS account but must be able to deploy to other AWS accounts.
All CICD config will be done via infrastructure-as-code so I can easily setup pipelines for new services as I create them. Console access will be strictly read-only.

I have decided to use AWS CodePipeline and CodeBuild to create the pipelines as they’re serverless and have IAM support built-in that I can use to securely deploy to prod servers. CodePipeline acts as the orchestrator whereas CodeBuild acts as the task runner. Each stage in the CodePipeline pipeline consists of one or more actions which call out to a CodeBuild project that contains the actual commands to create a release package, run tests, do deployment, etc.

Pipeline overview

For the first version of my CICD process, I will be deploying a single service ac-rest-api (which currently only contains a single test Lambda function + API GW route). This pipeline will work as follows:

GitHub source action triggers whenever code is pushed to the master branch of the repo hosting my ac-rest-api service.
Run lint + unit/integration tests, and build deployment package (using sls package command)
Deploy package (using sls deploy) to the DEV stage and run acceptance tests against it.
Deploy package to the STAGING stage and run acceptance tests against it.
Deploy package to the PROD stage and run acceptance tests against it.

Here's what my completed pipeline looks like:

Managing pipelines as infrastructure-as-code

As with most AWS services, it’s easy to find tutorials on how to create a CodePipeline pipeline using the AWS Console, but difficult to find detailed infrastructure-as-code examples. CodeBuild fairs better in this regard as the docs generally seem to recommend using a buildspec.yml file in your repo to specify the build commands.

Since I will be creating several pipelines for each microservice I’ve identified as I go through the migration process, I want something I can easily reuse.

I considered these options for defining my pipelines:

Use raw CloudFormation YAML files.
Use CloudFormation inside the resources section of serverless.yml.
Use the new AWS Cloud Development Kit (CDK), which allows you to define high-level infrastructure constructs using popular languages (Javascript/Typescript, Java, C#) which generate CloudFormation stacks under-the-hood.

I’m relatively proficient with CloudFormation but I find its dev experience very frustrating and in the past have burned days getting complex CloudFormation stacks up and running. My main gripes with it are:

The slow feedback loop between authoring and seeing a deployment complete (fail or succeed)
The lack of templating/modules which makes it difficult to re-use without resorting to copying and pasting a load of boilerplate (I’m aware of tools like Troposphere that help in this regard but I’m a Javascript developer and don’t want to have to learn a new language (Python))
I always need to have the docs open in a browser tab to see the available properties on a resource

Given this, I decided to proceed with using the CDK which (in theory) should help mitigate all 3 of these complaints.

Creating my CDK app

The CDK provides a CLI that you use to deploy an “app”. An app in this context is effectively a resource tree that you compose using an object-oriented programming model, where each node in the tree is called a “construct”. The CDK provides ready-made constructs for all the main AWS resources and also allows (and encourages) you to write your own. A CDK app contains at least one CloudFormation stack construct, which it uses to instigate the deployment under the hood. So you still get the automatic rollback benefit that CloudFormation gives you.

For more detailed background on what the CDK is and how you can install it and bootstrap your own app, I'd recommended starting here.

I also decided to try out Typescript (instead of plain Javascript) as the CDK was written in Typescript and the strong typing seems to be a good fit for infrastructure resources with complex sets of attributes.

I started by creating a custom top-level construct called ServiceCicdPipelines which acts as a container for all the pipelines for each service I'll be creating and any supplementary resources. I've included all the source code in this gist, but the main elements of it are as follows:

cicd-pipelines-stack : single CloudFormation stack for deploying all the CICD related resources.
pipelines[]: List of ServicePipeline objects — a custom construct which encapsulates the logic for creating a single pipeline for a single Serverless service.
alertsTopic: SNS topic which will receive notifications about pipeline errors.

The custom ServicePipeline construct is where most of the logic lies. It takes a ServiceDefinition object as a parameter, which looks like this:

export interface ServiceDefinition {
    serviceName: string;
    githubRepo: string;
    githubOwner: string;
    githubTokenSsmPath: string;
    /** Permissions that CodeBuild role needs to assume to deploy serverless stack */
    deployPermissions: PolicyStatement[];
}

This object is where all the unique attributes of the service being deployed are defined. At the moment, I don’t have many options in here but I expect to add to this over time.

The ServicePipeline construct is also where each stage of the pipeline is defined:

export class ServicePipeline extends Construct {
    readonly pipeline: Pipeline;
    readonly alert: PipelineFailedAlert;

    constructor(scope: Construct, id: string, props: ServicePipelineProps) {
        super(scope, id);
        const pipelineName = `${props.service.serviceName}_${props.sourceTrigger}`;
        this.pipeline = new Pipeline(scope, pipelineName, {
            pipelineName,
        });

        // Read Github Oauth token from SSM Parameter Store
        // https://docs.aws.amazon.com/codepipeline/latest/userguide/GitHub-rotate-personal-token-CLI.html
        const oauth = new SecretParameter(scope, 'GithubPersonalAccessToken', {
            ssmParameter: props.service.githubTokenSsmPath,
        });

        const sourceAction = new GitHubSourceAction({
            actionName: props.sourceTrigger === SourceTrigger.PullRequest ? 'GitHub_SubmitPR' : 'GitHub_PushToMaster',
            owner: props.service.githubOwner,
            repo: props.service.githubRepo,
            branch: 'master',
            oauthToken: oauth.value,
            outputArtifactName: 'SourceOutput',
        });

        this.pipeline.addStage({
            name: 'Source',
            actions: [sourceAction],
        });

        // Create stages for DEV => STAGING => PROD.
        // Each stage defines its own steps in its own build file
        const buildProject = new ServiceCodebuildProject(this.pipeline, 'buildProject', {
            projectName: `${pipelineName}_dev`,
            buildSpec: 'buildspec.dev.yml',
            deployerRoleArn: CrossAccountDeploymentRole.getRoleArnForService(
                props.service.serviceName, 'dev', deploymentTargetAccounts.dev.accountId,
            ),
        });
        const buildAction = buildProject.project.toCodePipelineBuildAction({
            actionName: 'Build_Deploy_DEV',
            inputArtifact: sourceAction.outputArtifact,
            outputArtifactName: 'sourceOutput',
            additionalOutputArtifactNames: [
                'devPackage',
                'stagingPackage',
                'prodPackage',
            ],
        });
        this.pipeline.addStage({
            name: 'Build_Deploy_DEV',
            actions: [buildAction],
        });
        const stagingProject = new ServiceCodebuildProject(this.pipeline, 'deploy-staging', {
            projectName: `${pipelineName}_staging`,
            buildSpec: 'buildspec.staging.yml',
            deployerRoleArn: CrossAccountDeploymentRole.getRoleArnForService(
                props.service.serviceName, 'staging', deploymentTargetAccounts.staging.accountId,
            ),
        });
        const stagingAction = stagingProject.project.toCodePipelineBuildAction({
            actionName: 'Deploy_STAGING',
            inputArtifact: sourceAction.outputArtifact,
            additionalInputArtifacts: [
                buildAction.additionalOutputArtifact('stagingPackage'),
            ],
        });
        this.pipeline.addStage({
            name: 'Deploy_STAGING',
            actions: [stagingAction],
        });

        // Prod stage requires cross-account access as codebuild isn't running in same account
        const prodProject = new ServiceCodebuildProject(this.pipeline, 'deploy-prod', {
            projectName: `${pipelineName}_prod`,
            buildSpec: 'buildspec.prod.yml',
            deployerRoleArn: CrossAccountDeploymentRole.getRoleArnForService(
                props.service.serviceName, 'prod', deploymentTargetAccounts.prod.accountId,
            ),
        });
        const prodAction = prodProject.project.toCodePipelineBuildAction({
            actionName: 'Deploy_PROD',
            inputArtifact: sourceAction.outputArtifact,
            additionalInputArtifacts: [
                buildAction.additionalOutputArtifact('prodPackage'),
            ],
        });
        this.pipeline.addStage({
            name: 'Deploy_PROD',
            actions: [prodAction],
        });

        // Wire up pipeline error notifications
        if (props.alertsTopic) {
            this.alert = new PipelineFailedAlert(this, 'pipeline-failed-alert', {
                pipeline: this.pipeline,
                alertsTopic: props.alertsTopic,
            });
        }
    }
}

export interface ServicePipelineProps {
    /** Information about service to be built & deployed (source repo, etc) */
    service: ServiceDefinition;
    /** Trigger on PR or Master merge?  */
    sourceTrigger: SourceTrigger;
    /** Account details for where this service will be deployed to */
    deploymentTargetAccounts: DeploymentTargetAccounts;
    /** Optional SNS topic to send pipeline failure notifications to */
    alertsTopic?: Topic;
}

/** Wrapper around the CodeBuild Project to set standard props and create IAM role */
export class ServiceCodebuildProject extends Construct {
    readonly buildRole: Role;
    readonly project: Project;

    constructor(scope: Construct, id: string, props: ServiceCodebuildActionProps) {
        super(scope, id);

        this.buildRole = new ServiceDeployerRole(this, 'project-role', {
            deployerRoleArn: props.deployerRoleArn,
        }).buildRole;

        this.project = new Project(this, 'build-project', {
            projectName: props.projectName,
            timeout: 10, // minutes
            environment: {
                buildImage: LinuxBuildImage.UBUNTU_14_04_NODEJS_8_11_0,
            },
            source: new CodePipelineSource(),
            buildSpec: props.buildSpec || 'buildspec.yml',
            role: this.buildRole,
        });
    }
}

Quick note on pipeline code organisation

I created all these CDK custom constructs within a shared autochart-infrastructure repo which I use to define shared, cross-cutting infrastructure resources that are used by multiple services.
The buildspec files that I cover below live in the same repository as their service (which is how it should be). I’m not totally happy with having the pipeline definition and the build scripts in separate repos and I will probably look at a way of moving my reusable CDK constructs into a shared library and have the source for the CDK apps defined in the service-specific repos while referencing this library.

Writing the CodeBuild scripts

In the above pipeline definition code, you may have noticed that each deployment stage has its own buildspec file (using the naming convention buildspec.<stage>.yml). I want to minimise the amount of logic within the pipeline itself and keep it solely responsible for orchestration. All the logic will live in the build scripts.

Building and deploying to DEV stage

As soon as a push occurs to the master branch in Github, the next pipeline step is to run tests, package and deploy to the DEV stage. This is shown below:

# buildspec.dev.yml
version: 0.2

env:
  variables:
    TARGET_REGION: us-east-1
    SLS_DEBUG: ''
    DEPLOYER_ROLE_ARN: 'arn:aws:iam::<dev_account_id>:role/ac-rest-api-dev-deployer-role'

phases:
  pre_build:
    commands:
      - chmod +x build.sh
      - ./build.sh install
  build:
    commands:
      # Do some local testing
      - ./build.sh test-unit
      - ./build.sh test-integration
      # Create separate packages for each target environment
      - ./build.sh clean
      - ./build.sh package dev $TARGET_REGION
      - ./build.sh package staging $TARGET_REGION
      - ./build.sh package prod $TARGET_REGION
      # Deploy to DEV and run acceptance tests there
      - ./build.sh deploy dev $TARGET_REGION dist/dev
      - ./build.sh test-acceptance dev

artifacts:
  files:
    - '**/*'
  secondary-artifacts:
    devPackage:
      base-directory: ./dist/dev
      files:
        - '**/*'
    stagingPackage:
      base-directory: ./dist/staging
      files:
        - '**/*'
    prodPackage:
      base-directory: ./dist/prod
      files:
        - '**/*'

There are a few things to note here:

A DEPLOYER_ROLE_ARN environment variable is required which defines a pre-existing IAM role which has the permissions to perform the deployment steps in the target account. We will cover how to set this up later.
Each build command references a build.sh file. The reason for this indirection is to make it easier to test the build scripts during development time, as I’ve no way of invoking a buildspec file locally (this is based on the approach Yan Cui recommends in the CI/CD module of his Production-Ready Serverless course)
I run unit and integration tests before doing the packaging. This runs the Lambda function code locally (in the CodeBuild container) and not via AWS Lambda, though the integration tests may hit existing downstream AWS resources which functions call on to.
I am creating 3 deployment artifacts here, one for each stage, and including them as output artifacts which can be used by the future stages . I wasn't totally happy with this as it violates best Devops practice of having a single immutable artifact flow through each stage. My reason for doing this was that the sls package command that the build.sh file uses requires a specific stage to be provided to it. However, I’ve since learned that there is a workaround for this, so I will probably change this soon.
I run acceptance tests (which hit the newly deployed API Gateway endpoints) as a final step.

The build.shfile is below:

#!/bin/bash
set -e
set -o pipefail

instruction()
{
  echo "usage: ./build.sh package <stage> <region>"
  echo ""
  echo "/build.sh deploy <stage> <region> <pkg_dir>"
  echo ""
  echo "/build.sh test-<test_type> <stage>"
}

assume_role() {
  if [ -n "$DEPLOYER_ROLE_ARN" ]; then
    echo "Assuming role $DEPLOYER_ROLE_ARN ..."
    CREDS=$(aws sts assume-role --role-arn $DEPLOYER_ROLE_ARN \
        --role-session-name my-sls-session --out json)
    echo $CREDS > temp_creds.json
    export AWS_ACCESS_KEY_ID=$(node -p "require('./temp_creds.json').Credentials.AccessKeyId")
    export AWS_SECRET_ACCESS_KEY=$(node -p "require('./temp_creds.json').Credentials.SecretAccessKey")
    export AWS_SESSION_TOKEN=$(node -p "require('./temp_creds.json').Credentials.SessionToken")
    aws sts get-caller-identity
  fi
}

unassume_role() {
  unset AWS_ACCESS_KEY_ID
  unset AWS_SECRET_ACCESS_KEY
  unset AWS_SESSION_TOKEN
}

if [ $# -eq 0 ]; then
  instruction
  exit 1
elif [ "$1" = "install" ] && [ $# -eq 1 ]; then
  npm install
elif [ "$1" = "test-unit" ] && [ $# -eq 1 ]; then
  npm run lint
  npm run test
elif [ "$1" = "test-integration" ] && [ $# -eq 1 ]; then
  echo "Running INTEGRATION tests..."
  npm run test-integration
  echo "INTEGRATION tests complete."
elif [ "$1" = "clean" ] && [ $# -eq 1 ]; then
  rm -rf ./dist
elif [ "$1" = "package" ] && [ $# -eq 3 ]; then
  STAGE=$2
  REGION=$3
  'node_modules/.bin/sls' package -s $STAGE -p "./dist/${STAGE}" -r $REGION
elif [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
  STAGE=$2
  REGION=$3
  ARTIFACT_FOLDER=$4
  echo "Deploying from ARTIFACT_FOLDER=${ARTIFACT_FOLDER}"
  assume_role
  # 'node_modules/.bin/sls' create_domain -s $STAGE -r $REGION
  echo "Deploying service to stage $STAGE..."
  'node_modules/.bin/sls' deploy --force -s $STAGE -r $REGION -p $ARTIFACT_FOLDER
  unassume_role
elif [ "$1" = "test-acceptance" ] && [ $# -eq 2 ]; then
  STAGE=$2
  echo "Running ACCEPTANCE tests for stage $STAGE..."
  STAGE=$STAGE npm run test-acceptance
  echo "ACCEPTANCE tests complete."
elif [ "$1" = "clearcreds" ] && [ $# -eq 1 ]; then
  unassume_role
  rm -f ./temp_creds.json
else
  instruction
  exit 1
fi

The main thing to note in the above script is the assume_role function which gets called before the deploy command. In order for CodeBuild to deploy to a different AWS account, the sls deploy command of the serverless framework needs to be running as a role defined in the target account. To do this, the Codebuild IAM role (which is running in the DEV account) needs to assume this role. I’m currently doing this by invoking the aws sts assume-role command of the AWS CLI to get temporary credentials for this deployment role. However, this seems messy, so if anyone knows a cleaner way of doing this, I’d love to hear it.

Deploying to Staging and Production

The script for deploying and running tests against staging is similar to the dev script, but simpler:

# buildspec.staging.yml
version: 0.2

env:
  variables:
    TARGET_REGION: us-east-1
    SLS_DEBUG: '*'
    DEPLOYER_ROLE_ARN: 'arn:aws:iam::<staging_account_id>:role/ac-rest-api-staging-deployer-role'

# Deploy to STAGING stage and run acceptance tests.
phases:
  pre_build:
    commands:
      - chmod +x build.sh
      - ./build.sh install
  build:
    commands:
      - ./build.sh deploy staging $TARGET_REGION $CODEBUILD_SRC_DIR_stagingPackage
      - ./build.sh test-acceptance staging

The key things to note here are:

I use the $CODEBUILD_SRC_DIR_stagingPackage environment variable to access the directory where the output artifact named stagingPackage from the last pipeline step is located.

The buildspec.prod.yml file is pretty much the same as the staging one, except it references the production IAM role and artefact source directory.

Deployment Permissions and Cross-account access

This was probably the hardest part of the whole process. As I mentioned above, the deployment script assumes a pre-existing deployer-role IAM role that exists in the AWS account of the stage being deployed to.
To set up these roles, I again used the CDK to define a custom construct CrossAccountDeploymentRole as follows:

export interface CrossAccountDeploymentRoleProps {
    serviceName: string;
    /** account ID where CodePipeline/CodeBuild is hosted */
    deployingAccountId: string;
    /** stage for which this role is being created */
    targetStageName: string;
    /** Permissions that deployer needs to assume to deploy stack */
    deployPermissions: PolicyStatement[];
}

/**
 * Creates an IAM role to allow for cross-account deployment of a service's resources.
 */
export class CrossAccountDeploymentRole extends Construct {
    public static getRoleNameForService(serviceName: string, stage: string): string {
        return `${serviceName}-${stage}-deployer-role`;
    }

    public static getRoleArnForService(serviceName: string, stage: string, accountId: string): string {
        return `arn:aws:iam::${accountId}:role/${CrossAccountDeploymentRole.getRoleNameForService(serviceName, stage)}`;
    }

    readonly deployerRole: Role;
    readonly deployerPolicy: Policy;
    readonly roleName: string;

    public constructor(parent: Construct, id: string, props: CrossAccountDeploymentRoleProps) {
        super(parent, id);
        this.roleName = CrossAccountDeploymentRole.getRoleNameForService(props.serviceName, props.targetStageName);
        // Cross-account assume role
        // https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-iam.html#configuring-an-externalid
        this.deployerRole = new Role(this, 'deployerRole', {
            roleName: this.roleName,
            assumedBy: new AccountPrincipal(props.deployingAccountId),
        });
        const passrole = new PolicyStatement(PolicyStatementEffect.Allow)
            .addActions(
                'iam:PassRole',
            ).addAllResources();
        this.deployerPolicy = new Policy(this, 'deployerPolicy', {
            policyName: `${this.roleName}-policy`,
            statements: [passrole, ...props.deployPermissions],
        });
        this.deployerPolicy.attachToRole(this.deployerRole);
    }
}

The props object that is passed to the constructor contains the specific requirements of the service. The main thing here is the deployPermissions which is a set of IAM policy statements to enable the user running the Serverless framework’s sis deploy command to deploy all the necessary resources (Cloudformation stacks, Lambda functions, etc).

An important thing to note is that this set of permissions is different from the runtime permissions your serverless service needs to execute. Working out what deploy-time IAM permissions a service needs is a well-known problem in the serverless space and one that I haven’t yet found a nice solution for. I initially started by giving my role admin access until I got the end-to-end pipeline working, and then worked backwards to add more granular permissions, but this involved a lot of trial and error.

For more information on how IAMs work with the Serverless framework, I’d recommend reading The ABCs of IAM: Managing permissions with Serverless.

A few tips for building and testing your own pipelines

Always run the build.sh file locally first to make sure it works on your machine
Use the "Release Change" button in the CodePipeline console in order to start a new pipeline execution without resorting to pushing dummy commits to Github.

Future Enhancements

Going forward, here are a few additions I’d like to make to my CICD process:

Add a manual approval step before deploying to PROD.
Add an automatic rollback action if acceptance tests against the PROD deployment fail.
Creations/updates of Pull Requests should trigger a shorter test-only pipeline. This will help to identify integration issues earlier.
Slack integration for build notifications.
Update my package step so that it only builds a single immutable package that will be deployed to all environments.
Get extra meta, and have a pipeline for the CICD code 🤯. Changes to my pipeline will have tests run against them before deploying a new pipeline.

Next Steps

Next up in my migration plan is getting an auth mechanism built into API Gateway that new routes can use. That will be the first real production code being tested by my new CICD pipeline.

✉️ If you enjoyed this article and would like to learn more about serverless, you might enjoy my weekly newsletter on building serverless apps in AWS.

You also might enjoy:

Originally published at winterwindsoftware.com.

DEV Community: Paul Swail

The Simple Guide to Testing within your Serverless CI/CD Pipelines

Workflows overview

Pull Request Workflow (CI only)

Mainline Workflow (CI + CD)

Build stage

Test stage

Staging stage

Manual approval

Production stage

Summary of test types

Building out your own pipeline

Integration and E2E tests are the primary confidence drivers for serverless apps

What can go wrong?

Covering these failure modes with tests

Conclusion

The testing trade-off triangle

Get back to the basics of testing

The tension between each objective

Example 1: Optimise for maximum confidence

Example 2: Optimise solely for fast feedback

Work out how to define "just enough" for your context

Further reading

Add type definitions to your Lambda functions

What problem does this solve?

Some examples

Optimise your Lambda functions using Webpack

What problem does this solve?

How does it work?

Setting it up

Handling edge cases

Async Initialisation of a Lambda Handler

Give developers their own AWS account

Fully local development workflows are suboptimal or even impossible in serverless stacks

You give developers more exposure to infrastructure management

You give less experienced developers more confidence to experiment

Common objections

Isn’t this a big admin overhead to have to set this up?

How will we control costs?

What about security?

Our IT department simply won’t allow this

How to access VPC and internet resources from Lambda without paying for a NAT Gateway

Using a VPC proxy Lambda function

Limitations

Comparing multi and single table approaches to designing a DynamoDB data model

What are the key differences between each approach?

My experience with both approaches

Big design up front

The design process

Provisioning and configuration management

Implementing data access in the codebase

Writing objects to the database

Reading objects from the database

Strategies for controlling code complexity in a single-table design

Schema migrations

Integrating with other data stores

When should you use either approach?

Learn more

How to build a serverless photo upload service with API Gateway

Considering implementation options

Using S3 presigned URLs for upload

Step 1: Create the S3 bucket

Step 2: Create "Initiate Upload" API Gateway endpoint

Step 3: Uploading file from the web app

Step 4: Pushing photo data into database

Future enhancements

Migrating authentication from Express.js to API Gateway using a Lambda Authorizer

Using a Lambda Authorizer to authenticate API requests

Authentication vs Authorization

Reverse engineering the Express authentication logic

Allowing and denying requests

Wiring up auth logic to a private endpoint

To cache or not to cache

Next Steps

Building CICD pipelines for serverless microservices using the AWS CDK

Pipeline overview

Managing pipelines as infrastructure-as-code

Creating my CDK app

Quick note on pipeline code organisation

Writing the CodeBuild scripts