Speed up your AWS SAM development by deploying individual functions

Cover image from https://aws.amazon.com/lambda/getting-started/

The Amazon Web Services Serverless Application Model (or AWS SAM, for short) is a framework for simplified development of serverless applications on top of AWS. SAM ships with a CLI that provides common operations, such as packaging and deploying your application, or spinning up a dev server for local testing.

Behind the scenes, SAM relies on CloudFormation, a mature template-based language for describing and deploying entire AWS stacks. Whenever you deploy your application, SAM will compile your code, upload it to S3, and use CloudFormation to update your stack.

The problem

While this process is (usually) reliable and leads to reproducible deployments, it can also become painfully slow as your stack grows. Given a stack with about 200 resources, a deployment where nothing has changed takes about 10 minutes from start to end. Compiling the entire codebase takes time, as well as uploading it and letting CloudFormation run its course.

Though 10 minutes is acceptable for a full deployment, it is not for trying out your changes while developing. You need a much faster feedback loop if you want to stay in flow. To address this, other frameworks such as serverless.com provide a command deploy function that bypasses CloudFormation entirely, and simply uses the AWS Lambda API to update the function code directly. Unfortunately, this is not available as part as the AWS SAM CLI. While researching this, I found a CLI called sampic built by a community member on top of SAM, but it seemed too heavyweight for what I was looking for.

Hacking a solution

Given that switching from SAM to serverless.com can be too disruptive in an already established project, I set out to reproduce the deploy function behavior manually for our webpack-powered typescript AWS SAM project. The end result is a 120-lines script, most of which is just logging, that picks a subset of functions from a SAM template, compiles them, and uploads them via the AWS Lambda API.

Note that this solution only works if the function has already been created by sam deploy. The script will not create the function if it doesn't exist, it only knows how to update its code.

In this post, I'll walk through the necessary steps to build this solution - but you can also just jump directly to the entire code snippet at the end of the post!

Getting started

The first step is to choose which functions to deploy. To do this, we'll compare all function names in the template to an fname substring provided by the user, and filter only those that match.

We'll use the yaml-cfn package for parsing the template, since CloudFormation accepts a set of non-standard YAML tags for intrinsic functions which can break a vanilla yaml parser. And we'll use lodash, because I can't code in js without it.

const { readFileSync } = require('fs');
const { yamlParse } = require('yaml-cfn');
const { map, pickBy } = require('lodash');
const TEMPLATE_FILENAME = 'template.yaml';

const template = yamlParse(readFileSync(TEMPLATE_FILENAME, 'utf8'));
const filtered = pickBy(template.Resources, (resource, name) => (
  name.toLowerCase().includes(fname.toLowerCase()) 
  && resource.Type === 'AWS::Serverless::Function'
));
const functions = map(filtered, (fn, name) => (
  { Name: name, ...fn.Properties }
));

This will return an array of objects like the one below, with the metadata of the functions to be deployed.

Name: 'UserInviteFunction',
FunctionName: { 'Fn::Sub': '${AWS::StackName}-${Stage}-user-invite-fn' },
CodeUri: './build/users',
Handler: 'index.inviteUser',
Events: { InviteUser: [Object] }

Compiling only what's needed

Now that we know which functions we want to deploy, it's time to compile them. I'm currently using webpack, with one entry point for each set of related endpoints; but if you're using a different bundler, you may need to adapt the following steps to it.

// webpack.config.js
module.exports = {
  entry: {
    authorizer: './src/authorizer/index.ts',
    projects: './src/endpoints/projects.ts',
    organizations: './src/endpoints/organizations.ts',
    actions: './src/endpoints/actions.ts',
    users: './src/endpoints/users.ts',
    ...
  },
  output: {
    filename: '[name]/index.js',
    libraryTarget: 'commonjs2',
    path: resolve(__dirname, 'build'),
    sourceMapFilename: '[name]/index.js.map',
  },
  ...
}

To speed up compilation, we will filter the set of entry points defined in our webpack config so we only compile those necessary for the functions to be deployed. If you have over 25 different entrypoints like I do, this can be a major time saver!

const { basename, resolve } = require('path');
const { uniq, pick } = require('lodash');

const WEBPACK_CONFIG = './webpack.config.js';
const config = require(resolve(WEBPACK_CONFIG));

// We map from CodeUris like './build/users' to entries like 'users'
const entryPoints = uniq(functions.map(f => basename(f.CodeUri)));
config.entry = pick(config.entry, entryPoints);

Before we trigger the compilation, we'll add an extra step. When using sam deploy, the SAM CLI takes care of zipping and uploading the code for us, but since we are uploading the code manually, we'll need to zip it ourselves.

Unfortunately, webpack's compression-webpack-plugin does not support zip files, and the zip-webpack-plugin does not support multiple entry points. So, we will just tap into the assetEmitted webpack hook and zip from there. We'll just use the zip OS command to do this, but we could also use a package like yazl or jszip.

const { dirname, extname } = require('path');
const { promisify } = require('util');
const exec = promisify(require('child_process').exec);
const webpack = require('webpack');

const compiler = webpack(config);
compiler.hooks.assetEmitted.tapPromise('zip', async (filename) => {
  if (extname(filename) === '.map') return;
  const cwd = resolve(config.output.path, dirname(filename));
  await exec(`zip index.zip index.js`, { cwd });
});

With all of this set up, we can finally set out to compile our functions.

const stats = await promisify(compiler.run.bind(compiler))();
if (stats.hasErrors()) throw new Error(stats.toJson().errors);

Resolving function names

We now know what functions to deploy and have compiled them. But before uploading the code, we need to know the physical resource names of the lambdas we are targeting.

In a CloudFormation stack, the name you assign to each resource is called the logical name. However, this is not the name used when creating the resources. Given that your resources must have unique names within an account and region, CloudFormation will use a combination of your stack name, the logical name, and a unique identifier for the actual physical name. You can also control the name for each resource by setting a Name property in your template.

Fortunately, the CloudFormation API allows us to retrieve the physical name for a resource given its logical name and stack via a DescribeStackResource method.

const { CloudFormation } = require('aws-sdk');
const STACKNAME = 'MY-STACK';
const cfn = new CloudFormation();

const resolvedFunctions = await Promise.all(functions.map(async (fn) => {
  const response = await cfn.describeStackResource({
    LogicalResourceId: fn.Name, StackName: STACKNAME 
  }).promise();
  const resourceId = response.StackResourceDetail.PhysicalResourceId;
  return { ...fn, ResourceId: resourceId };
}));

Armed with the actual lambda function names, we can now finally use the UpdateFunctionCode API method to upload our code directly to our Lambda functions.

const { Lambda } = require('aws-sdk');
const lambda = new Lambda();

await Promise.all(resolvedFunctions.map(async (fn) => {
  const zipFilePath = resolve(fn.CodeUri, 'index.zip');
  await lambda.updateFunctionCode({
    FunctionName: fn.ResourceId, ZipFile: readFileSync(zipFilePath) 
  }).promise();
}));

Measuring it

When compiling and deploying a single function, the script above takes roughly 20-30 seconds, most of which is spent in compilation. To compare, deploying the stack via sam deploy takes about 10 minutes, which means we're in for a 20x speed increase during development. A faster feedback loop means more development productivity, so achieving this kind of gains can be a major improvement to your project.

Putting it all together

Here is the entire script we are using today for development, which puts together all the steps described above, along with some logging and validations. Keep in mind that some bits may be different on your own setup, such as the paths where the artifacts are generated, or how your template is organized. Nevertheless, I expect it to work in most setups with just minor tweaks.

Happy coding!