Managing Multiple Functions with AWS SAM and Webpack

I've been spending some quality time with the Serverless Application Model from AWS. I'm really enjoying this tool and I see it as a fantastic way for developers to learn more about the AWS ecosystem while being productive. However, it doesn't natively support TypeScript, my language of choice, and the build system it ships with leaves something to be desired. I also wanted to come up with a way to share some code between functions without the complexity of publishing and installing a private npm package or using lambda layers. The built-in tree-shaking and minification that webpack provides are also very attractive.

tl;dr

Feel free to skip ahead to the repo.

Table of Contents

• AWS SAM
• Webpack
• Plugin
• Toolchain
• Project Structure
• Install Packages
• Hello Lambda
• Goodbye Lambda
• Unit Tests
• Webpack Config
• Another Dependency
• SAM Template
• Local Test
• Deploy
• Next Steps
• Update

AWS SAM

SAM is a cli tool that enables local development and also provides access to an extended version of CloudFormation. I have previously written about combining SAM with AWS Cloud Developer Kit. In this case I'm just going to be using SAM.

Webpack

Many of us have struggled with messy webpack configuration. Copy-pasted webpack files can be tough to handle, with weird plugins and orphaned config options. Believe me when I say it doesn't have to be that way. The best way to deal with webpack is to start with a simple configuration and only add the options and plugins that you need and be sure to comment anything that's unclear. This is a bit like all coding if you think about it, but for some reason we don't always apply the same standards to our config files that we do to our source code.

So if you come to this article hating webpack, just bear with me and maybe you'll learn to do as I do and only allow the configuration options that you like and understand!

Plugin

I should mention there already exists a webpack plugin that seeks to solve the same problem, just in a different way. I haven't used it myself, but I read through the code and seems like it parses your template.yaml file to figure out what the entry points should be. I was thinking about contrasting the plugin with my approach, but there's already a great post on the subject. Check out aws-sam-webpack-plugin. Both approaches look quite viable to me. Mine has fewer moving parts, but maybe just a touch more work involved - adding a new function means touching both your template.yaml and your webpack.config.ts files.

Toolchain

I'm running Node 12 and the latest (as of this writing) version of SAM. sam local commands depend on having Docker installed as your function will be run in a containerized environment (and yeah that's ironic, but works very well).

% node --version
v12.18.2
% sam --version
SAM CLI, version 1.0.0
% docker --version
Docker version 19.03.8, build afacb8b

If you don't have these tools installed, see the guides for nodejs, SAM, and Docker.

Project Structure

The default project structure you get with sam init will put a template.yaml at the project root, then create a subdirectory complete with a package.json for each function. If you sam build, SAM will search your entire project structure for package.json files, copy everything into .aws-sam/build and npm install for each function you've staked out in your project. There isn't any way to add bundling or transpilation to the build process.

If I want to run my tests, I need to cd into the function directory, install my test runner (jest, mocha, whatever) there and run the test from that directory. Additionally the sam build step will be a bit slower as it it will run one npm install operation for each of my functions. Since each of these functions will have its own package.json file, that means each of them manages its dependencies independently and creates more work for me to keep my dependencies up-to-date.

My project will maintain a single package.json file at the root. Whatever dependencies my functions need will be shared. Because webpack does automatic tree-shaking, I can have confidence that unneeded dependencies won't be included in my bundles.

I'm going to put my functions in ./src/handlers and allow for the possibility of some shared modules under src.

Install Packages

I'm going to install most of my packages as devDependencies. In a traditional nodejs app, this would let me have a build process that eventually runs npm prune --production to remove all the devDependencies to make my node_modules smaller. But because I'm using webpack and the twice-aforementioned tree-shaking, this is really just a convention and not meaningful to my build process. This convention will be useful for other developers to understand my code. Anything installed under dependencies should be considered as a dependency of live code while devDependencies should be seen as libraries that help me build, test and lint my application.

npm i -D @types/webpack ts-loader ts-node typescript webpack webpack-cli

I'm installing the typings for webpack because I will write my config in TypeScript and include the typing for Configuration which will help me avoid configuration typos. I'm going to use the webpack cli (as opposed to programmatic) so I need to install both of those packages. I'll be coding in TypeScript, so I need that. ts-loader is a webpack plugin that enables the transpilation of .ts files into webpack bundles and ts-node is needed to write my configuration file in TypeScript. None of these dependencies will wind up in my Lambda functions unless I explicitly import them!

Since I'm using TypeScript, I need a tsconfig.json file.

{
  "compilerOptions": {
    "alwaysStrict": true,
    "module": "commonjs",
    "noImplicitAny": true,
    "target": "es2019"
  }
}

I could add a few more compiler options, but let's keep it simple for now. It's important that I've set module to commonjs in order for webpack to understand module loading. More on that here. I've set the target to es2019 as that version of ECMAScript is fully implemented in NodeJS 12. If I choose an earlier compilation target, TypeScript will shim functionality that wasn't available in that version of ECMAScript and my bundle will be larger (for no good reason).

I'm also going to install eslint to further help me avoid errors and keep my code as clean as possible. I wrote about the use of eslint in another article and I'm doing exactly the same thing here. I'll round out the devDependencies with rimraf because it's good to clean before you build.

Hello Lambda

I'm not improving on the sample project much here. My function is just going to say "Hello." I do want to install @types/aws-lambda so I can import the expected return type and make sure I'm doing something that Lambda expects.

import { APIGatewayProxyResult } from 'aws-lambda';

export const handler = async (): Promise<APIGatewayProxyResult> => {
  return { body: JSON.stringify({ message: 'hello' }), statusCode: 200 };
};

Even though my function doesn't actually do anything asynchronous, I need to make it an async function. Otherwise Lambda will expect a callback.

Goodbye Lambda

The same thing here. I'm putting these functions in ./src/handlers/hello.ts and ./src/handlers/goodbye.ts.

import { APIGatewayProxyResult } from 'aws-lambda';

export const handler = async (): Promise<APIGatewayProxyResult> => {
  return { body: JSON.stringify({ message: 'goodbye' }), statusCode: 200 };
};

Unit Tests

A great thing about writing Lambda Functions is they are very easy to test. I'm going to use jest and ts-jest. I simply call the function and test the output. Here I'm using jest snapshots because they make testing very easy.

import { handler } from './hello';

describe('hello handler', () => {
  it('should say hello', async () => {
    const result = await handler();
    expect(result).toMatchSnapshot();
  });
});

Webpack Config

I will create a webpack.config.ts file in the root of my project. Webpack is smart enough to realize that it should use that file by default and will employ ts-node to parse it, if installed (otherwise it's unlikely to work). This file just needs to export a configuration object.

import { resolve } from 'path';
import { Configuration } from 'webpack';

const config: Configuration = {

  // some stuff in here

};

export default config;

Okay, now to just get the important config bits. First of all, I'm going to be exporting two Lambda functions so I need two separate bundles. The way webpack builds multiple bundles is by specifying multiple entry points.

const config: Configuration = {
  entry: { hello: './src/handlers/hello.ts', goodbye: './src/handlers/goodbye.ts' },
  output: {
    filename: '[name].js',
    libraryTarget: 'commonjs2',
    path: resolve(__dirname, 'build'),
  },
  // more
};

This identifies my entry points as hello and goodbye and specifies the typescript files that will provide the source for those bundles. My output will sub in the name of the entry using commonjs2 and put everything in the build directory leaving me with ./build/hello.js and ./build/goodbye.js.

But this won't work yet. I need a few more config elements to bring it all together.

const config: Configuration = {
  // more
  module: {
    rules: [{ test: /\.ts$/, loader: 'ts-loader' }],
  },
  resolve: {
    extensions: ['.js', '.ts'],
  },
  target: 'node',
  // more
};

ts-loader is need here so that my TypeScript files get transpiled with tsc. I need to specify extensions for module resolution so that webpack knows import foo from './foo'; might get me foo.js or foo.ts. Even though I'm going to write all of my code in TypeScript, it's likely I'll want to import a dependency that's either written in or transpiled to JavaScript, so I need the .js extension as well. Targeting node lets webpack know that nodejs apis such as fs are available.

Lastly I'm going to give myself a way to run webpack in non-prod mode if I feel like it.

const config: Configuration = {
  // more
  mode: process.env.NODE_ENV === 'dev' ? 'development' : 'production',
  // more
};

If I set NODE_ENV to dev (for example NODE_ENV=dev npm run build) then my code won't be minified - something that might help with debugging. Ultimately I might like to set up a workflow where I always run my functions in dev mode while developing and then they are tested in production mode before being deployed. I also might want to look at adding webpack devtools as my project grows in complexity.

My build system is complete! Now I can run npx webpack and find goodbye.js and hello.js in my build directory. I'll add scripts to my package.json for ease.

% npm run build

> sam-typescript-webpack@0.0.1 build /Users/mattmorgan/mine/sam-typescript-webpack
> npm run clean && webpack


> sam-typescript-webpack@0.0.1 clean /Users/mattmorgan/mine/sam-typescript-webpack
> rimraf build

Hash: 5189a33d1ad6129ac421
Version: webpack 4.41.5
Time: 932ms
Built at: 02/10/2020 7:04:58 AM
     Asset      Size  Chunks             Chunk Names
goodbye.js  1.07 KiB       0  [emitted]  goodbye
  hello.js  1.07 KiB       1  [emitted]  hello
Entrypoint hello = hello.js
Entrypoint goodbye = goodbye.js
[0] ./src/handlers/hello.ts 188 bytes {1} [built]
[1] ./src/handlers/goodbye.ts 190 bytes {0} [built]

Another Dependency

Now I'll show what happens to my bundles when I introduce another dependency. Imagine I wanted to use winston for logging (actually console works pretty well in Lamba but perhaps I have a good reason). I'll npm i winston (my first real dependency), dash off a quick logging util and then add it to one of my handlers.

import { createLogger, format, transports } from 'winston';

const logger = createLogger({
  level: 'info',
  format: format.json(),
  transports: [new transports.Console()],
});

export default logger;

import { APIGatewayProxyResult } from 'aws-lambda';

import logger from '../util/logger';

export const handler = async (): Promise<APIGatewayProxyResult> => {
  logger.info(`time to say 'hello'`);
  return { body: JSON.stringify({ message: 'hello' }), statusCode: 200 };
};

I've added this to hello.ts but not goodbye.ts. When I build again, only the hello bundle has increased in size.

% npm run build

> sam-typescript-webpack@0.0.1 build /Users/mattmorgan/mine/sam-typescript-webpack
> npm run clean && webpack


> sam-typescript-webpack@0.0.1 clean /Users/mattmorgan/mine/sam-typescript-webpack
> rimraf build

Hash: cd9e2e299faff34c0ed6
Version: webpack 4.41.5
Time: 1224ms
Built at: 02/10/2020 7:09:16 AM
     Asset      Size  Chunks             Chunk Names
goodbye.js  1.07 KiB       0  [emitted]  goodbye
  hello.js   195 KiB       1  [emitted]  hello
Entrypoint hello = hello.js
Entrypoint goodbye = goodbye.js
  [3] external "util" 42 bytes {1} [built]
  [7] external "os" 42 bytes {1} [built]
 [17] external "stream" 42 bytes {1} [built]
 [74] external "fs" 42 bytes {1} [built]
[101] ./src/handlers/hello.ts 282 bytes {1} [built]
[102] ./src/util/logger.ts 295 bytes {1} [built]
[118] ./node_modules/logform sync ^\.\/.*\.js$ 1.09 KiB {1} [built]
[120] ./node_modules/logform/dist sync ^\.\/.*\.js$ 508 bytes {1} [built]
[138] external "path" 42 bytes {1} [built]
[168] external "zlib" 42 bytes {1} [built]
[187] external "tty" 42 bytes {1} [built]
[189] external "string_decoder" 42 bytes {1} [built]
[191] external "http" 42 bytes {1} [built]
[192] external "https" 42 bytes {1} [built]
[200] ./src/handlers/goodbye.ts 190 bytes {0} [built]
    + 186 hidden modules

hello.js is now 195kb while goodbye.js remains just over 1kb. Console is looking pretty good at this point. When writing Lambda functions, it's advisable to minimize imported dependencies. In fact some node dependencies (as opposed to browser app dependencies) don't minify very well. You may find yourself needing to separate your code and dependencies using something like webpack node externals to get around this problem.

Also note in my output above all the dependencies marked external. This is because I set target: node in my webpack config. It means webpack is assuming these libraries will simply be available in my target environment

SAM Template

The SAM templating engine is an extension of CloudFormation. SAM templates can refer to both SAM resources and CloudFormation resources. When deployed, SAM will create a CloudFormation stack, a group of resources created in your AWS account. Read more about SAM and CloudFormation.

SAM (and CloudFormation) templates are written in yaml. The cli tool has a built-in validator so you'll never need to worry about deploying invalid yaml. There's even a sam validate command which is run automatically with sam deploy.

% sam validate
/Users/mattmorgan/mine/sam-typescript-webpack/template.yaml is a valid SAM Template

Now as for the template itself, every SAM template will begin with something that looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-typescript-webpack

  Sample SAM Template for sam-typescript-webpack

AWS versions its specifications by the date it was published. These template versions don't change often and AWS is known for supporting backward compatibility.

Globals:
  Function:
    CodeUri: build/
    Runtime: nodejs12.x
    Timeout: 300

Globals are a feature unique to SAM that let me write a slightly more DRY config file and work as you'd expect. You can override any global at the resource level.

In this case, I'm specifying that all my code will come from the build directory, setting my node version and function timeout (5 minutes).

Resources:
  HelloFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      Handler: hello.handler
      Events:
        Hello:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
  GoodbyeFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      Handler: goodbye.handler
      Events:
        Goodbye:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /goodbye
            Method: get

I'm defining my functions here. hello.handler means that inside the CodeUri location (specified in Globals), there will be a hello.js that exposes a handler function.

I don't have to explicitly declare the ApiGateway service as a Resource. Setting Type: Api on my Function is enough. I might prefer to use HttpApi, but it is still in beta and is not yet supported in sam local runs.

Finally I'll define some Outputs:

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloApi:
    Description: 'API Gateway endpoint URL for Prod stage for Hello function'
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/'
  GoodbyeApi:
    Description: 'API Gateway endpoint URL for Prod stage for Goodbye function'
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/goodbye/'

In this case, I'm outputting the URLs that ApiGateway assigned to my functions so I can start testing.

Local Test

sam local can be used to invoke a Lambda function and test different payloads and triggers. In this case I will use start-api to run my Functions locally with an Api Gateway.

% sam local start-api
Mounting GoodbyeFunction at http://127.0.0.1:3000/goodbye [GET]
Mounting HelloFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-02-09 22:57:15  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

I navigate to http://127.0.0.1:3000/hello and see {"message":"hello"}. Easy stuff.

Deploy

The first time I deploy, it's a good idea to use sam deploy --guided. I of course need credentials for my AWS account to do this.

% sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: sam-typescript-webpack
    AWS Region [us-east-1]:
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]:
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]:
    Save arguments to samconfig.toml [Y/n]:

    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-16ecthy96ctvq
        A different default S3 bucket can be set in samconfig.toml

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

    Deploying with following values
    ===============================
    Stack name                 : sam-typescript-webpack
    Region                     : us-east-1
    Confirm changeset          : False
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-16ecthy96ctvq
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================
Uploading to sam-typescript-webpack/29e43b24a388e6fc03dfca21d52effe2  57738 / 57738.0  (100.00%)
Uploading to sam-typescript-webpack/1d786b318a187cf6c9952adbe105dfb8.template  1387 / 1387.0  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                         LogicalResourceId                                                 ResourceType
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                             GoodbyeFunctionGoodbyePermissionProd                              AWS::Lambda::Permission
+ Add                                                             GoodbyeFunctionRole                                               AWS::IAM::Role
+ Add                                                             GoodbyeFunction                                                   AWS::Lambda::Function
+ Add                                                             HelloFunctionHelloPermissionProd                                  AWS::Lambda::Permission
+ Add                                                             HelloFunctionRole                                                 AWS::IAM::Role
+ Add                                                             HelloFunction                                                     AWS::Lambda::Function
+ Add                                                             ServerlessRestApiDeploymenteca00d9d3f                             AWS::ApiGateway::Deployment
+ Add                                                             ServerlessRestApiProdStage                                        AWS::ApiGateway::Stage
+ Add                                                             ServerlessRestApi                                                 AWS::ApiGateway::RestApi
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:336848621206:changeSet/samcli-deploy1581337063/c7e95df9-bf73-4360-ab9f-9ce538bac740


2020-02-10 07:17:49 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                   ResourceType                                     LogicalResourceId                                ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                               AWS::IAM::Role                                   GoodbyeFunctionRole                              Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::IAM::Role                                   HelloFunctionRole                                Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::IAM::Role                                   GoodbyeFunctionRole                              -
CREATE_IN_PROGRESS                               AWS::IAM::Role                                   HelloFunctionRole                                -
CREATE_COMPLETE                                  AWS::IAM::Role                                   GoodbyeFunctionRole                              -
CREATE_COMPLETE                                  AWS::IAM::Role                                   HelloFunctionRole                                -
CREATE_IN_PROGRESS                               AWS::Lambda::Function                            HelloFunction                                    -
CREATE_IN_PROGRESS                               AWS::Lambda::Function                            HelloFunction                                    Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::Lambda::Function                            GoodbyeFunction                                  Resource creation Initiated
CREATE_COMPLETE                                  AWS::Lambda::Function                            HelloFunction                                    -
CREATE_IN_PROGRESS                               AWS::Lambda::Function                            GoodbyeFunction                                  -
CREATE_COMPLETE                                  AWS::Lambda::Function                            GoodbyeFunction                                  -
CREATE_IN_PROGRESS                               AWS::ApiGateway::RestApi                         ServerlessRestApi                                Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::ApiGateway::RestApi                         ServerlessRestApi                                -
CREATE_COMPLETE                                  AWS::ApiGateway::RestApi                         ServerlessRestApi                                -
CREATE_IN_PROGRESS                               AWS::ApiGateway::Deployment                      ServerlessRestApiDeploymenteca00d9d3f            -
CREATE_IN_PROGRESS                               AWS::Lambda::Permission                          HelloFunctionHelloPermissionProd                 -
CREATE_IN_PROGRESS                               AWS::Lambda::Permission                          GoodbyeFunctionGoodbyePermissionProd             -
CREATE_IN_PROGRESS                               AWS::ApiGateway::Deployment                      ServerlessRestApiDeploymenteca00d9d3f            Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::Lambda::Permission                          HelloFunctionHelloPermissionProd                 Resource creation Initiated
CREATE_IN_PROGRESS                               AWS::Lambda::Permission                          GoodbyeFunctionGoodbyePermissionProd             Resource creation Initiated
CREATE_COMPLETE                                  AWS::ApiGateway::Deployment                      ServerlessRestApiDeploymenteca00d9d3f            -
CREATE_IN_PROGRESS                               AWS::ApiGateway::Stage                           ServerlessRestApiProdStage                       -
CREATE_IN_PROGRESS                               AWS::ApiGateway::Stage                           ServerlessRestApiProdStage                       Resource creation Initiated
CREATE_COMPLETE                                  AWS::ApiGateway::Stage                           ServerlessRestApiProdStage                       -
CREATE_COMPLETE                                  AWS::Lambda::Permission                          HelloFunctionHelloPermissionProd                 -
CREATE_COMPLETE                                  AWS::Lambda::Permission                          GoodbyeFunctionGoodbyePermissionProd             -
CREATE_COMPLETE                                  AWS::CloudFormation::Stack                       sam-typescript-webpack                           -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Stack sam-typescript-webpack outputs:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OutputKey-Description                                                                             OutputValue
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HelloApi - API Gateway endpoint URL for Prod stage for Hello function                             https://w32i72j3pc.execute-api.us-east-1.amazonaws.com/Prod/hello/
GoodbyeApi - API Gateway endpoint URL for Prod stage for Goodbye function                         https://w32i72j3pc.execute-api.us-east-1.amazonaws.com/Prod/goodbye/
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sam-typescript-webpack in us-east-1

Everything looks good! I can navigate to the urls about in my browser and then find output in CloudWatch.

Next Steps

SAM is a very complete tool when it comes to building out new serverless applications, but some additional work is required if you want to connect to an existing RDS or enhance an existing local environment with serverless functions. However this is possible, so maybe I'll go through that in another post.

Update

Thanks to some very helpful feedback from @sandinosaso and @borduhh in the comments, the entries are determined dynamically by reading the SAM template. It could make a lot of sense to follow this pattern if you're putting a lot of functions in a single stack. If you aren't doing that many, hardcoding could still make sense.

Cover Image - St. Brendan's ship on the back of a whale, and his men praying, in Honorius Philoponus' Nova typis transacta navigation 1621; image from Sea Monsters on Medieval and Renaissance Maps by Chet Van Duzer

Comments

[Erick]

Great article! I'm coming from the J2EE world and this is a huge help on how to get a project up & going!

Quick question - how did you get the prettier output from webpack? Really like the pretty/organized look.

[Matt Morgan]

Hi Erick, thanks for reading. I can't really claim any credit for that output. I think webpack 5 provides that out of the box. Did you have trouble seeing something like that in your own project?

[Erick]

Yeah...what I posted is the current webpack. I'll just play around with it.

Thanks for the fast reply...if I find something neat I'll follow up here.

[Ayoub Korkot]

Hello Matt Morgan
Thanks a lot for this great article.

I tried to use this boilerplate as a starter kit for one of our new serverless applications.

I used "webpack-node-externals" as an alternative to tell webpack to ignore node_modules libs, but unfortunately SAM does not recognize packages outside generated webpackes microservices when running "sam local start-api or start-lambda"....

After searching on the internet it seems there is no way to tell SAM to load externals node_modules folder when trying to simulate api / lambda locally...

Do you have any Idea about this issue ? Maybe a workaround for this please ?

Thanks in advance for your help!

[Matt Morgan]

Hi Ayoub, working with node_modules externals is a bit odd in SAM. This blog post may help you understand how it works: aws.amazon.com/blogs/compute/worki...

I was able to get this project working by following that technique. You can see this branch: github.com/elthrasher/sam-typescri...

My preference is to bundle modules in my functions and not make them external, but if you want to do that, that should get you started. I really don't like having to repeat the package.json and if you have a lot of dependencies, that could really be a burden.

Also since writing this blog post, I've moved most of my bundling to esbuild.github.io/. Recommend you give that a look as well as it really speeds things along.

[OMID EIDIVANDI]

Hi i'm challenging with that in a serverless , i like to get working sam ;)

i have a structure as

configs/
cfn/
DoThis
func1.yml
DoThat
func1.yml
DataStoreProcess
dynamodb.yml
StorageProcess
s3.yml
ProcessMessageBus
SNS.yml
SQS.yml

src
/node
/netcore
infra
scripts

have you any idea about theses kind of Situ

[OMID EIDIVANDI]

finally i foud the way to handle all nested templates using this part of code

const templates = walkSync(join(__dirname, "../../cfn/"));
const AllEnteries: any[] = [];

templates.forEach((fl) => {
    const { Globals, Resources } = yamlParse(readFileSync(fl, "utf-8"));
    const GlobalFunction = Globals?.Function ?? {};
    if (Resources) {
        const entries = Object.values(Resources)
            .filter((resource: ISamFunction) => resource.Type === "AWS::Serverless::Function")
            .filter((resource: ISamFunction) => (resource.Properties?.Runtime ?? GlobalFunction.Runtime).startsWith("nodejs"))
            .map((resource: ISamFunction) => ({
                filename: resource.Properties.Handler.split(".")[0],
                entryPath: resource.Properties.CodeUri.split("/").join("/"),
                name: resource.Properties.CodeUri.split("/").pop()
            }))
            .reduce(
                (resources, resource) =>
                    Object.assign(resources, {
                        [`${resource.name}`]: `${handlerPath}/${resource.name}/${resource.filename}.ts`,
                    }),
                {},
            );

        AllEnteries.push(entries);
    }
});

const webpackEnteries = Object.assign({}, ...AllEnteries);

[Nick Bordeau]

Great article! Whenever I try to use a "build" or "dist" folder I get the following errors:

Build Failed
Error: NodejsNpmBuilder:NpmPack - NPM Failed: npm ERR! code ENOLOCAL
npm ERR! Could not install from "/path/to/dist/folder" as it does not contain a package.json file.

I would love to learn about how you got around that!

[Matt Morgan]

Hey Nick, thanks for reading! If you're getting that error, I assume you're doing a sam build? Using the technique in this article, you won't use sam build and the reasons for that are outlined in the article. Instead you npm run build and then sam local start-api or sam deploy. If that's what you're doing and you're still getting the error above, let me know and I'll try to figure it out. It's been a bit since I've run this.

[Nick Bordeau]

How would this work if I were to try to add dependency layers?

[Matt Morgan]

I haven't actually done that myself with webpack - since the point of webpack is to bundle and tree-shake all the dependencies - but if you are solid on creating a Lambda layer with your modules, you'd just need to set whatever dependencies are in the layer as an external. See webpack.js.org/configuration/exter...
A lot of people do this with aws-sdk since it's already available in Lambda, but I've seen benchmarks show that you can actually get slightly faster cold starts if you bundle it so that's why I didn't do that in this example (though I have done it and TBH haven't noticed any difference either way).
If already using webpack with SAM, I'd probably only worry about Lambda layers if A) I had some huge dependency or B) I had some layer I wanted to share across many projects.

[Nick Bordeau]

Do you have an example of what that structure looks like? Right now, I package everything separately with Webpack and then build the application with sam build. Here's the basic repo structure I use: github.com/Borduhh/serverless-SAM-...

I'm curious to know if this might be a more efficient way to do it though.

[Matt Morgan]

The main thing I'm going for is a single package.json at the root of my repo, not multiple throughout. I introduced lerna into a UI project last year and my team hated having to manage dependencies (i.e. update them!) at multiple layers throughout the project. We ended up pulling that out for a single webpack build that could produce multiple bundles from the entrypoints. It's much cleaner and builds much faster!
So my article is about applying those same principles to building multiple Lambda functions from the same code base without multiple package.json files.
Like I said, I haven't worked with Lambda layers because I just webpack my entire project. What's the use case for using layers in your app? Are you just trying to make the bundle smaller? Share something across several code repos?

[Nick Bordeau]

Yeah, that makes a lot of sense. I am using layers for precisely that reason, to avoid having to go into each package and update shared dependencies (i.e. a lot of our functions will use the aws-sdk package so I have an AwsSdkLayer which loads that subset of tools). That way I can pop into the layer, and update the dependency once and I am done.

It sounds like this would be similar though in a more efficient manner. I am just am having trouble wrapping my head around what a larger project structure would look like with multiple functions that share the same code.

[Matt Morgan]

The project structure ends up looking a lot like your typical Express or NestJS app. That's the great thing about this approach. So we might do something like
/aws
/handlers
/repositories
/utils

The handlers are like controllers or routes in your average NodeJS app and everything else is shared or at least potentially shared. If you are somewhat careful in your imports, you should be able tree-shake away the things you don't need when you build. I did have a developer point out a little leakage with some extra unnecessary code showing up in a function due to the way things were being imported, but it hardly added any size and our functions are around 5MB, well below the 50MB limit.

I don't really have any other insights around lambda layers except I haven't felt the need to use them except for adding binaries to a lambda runtime.

We also follow the practice of putting a unit test right next to each of the code modules. The tree-shaking approach helps there too since obviously none of the production code has imports from the tests.

[Sandino]

Thanks for sharing this is a great post!!.

Regarding your concern about your approach "adding a new function means touching both your template.yaml and your webpack.config.ts files." someone else figured out how to keep them in sync:

// Extract the AWS::Serverless::Function Resources as they
// are the entires we need to compile.
const { Resources } = CloudFormation.yamlParse(fs.readFileSync('template.yml'))

const entries = Object.values(Resources)
  .filter(resource => resource.Type == 'AWS::Serverless::Function')
  .filter(resource => resource.Properties.Runtime.startsWith('nodejs'))

  .map(resource => {
    const file = resource.Properties.Handler.split('.')[0]
    const prefix = resource.Properties.CodeUri.substr(9)

    return {
      name: `${prefix}/${file}`,
      entry: `${prefix}/${file}.ts`,
    }
  })

  .reduce((accumulator, resource) => {
    const { name, entry } = resource

    return Object.assign(accumulator, {
      [name]: path.resolve(entry),
    })
  }, {})

module.exports = {
  entry: entries,
  ...
}

Here is the full config: gist.github.com/henrikbjorn/d2eef5...

I think combining both approaches is a great toolchain to have.

Thank you.

[Nick Bordeau]

Here's the Typescript version as well.

/** Interface for AWS SAM Function */
interface ISamFunction {
  Type: string;
  Properties: {
    AssumeRolePolicyDocument?: JSON;
    AutoPublishAlias?: string;
    AutoPublishCodeSha256?: string;
    CodeUri?: string;
    Description?: string;
    Environment?: {
      Variables: {
        [key: string]: string;
      };
    };
    Events?: EventSource;
    FunctionName?: string;
    Handler: string;
    Layers?: { [Ref: string]: string }[];
    Runtime: string;
    Timeout?: number;
    Tracing?: string;
    VersionDescription?: string;
  };
}

const { resources } = yamlParse(readFileSync(conf.templatePath, 'utf-8'));

const entries = Object.values(resources)

  .filter((resource: ISamFunction) => resource.Type === 'AWS::Serverless::Function')

  .filter(
    (resource: ISamFunction) =>
      resource.Properties.Runtime && resource.Properties.Runtime.startsWith('nodejs')
  )

  .map((resource: ISamFunction) => ({
    filename: resource.Properties.Handler.split('.')[0],
    entryPath: resource.Properties.CodeUri.split('/').splice(3).join('/'),
  }))

  .reduce(
    (resources, resource) =>
      Object.assign(resources, {
        [`${resource.filename}`]: `./src/${resource.entryPath}${resource.filename}.ts`,
      }),
    {}
  );

[Matt Morgan]

Nice and thanks for the reminder. Got to get around to trying this!

[Matt Morgan]

After leaving the same silly comment twice, I finally got around to putting this in and it works great. Thanks so much for the input!

[Matt Morgan]

Thanks so much for the comment. I'll definitely try this!

[Patrick Foley]

This was super useful - thanks!

I remember when create-react-app didn't have typescript and then after a while, it did. I hope and ultimately suspect that sam will adopt an approach something like yours in the future as well. Think it will?

[Matt Morgan]

Glad you found this helpful, Patrick!

As for the future of sam build, I don't doubt that eventually we'll get more customization options, however the sam team has their work cut out for them in supporting node, dotnet, ruby, golang, python and java. The sam build process is a lot less skippable for some of those other languages as they all package dependencies in different ways.

I'll also say that while I love this tool, I find when I read through AWS docs and examples, that I have a philosophical difference. For example, CDK TypeScript examples that write lambda functions in vanilla JavaScript. If you're already writing TypeScript and have committed to a build process, why not gain all the benefits? Another thing I notice is multiple package.json files in the same project tree. Seems like unnecessary complexity and that's why I wanted to blog about simpler alternatives.

At the end, these problems can, will and are being solved by the community (and not just me) and my suspicion is that AWS is quite fine to leave it to us while they focus on the really hard problems like making linux faster.

[Matt Morgan]

Just FYI, was discussing a related topic with a colleague and I came across this: github.com/awslabs/aws-sam-cli/blo...

This isn't implemented. You can find an issue for a design and an aborted PR on github, but they definitely plan to have a more flexible build system, most likely that will entail adding your own transpilation/etc step, at least to start.

[Matt Martz]

For your resolve extensions... you list them in order of .js and then .ts... (which is the order that webpack looks for them in)... shouldn't you have .ts first?

I guess if you have a module with both .js and .ts extensions you have other problems anyways... so maybe it doesn't matter?

[Matt Morgan]

That would only matter if some directory had both js and ts, correct. I'm not supporting js because I think I might write some but because I'm webpacking node modules. In the unlikely event some module has both a js and ts file, I probably want to prioritize the js, as that's what the author intended.