Optimising package size for Typescript AWS Lambda functions using serverless-esbuild

Hugh — Thu, 18 Jul 2024 11:05:37 +0000

Here is how I reduced the package size of my lambda functions from 60MB per function to ~1MB and cold-start init times from 606ms to 291ms.

Introduction

We have recently been using the serverless-esbuild plugin over serverless-plugin-typescript package in new Typescript serverless projects as it can transpile much faster (saving development time) and is easy to optimize for bundle sizes.

However, without proper configuration, your transpiled code can end up surprisingly large. This can impact the speed of deployment pipelines and cold start times of your lambda functions.

Related article: Size is (almost) all that matters for optimizing AWS Lambda cold starts | by Adrian Tanasa | Medium.

Furthermore, if the package size of a single function gets too large (50MB zipped), it will actually fail to deploy due to AWS limits on lambda package size (this happened to me and was the motivation for me to optimise file size and make this guide).

So, here are the steps to optimise serverless-esbuild settings:

Analyzing your bundle size

To package your serverless project locally, run:

serverless package -s offline

Note: the stage option (-s offline) is optional, depends on if you use different stages or not.

This generates a .serverless folder in the root of your project containing a zip file of your package. Open the zip file to check the index.js and index.js.map file sizes.

By setting the metafile option in serverless.yml, you can analyze the size of dependencies.

# serverless.yml
custom:
  esbuild:
    metafile: true # for local analysis only!

Don’t forget to comment out this option or revert to false before deploying, as the metafile is unnecessary in production and will increase the package size.

Afte packaging your project again, you should see the meta files in the zipped package (index-meta.json).

You can use the esbuild - Bundle Size Analyzer tool to easily analyze the size of your dependencies:

If you see packages that shouldn't be there in the function (e.g. a mysql library where your function doesn't perform any DB operations), find where it's imported and remove it.

If you're unsure how the package was imported, it may be a dependency of another package. To locate which package uses the subpackage, you can use the npm explain command:

npm explain package-name

Optimize esbuild settings

#serverless.yml
plugins:
  - serverless-esbuild

package:
  individually: true

custom:
  # Here you can define your esbuild options
  esbuild:
    bundle: true
    sourcemap: true
    sourcesContent: false # Omits original source code from sourcemaps (set to false to allow debugging in AWS)
    minify: true
    exclude: ['@aws-sdk']
    # custom plugin to remove vendor sourcemaps
    plugins: esbuild-plugins.js

Explanation of settings:

package:
  individually: true

The option will create a separate zip file for each lambda function, speeding up deployment and lowering size per function. If you don’t set this to true, every one of your lambda functions will contain the entire source code of the whole project and all dependencies, potentially increasing package size a lot.

sourcemap: true

Sourcemap option should be true because we want to be able to locate stack traces thrown in production environments easily. Note: to enable sourcemaps in AWS, the environment variable NODE_OPTIONS=--enable-source-maps must also be set.

sourcesContent: false

By default esbuild will duplicate your source code in the sourcemaps, so that debugging can be possible with the AWS debugger. However, this can increase bundle size significantly.

However, there is a major downside: sourcemaps will include all node_modules packages resulting in an increase in package size. I will explain how to remedy this in the next section.

minify: true

Minifies code, resulting in further file size reduction

exclude: ['@aws-sdk']

Exclude aws-sdk from your package (and move it from dependencies to devDependencies in your package.json), since Node.js lambda functions automatically have access to the AWS libraries when deployed!

Removing vendor source maps

The source map files generated by esbuild can be extremely large if you have many heavy dependencies (larger than the source code). This is because esbuild generates sourcemaps not just for your code, but for all imported node_modules too.

While it can sometimes be useful to have vendor sourcemaps when debugging code, the use case for debugging 3rd party package’s source code from production stack traces is quite uncommon.

To exclude vendor scripts from your source files, we need to write a custom plugin for esbuild.

First add this option to your serverless.yml:

# serverless.yml
custom:
  esbuild:
    plugins: esbuild-plugins.js

Then create a esbuild-plugins.js file in the same directory as the serverless.yml file:

// esbuild-plugins.js

const { readFileSync } = require('fs');

const excludeVendorFromSourceMapPlugin = {
  name: 'excludeVendorFromSourceMap',
  setup(build) {
    build.onLoad({ filter: /node_modules/ }, (args) => {
      if (args.path.endsWith('.js') || args.path.endsWith('.mjs') || args.path.endsWith('.cjs')) {
        return {
          contents:
            readFileSync(args.path, 'utf8') +
            '\n//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpbIiJdLCJtYXBwaW5ncyI6IkEifQ==',
          loader: 'default',
        };
      }
    });
  },
};

module.exports = [excludeVendorFromSourceMapPlugin];

What this script does is replace sourcemaps of files from the node_modules folder with an empty sourcemap template.

Applying this plugin reduced sourcemap file in each lambda function of my project from 10MB to about 2.5MB.

Avoid importing dependencies that aren’t needed

ESBuild has tree shaking built in, meaning that only modules your lambda function imports (from the module containing lambda handler's entry point) will be included in the bundle. However, it is easy to unintentionally import modules that your lambda function doesn’t need indirectly through commonly shared modules.

Use the bundle analyzer and find out if there are any large dependencies that look like they shouldn’t be needed in the specific lambda function you’re analyzing.

In one project, I used the sequelize package to connect to a database. However, only some of these functions actually required a database connection.

The code was unintentionally importing the ‘sequelize’ package in every lambda function indirectly through a commonly imported module, adding around 1MB of script size per lambda function.

A simple refactor to remove the sequelize import when a DB connection wasn’t needed reduced the file size significantly.

Conclusion

If your main goal is to save on infrastructure costs, spending a lot of time to optimise your lambda package sizes may not be worth the effort.

Although you won't be charged extra by AWS for having larger package sizes, there may be a significant performance impact on cold starts.

Double checking your esbuild configuration is a low-hanging-fruit that can reduce a lot of "digital waste" with minimal effort.

Before optimising esbuild, my lambda package was 60MB, which was higher than the limit allowed by AWS.

After reducing this to 35MB with excluding vendor sourcemaps, I still hadn’t packaged functions individually, so every single lambda function would have contained the full 35MB of source code + vendor scripts, making the total storage space used 630MB over 18 lambda functions.

These optimization changes reduced package size to around 0.5MB-1.5MB per function, totalling only 16MB uploaded to AWS, a dramatic reduction.

Furthermore, the cold start init duration of a simple ‘Hello World’ function was reduced from 606ms to 291ms.

Summary

Add aws-sdk to the exclude array in esbuild options (reducing script slightly) - the aws library should exist in the deployed environment.
Set the esbuild minify option to true in serverless.yml (another slight reduction)
Package functions individually using serverless.yml setting (large reduction of total size uploaded to AWS for multiple functions)
Add a plugin to exclude vendor sourcemaps from the scripts (big reduction) exclude node_modules from source map Issue #1685 · evanw/esbuild · GitHub
Use the serverless package command locally to check the file size output in .serverless folder.
Use esbuild - Bundle Size Analyzer (after setting metafile esbuild option to true) to observe dependencies size

DEV Community: Hugh