TypeScript is a popular language for developers of all kinds and it's made its mark on the serverless space. Most of the major Lambda frameworks now have solid TypeScript support. The days of struggling with webpack configurations are mostly behind us.
Table of Contents
- Stack Traces
- Emitting Source Maps
- Source Map Support
- CDK Example
- SAM Example
- Serverless Framework Example
- Benchmarks
- Conclusion
Stack Traces
If we're already transpiling our code to convert the TypeScript source to Lambda-friendly JavaScript, we might as well go ahead and minify and tree-shake the code as well. Smaller bundles can make deployments faster and may even help with cold start and execution time. However, this can make debugging difficult when we start seeing stack traces that look like this:
{
"errorType": "SyntaxError",
"errorMessage": "Unexpected end of JSON input",
"stack": [
"SyntaxError: Unexpected end of JSON input",
" at JSON.parse (<anonymous>)",
" at VA (/var/task/index.js:25:69708)",
" at Runtime.R8 [as handler] (/var/task/index.js:25:69808)",
" at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
]
}
The error message here lets us know that we're failing to parse a JSON string, but the stack trace itself is useless for finding the line where the code is failing. We'll have no choice but to look through our code and hope to find the error. We might be able to do a text search for JSON.parse but if that is happening in one of our dependencies, searching won't work. What's next? Add a bunch of log statements to the code? No! If we use Source Maps, we can get more useful stack traces:
{
"errorType": "SyntaxError",
"errorMessage": "Unexpected end of JSON input",
"stack": [
"SyntaxError: Unexpected end of JSON input",
" at JSON.parse (<anonymous>)",
" at VA (/fns/db.ts:39:8)",
" at Runtime.R8 (/fns/list.ts:6:24)",
" at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
]
}
Now we can see the stack includes a call from line 6 of list.ts which finally fails on line 39 of db.ts.
To enable Source Maps in our application, we need to tell our build tool to emit Source Maps and we need to enable Source Map support in our runtime.
Emitting Source Maps
Emitting Source Maps is very easy with esbuild. We simply set the boolean property in our configuration. Now when we run the build, we'll get an index.js.map file as well as our index.js. This file must be uploaded to the Lambda service. We'll see how to do that with AWS CDK, AWS SAM and the Serverless Framework a bit later in this article.
Source Map Support
Having the index.js.map file in our Lambda runtime isn't sufficient to enable Source Maps. We also have to make sure the runtime knows to make use of them. Fortunately this is very easy ever since Node.js version 12.12.0. We just have to set the --enable-source-maps command line option. Command line options can be set in AWS Lambda by setting the NODE_OPTIONS environment variable. At the time of this writing, AWS Lambda supports Node.js versions 12 and 14. AWS does not publish the minor versions in use in Lambda, but we can discover it by logging out process.version in a function. As of late January, 2022, the Node.js versions in use in Lambda in us-east-1 are v12.22.7 and v14.18.1 so we'll have no trouble using Source Maps.
If we needed to enable Source Maps in a runtime that doesn't support the native version, we could always use Source Map Support.
CDK Example
All example code is available on GitHub.
AWS CDK is my preferred tool for writing and deploying serverless applications, in part because of the aws-lambda-nodejs construct. This construct makes it very easy to work with TypeScript. It wraps esbuild and exposes options. It also supports setting environment variables.
When I'm working with multiple Lambda functions, I often find it helpful to create a single props object that then gets shared among multiple functions.
const lambdaProps = {
architecture: Architecture.ARM_64,
bundling: { minify: true, sourceMap: true },
environment: {
NODE_OPTIONS: '--enable-source-maps',
},
logRetention: RetentionDays.ONE_DAY,
runtime: Runtime.NODEJS_14_X,
memorySize: 512,
timeout: Duration.minutes(1),
};
new NodejsFunction(this, 'FuncOne', {
...lambdaProps,
entry: `${__dirname}/../fns/one.ts`,
});
new NodejsFunction(this, 'FuncTwo', {
...lambdaProps,
entry: `${__dirname}/../fns/two.ts`,
});
As we can see, it's very simple to enable Source Maps when already using AWS CDK and NodejsFunction.
Serverless Stack
Serverless Stack is a very cool value add that builds on top of AWS CDK delivering an awesome developer experience and dashboard. SST's own version of NodejsFunction, Function automatically emits source maps. Their use can be enabled simply by setting NODE_OPTIONS as described.
SAM Example
SAM support for TypeScript has been lagging for some time, but a pull request was just merged that should change that. It looks like we'll be able to add an aws_sam key in the package.json file to enable building within the SAM engine as part of sam build. Although this PR has been merged to aws-lambda-builders, the engine behind sam build, it will still need to be added to aws-sam-cli and released (to much fanfare, one expects) before it can be used with SAM.
Meanwhile - or if we're considering other options for deploying our functions - we can add an extra build step. We'll create an esbuild.ts file that transpiles the functions and then point our SAM template.yaml file at the output of that step.
import { build, BuildOptions } from 'esbuild';
const buildOptions: BuildOptions = {
bundle: true,
entryPoints: {
['create/index']: `${__dirname}/../fns/create.ts`,
['delete/index']: `${__dirname}/../fns/delete.ts`,
['list/index']: `${__dirname}/../fns/list.ts`,
},
minify: true,
outbase: 'fns',
outdir: 'sam/build',
platform: 'node',
sourcemap: true,
};
build(buildOptions);
SAM templates will take everything in the CodeUri so the above code will transpile a TypeScript file at fns/list.ts and output sam/build/list/index.js and sam/build/list/index.js.map. By setting CodeUri: sam/build/list, SAM will package the two files and upload them to Lambda.
Now we just need to set the environment variable. This is easy enough to do in a SAM template. We can set it as a global so it only needs to be in the template once
Globals:
Function:
Environment:
Variables:
NODE_OPTIONS: '--enable-source-maps'
In order to make sure we always build before deploying, we can add some npm scripts.
"scripts": {
"build:lambda": "npm run clean && ts-node --files sam/esbuild.ts",
"clean": "rimraf cdk.out sam/build",
"deploy:sam": "npm run build:lambda && sam deploy --template template.yaml",
"destroy:sam": "sam delete"
}
This works well enough, but does take some extra effort. SAM users are no doubt eagerly awaiting better TypeScript support.
Architect
Alternately, use Architect. Architect is a 3rd party developer experience that builds on top of AWS SAM. Architect includes a TypeScript plugin.
Serverless Framework Example
The Serverless Framework is known for its plugin system. serverless-esbuild brings bundling and all the options we need to support Source Maps in TypeScript into the sls package and sls deploy commands.
The plugin is configured in our serverless.yml file.
custom:
esbuild:
bundle: true
minify: true
sourcemap: true
And then we just point our functions at our TypeScript handlers.
functions:
create:
handler: fns/create.handler
Much like SAM, Serverless lets us set global environment variables for our functions.
provider:
name: aws
lambdaHashingVersion: 20201221
runtime: nodejs14.x
environment:
NODE_OPTIONS: '--enable-source-maps'
Much like AWS CDK, this is a good experience for TypeScript developers. Those already using Serverless Framework should have an easy time adding Source Maps to their applications.
Benchmarks
There's a lot of guidance against using Source Maps in production because of a supposed negative performance impact. Let's measure the admittedly-simple list function and see if minification or the use of source maps has any noticeable effect.
I used autocannon to test the function at 100 concurrent executions for 30 seconds. I also used Lambda Power Tuning to find the ideal memory configuration, which proved to be 512MB. All the results are available.
Not Minified without Source Maps
The unminified function is 1.2MB in size. This is mostly from @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb. Whether sticking with SDK v2 might be better performance would be a good topic for another post. This function is over one MB, despite a small amount of custom code.
Running the test, the function has an average execution of 46.99ms and a max of 914ms. 99% of my requests are at or below 90ms.
Minified without Source Maps
Minifying the function drops the size to 534.8kb, less than half the size. My test showed an average response of 47.83ms, a max of 1010ms and 90ms at the 99th percentile. This is slightly worse, but not statistically significant. I expect if I ran these tests over and over again, I would see that minifying the code has no real effect on performance. This is not a surprise. 1.2MB is still fairly small and I don't expect to see much in the way of added latency at that size.
Minified with Source Maps
The minified function is still 534.8kb, of course, but the Source Map is 1.5MB so this will be the largest upload, not that ~2MB is a lot or will significantly slow down our deployments.
The average response time for this test is 46.52ms, max is 968mb and the 99th percentile is 82ms. This is the best result so far, but not statistically significant. I would say this is truly a three-way tie which tells us that adding Source Maps to this function did not increase latency.
That's because of the native support in Node.js! Since no stack traces were emitted, the Source Maps were never referenced. It's possible if we had to rely on a library for this capability, we wouldn't have the same outcome.
Error Minified without Source Maps
Will triggering error conditions make a difference? Let's see. I ran the same test but this time the function has that JSON.parse error in it. This happens before the call to DynamoDB, so we can expect it to be a little faster than the successful function. We get 37.86ms as the average, 1004ms as the max and 58ms at 99%. It's impressive the call to DynamoDB only seems to add about 10ms of latency!
Error Minified with Source Maps
Enabling Source Maps for the errors does impact performance. Our average has dropped to 97.54ms, max at 1129ms and 99% is 243. This is a significant increase. Source Maps do impact latency when an error occurs. This makes sense and confirms the idea that the Source Maps are only referenced when an error occurs - but now that Source Map must be parsed and that takes time.
Conclusion
Use Source Maps in production! In my view, if you are getting so many errors that the performance hit from Source Maps is impacting your bottom line, you should probably go and fix those errors. It's very easy to implement Source Maps in a variety of popular Lambda frameworks and they don't impact successful execution. When functions do fail, the useful stack trace is going to be worth the added latency. Developer hours are always going to be more expensive than a few milliseconds of Lambda execution.
Top comments (4)
Thanks for this! Any thoughts on externalizing source maps from the bundle? For instance by storing them on a web server and providing a source map reference as
https://link? That way we can deploy a much smaller lambda function. However, I guess that would expose all our server source code, unless there would be a way to put the source map somewhere only the Lambda can access it?Hi Max, thanks for reading! I think keeping the source map in Lambda storage is what you want to do. I doubt deploy speeds are going to be impacted too much unless you have really huge Lambda functions. The main reason I would NOT enable SourceMaps is if you are using try/catch for purposes other than rare error conditions (and ideally ones you intend to fix). Some devs use try/catch for flow control, such as:
If your code, or a library you consume does this, then SourceMaps might add significant latency to your app. Likewise if you have a high error rate, SourceMaps might compound your woes by making the customer experience even worse and making your app more expensive to operate.
Hope this helps!
Yes that sounds reasonable - it is a bit unfortunate that source maps easily double the deployment size for Lambdas - but as per your measurements the performance impact from that increased size seems to be quite negligible. I guess it should also be possible to have CloudWatch log out the stack trace without source mapping and then reverse-engineer that locally, although I didn't seem to find too much information on how to do that!
I don't know of a way to reverse-engineer the stack trace into something readable using a SourceMap. It might not be possible because your logged stack trace is probably missing some details.
IMO the cost of a larger bundle isn't a very high price since it doesn't impact latency unless you throw an error. I suspect a lot of people will compromise by enabling sourcemaps early in projects or when there's a tricky bug to fix.