DEV Community

Cover image for Cleanup the node_modules for a lighter Lambda Function

Cleanup the node_modules for a lighter Lambda Function

Vikas Solegaonkar on December 09, 2020

Any nodejs project carries a bulky folder - the node_modules - that carries all the modules and dependencies that the application would need. If yo...
Collapse
 
zachlitz profile image
Zach

You raise some good points here.
But surely you would be 100% better off simply using Webpack or some other bundler/packager.
Everything you are doing in this post gets undone on npm update/install, and is not very efficient. Modern tools will do a much better job and also things lime dead code removal and tree shaking.
Webpack is a good start because it has so many plugins and is easy to add ne functionality (though it does have a bit of a learning curve).

For all the people out there, this post gives good thing to think about, but there are much cleaner, more robust, and more efficient ways.
And yes, Weback will handle pretty much any workflow and file/dependency type. But even if you're doing something very unusual you can still use other tools along side Webpack.

Collapse
 
brianleroux profile image
xnoɹǝʃ uɐıɹq

Webpack is a great solution until you need to debug async code in a running Lambda. No line numbers on stack traces. No idea what failed or where it is. Now you have to redeploy just to debug or play the commenting/console.log dart throwing game. We found in arc.codes that sub 5mb Lambdas coldstart sub second. The solution is single responsibility lambdas with minimal deps as the author suggests. (Using a framework like arc.codes also helps because it encourages small lambdas and bakea in everything you need.)

Collapse
 
danmactough_66 profile image
Dan MacTough

It's possible to add source map support to your lambdas. I've started a write-up that may be helpful (or may not be -- it's pretty bare-bones at the moment): dev.to/danmactough_66/add-source-m...

But we're using that setup in production and getting stack traces with correct file names and line/column numbers.

Collapse
 
zachlitz profile image
Zach • Edited

Not quite. Source maps exist for a reason.
You can debug exact source even when using a complicated package/transpile chain. You just use source maps. That's pretty standard.
Anybody who tries to debug built/compiled/packages/transpiled code is in for a world of hurt. But source maps let you debug THE EXACT source.

EDITED: also, you may need to set a Node env variable to enable source maps on Lambda. But there are plenty of instructions that a search will turn up.

Thread Thread
 
brianleroux profile image
xnoɹǝʃ uɐıɹq

Lambda does not support source maps.

Thread Thread
 
zachlitz profile image
Zach

Are you debugging a production setup? Because you can definitely setup an environment to debug using source maps. I program in TypeScript, I debug in TypeScript (using source maps). My typical build includes packaging, tree shaking, and minifying. There is no way to debug that without maps.
And in any case, you can get source mapped stack traces regardless of the environment by using the sourcemap-support package. It's very handy but less relevant these days (as source maps are supported pretty much every where now.)

Collapse
 
foresthoffman profile image
Forest Hoffman • Edited

I love to see posts on optimization and organizing dependencies, however the cover image isn't really relevant. It's a bit insensitive to the model from whichever photo this has been photoshopped (it's a good idea to provide attribution for the original photo, even if it's under a Creative Commons License).

Please review the Community Code of Conduct, and adjust the cover image accordingly.

Cheers.

Collapse
 
solegaonkar profile image
Vikas Solegaonkar • Edited

Thanks for pointing out. I have changed the image.

Collapse
 
taufik_nurrohman profile image
Taufik Nurrohman

Wait until you do npm update.

Collapse
 
solegaonkar profile image
Vikas Solegaonkar

Haha. Better deploy before that

Collapse
 
jollytoad profile image
Mark Gibson

I've had success recently using esbuild to bundle my lambda's. Previously I've tried rollup and webpack without much luck.
esbuild is both very fast and has virtually zero config, and supports TS out of the box.

Collapse
 
solegaonkar profile image
Vikas Solegaonkar

Thanks for sharing

Collapse
 
njitmann profile image
njitman • Edited

Lambda runtimes come with more than aws-sdk included, so there may be additional packages you can move to dev dependencies and enjoy even more gains without the need to minify, etc. See the following project that you can run in your AWS account to get a list of the included packages.

github.com/alestic/lambdash

Better instructions here:
alestic.com/2015/06/aws-lambda-she...

For example, in the node.js 10.x runtime, the following packages were included:
assert
async_hooks
awslambda
aws-sdk
base64-js
buffer
child_process
cluster
console
constants
crypto
dgram
dns
domain
dynamodb-doc
events
fs
http
http2
https
ieee754
imagemagick
inspector
isarray
jmespath
lodash
module
net
os
path
perf_hooks
process
punycode
querystring
readline
repl
sax
stream
string_decoder
timers
tls
trace_events
tty
url
util
uuid
v8
vm
xml2js
xmlbuilder
zlib

Collapse
 
solegaonkar profile image
Vikas Solegaonkar

Thanks for sharing

Collapse
 
lal12 profile image
Luca Adrian L

I agree to some commentors here, that bundling is the better choice. I have a project where I deploy all libs as a bundle but keep my own source as separate files. This helps for easier debugging on the fly. Though this probably is not necessary/useful when deploying a container or lambda, where a simple bundle and a source map (stored somewhere else) is the better easier choice.

Collapse
 
abhishekshetty profile image
abhishekshetty

Won't it be better to use Webpack additionally. It minifies and does treeshaking and will bring down the size to a much small size.

Collapse
 
solegaonkar profile image
Vikas Solegaonkar

Thanks for your input. I found webpack messed it up when there was non js dependency. Did you see this problem? How did you overcome that?

Collapse
 
lal12 profile image
Luca Adrian L

It is true that this can be a bit tricky, since there isn't a universal way for every kind and type of non js dependency. But there is one way or another for every one. For most cases e.g. there is a generic or specific webpack loader. And I personally rarely experienced such thing. Mostly regarding native modules for which there is a webpack loader, but might require special handling for Lambda one way or another.

Collapse
 
josiasaurel profile image
Josias Aurel

Thanks for this useful article.

Collapse
 
rolfstreefkerk profile image
Rolf Streefkerk

Is there a benchmark that shows difference between layer package size versus lambda deployment size with dependencies on cold starts