Make your lambda functions lightweight

I see a lot of articles on how to create a lambda with simple steps, always showing the hello world. But every time you start using dependencies in your lambda function, it starts to weigh.

So, this article will show you how to make your lambda more lightweight and have only the function that we need to execute

Before jumping into the code you need these requirements:

1) AWS CLI installed
2) Configure your aws credentials locally with aws configure.
3) Obviously node installed (We are going to use node v12)
4) Serverless installed npm install -g serverless

Do you have it all configured? Alright! Let's start coding!! 🙂

First, I wanna show you the problem

Create a folder or create a repo and clone it. In my case, I created a repo and clone it.

I will use npm to install the dependencies, you could use yarn if you want to

> npm init -y

This will create a package.json

Now we are going to create our lambda function with

> sls init aws-node-rest-api

We are going to make a few changes to the project.

The aws-node-rest-api will change it to src and copy serveless.yml from src in our root folder

> mv aws-node-rest-api src
> mv src/serveless.yml ./serverless.yml

And the last thing to change will be the path of where is our lambda function

Before

app: aws-node-rest-api
service: aws-node-rest-api

provider:
  name: aws
  runtime: nodejs12.x

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: /
          method: get

After

app: aws-node-rest-api
service: aws-node-rest-api

provider:
  name: aws
  runtime: nodejs12.x

functions:
  hello:
    **handler: src/handler.hello**
    events:
      - http:
          path: /
          method: get

I highlighted the changed path of the lambda function.

And that's it. Let's deploy our function!!

> serverless deploy

After a while, you'll get the following picture

And if you go to aws you can see your new lambda function!! YEY!!! GOOD JOB!!!

And if go to see what is deployed in our lambda function we can see the code

Now that we are experts in serverless and lambda functions we want to add some packages

Our lambdas won't we simple right? most of the time we are using packages, to do some calls to the database, call an aws feature, call an api, manipulate an image, etc.

Now, let's install some packages. Let's say we are going to manipulate some images, in our case we are going to use jimp (this is only for the example. I needed a heavy npm package)

So in our console let's type the following command

> npm install --save jimp

And now let's deploy again a see what's going on

> severless deploy

Wait! What? Cannot see the code? What's going on?

Well, with the simple configuration we are uploading the node_modules folder into our lambda function and the package that we just installed makes the lambda too large to show the code.

How can avoid this and see my code again!!? Lambda Layers to the rescue!!

That's right! serverless has the ability to create Lambda Layers. A Lambda Layer is a ZIP archive that contains libraries or other dependencies. With that, we can make our lambda function smaller again.

How we can achieve this? We are going to put our node_modules folder in a Lambda Layer.

For this, we are going to make a few changes to our code.

First, we are going to install this package

> npm i --save-dev serverless-hooks-plugin

and after that, we are creating a deployment folder and create a script where it has all the things that we need to create the layer.

> mkdir deployment
> touch deployment/prepare.sh

In our prepare.sh we are going to copy the following code

echo '****** Starting Pre Deploy Script ******'
echo '1- Creating folder for layers and copy package.json' 
rm -rf ./.dist
rm -rf ./.serverless-layers
mkdir -p .serverless-layers/node-layers/nodejs
cp package.json .serverless-layers/node-layers/nodejs/
echo 'DONE!' 

echo '2 - Change path to serverless-layer, adding LIB dependency, remove npm and yarn files'
cd .serverless-layers/node-layers/nodejs
npm i --production
rm package.json
rm package-lock.json
cd ../../..
echo 'DONE!' 

echo '****** Finished Pre Deploy Script ******'

Basically we are creating a nodejs folder inside .serveless-layes/node-layers, copying the package.json from our root folder and install all the dependencies.

Then, in our package.json we are adding a new script

"deploy:prepare": "sh deployment/prepare.sh"

Leaving our package.json something like this.

{
  "name": "serverless-aws-node-layer-example",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "deploy:prepare": "sh deployment/prepare.sh",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/cesmunoz/serverless-aws-node-layer-example.git"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/cesmunoz/serverless-aws-node-layer-example/issues"
  },
  "homepage": "https://github.com/cesmunoz/serverless-aws-node-layer-example#readme",
  "dependencies": {
    "jimp": "^0.16.1"
  },
  "devDependencies": {
    "serverless-hooks-plugin": "^1.1.0"
  }
}

And the last thing, we need to make this steps in our serveless.yml.

Adding the following things:

1) Using the custom hook that the package serverless-hooks-plugin provides us

plugins:
  - serverless-hooks-plugin

custom:
  hooks:
    before:package:createDeploymentArtifacts:
      - npm run deploy:prepare

2) Creating the layer

layers:
  nodeModules:
    path: ./.serverless-layers/node-layers
    name: My-App-Node-Dependencies
    description: Node Modules for My App
    compatibleRuntimes:
      - nodejs12.x
    package:
      include:
        - ./**

3) Make our function package individually and exclude everything

package:
  individually: true
  exclude:
    - ./**

4) Include only our handler.js in the lambda function and make use of the lambda layer

functions:
  hello:
    handler: src/handler.hello
    layers:
      - { Ref: NodeModulesLambdaLayer }
    package:
      include:
        - src/handler.js        
    events:
      - http:
          path: /
          method: get

The final serveless.yml will be something like this:

app: aws-node-rest-api
service: aws-node-rest-api

provider:
  name: aws
  runtime: nodejs12.x

plugins:
  - serverless-hooks-plugin

custom:
  hooks:
    before:package:createDeploymentArtifacts:
      - npm run deploy:prepare

layers:
  nodeModules:
    path: ./.serverless-layers/node-layers
    name: My-App-Node-Dependencies
    description: Node Modules for My App
    compatibleRuntimes:
      - nodejs12.x
    package:
      include:
        - ./**

package:
  individually: true
  exclude:
    - ./**

functions:
  hello:
    handler: src/handler.hello
    layers:
      - { Ref: NodeModulesLambdaLayer }
    package:
      include:
        - src/handler.js        
    events:
      - http:
          path: /
          method: get

Let's deploy it again and see what happens

> serverless deploy

Woala! We can see our code again!

And Where is our lambda layer?

We can see and the lambda function has a dependency on our new lambda layer

And if we go to lambda layers we can see it's there

So with that configuration we can view always our code.

Hope you found it useful as I do.

Repo: https://github.com/cesmunoz/serverless-aws-node-layer-example

See you next time!!

C.

Comments

[Ross Coundon]

Beware of layers they can ultimately complicate your deployment, particularly if you have many lambdas dependent on a layer because any updates you make to that layer is not picked up by your lambdas unless you update each one to point to the new version and redeploy.
They also make testing your functions more difficult, harder to invoke when running locally (although this should be avoided anyway - use something like serverless stack to deploy to AWS but debug/run code locally - a game changer for us)
A good summary of pros and cons here - lumigo.io/blog/lambda-layers-when-...

[Cesar Muñoz]

I use npm packages in the layers (if I’m facing the situation on not viewing the code on prod). The handler and the shared code will be in the lambda. Also depends on how you structure the application. I faced several ways of structure the project .
I like the article I will added it into my bookmarks
I’m only make the layer on deployment time so locally with serverless offline and other plug-in (for example serverless Dynamo) works just fine.

[Vishnu Prassad]

We have tried lambda layers to manage node_modules eariler, and over the time it became very hard to manage it. Also every lamda which uses the same layer have to unpack the modules which are actually not needed for it. Layer can help with reducing size during deployment, but if the total size of lamdba code + layers cannot exceed 250mb.

Webpack does the job very neatly to handle the node_modules and helps to reduce the size of the functions by keeping only the required node_modules and files needed by it.

I have found layers useful when you have to run thirdparty binaries/modules which are not availbe via NPM, such as FFMPEG/SHARP etc.

[Pravin Kumar Damodaran]

How do layers make lambda lightweight? Does it cause any improvement in lambda performance, other than making the code visible in the lambda console?

[Cesar Muñoz]

Hi Pravin, it will be the same performance
The goal here is to make your lambda more consistent, to only have the code that needs to be run (everything else must be excluded)
Assume we have 100 lambdas and all of them need a dependency to MySQL.
Each function will have the node_modules with MySQL on it.
If you're deploying each artifact will package the node_modules, it will take so much time to deploy everything and upload to s3.
In the way I explain in the article, it will create a zip file with the modules and that's it.
All the functions will only have the code (and it will weigh nothing), so it will be fast in deployment time, and with configuration saying this lambda needs this layer