How to create your own AWS Lambda Middleware framework in Node.js?

#awslambda #designpatterns #middlewarepattern #node

AWS Lambda Middlewares Framework

We have around 25 lambdas and they share some common functionality like:

Integrating with Sentry
Removing unnecessary events
Logging information about the event being processed
Datadog tracing etc.

We needed a way to define these functionalities once and reuse them across lambdas. This post is about a middleware framework we wrote in Node.js to do this with very little code and no external libraries.

Middleware Pattern

We could have extracted each of these functionalities into a function and called it in the lambdas which would have allowed us to reuse the code. But we thought if we can bring in some convention, integrating/maintaining these common functionalities into lambdas would be much easier.

Having working with a lot of web frameworks which have concepts like filters in java, middlewares in Express, we felt a similar approach would work for us as well.

The idea is to implement a lambda handler function as a core function and a set of middleware functions.

The Core function will do what the lambda is supposed to do like transforming the incoming data, writing the events to a data store etc.
Each middleware will handle one functionality like integrating Sentry or logging etc.

This is the order in which the middlewares are called:

MiddleWare1 -> MiddleWare2 -> MiddleWareN-> CoreFunction

Middlewares have the same signature as the lambda handler function with an additional argument of the next middleware. In case of the last middleware, the next argument will be the core function. If the core function returns some value, middlewares typically return the same value.

And each middleware function can choose when to call the next middleware. This way we can divide a middleware function into three sections:

Before Section — This code is executed before the core function. We can do things like filtering events, adding more data to context etc.
Next Middleware Section — This could be the actual call to the core function. And the middleware has a choice to not call the core function at all and finish the lambda execution.
After Section — This code is executed after the core function. We can do things like error handling, logging or returning a different response etc.

Promises instead of Callbacks

Most of our lambda function code create or work with promises since they mostly query/write to external systems like Dynamodb, Elasticsearch etc. We thought it would be easier if our core function and middlewares work with promises instead of callbacks.

A typical AWS lambda definition in Node.js v6.10 would look like this:

exports._myHandler_ = function(event, context, callback){
   // Do something
   // callback(null, "some success message");
   // or
   // callback("some error type");  
}

And this is how we wanted our handler functions to look like:

const handlerFunction = (event, context) => {
  return new Promise()
}

Note: Node.js v8.10 supports async handlers which was not available when we wrote this middleware framework. This step might be redundant in Node.js v8.10.

Middleware Orchestrator

Because we have a different signature than what lambda handler is expected to have, we created a function withMiddlewares.

It takes the core function and an array of middlewares as input and returns a function which has the same signature as the lambda handler.

export._myHandler_ = withMiddlewares(
  handlerFunction, 
  [Middleware1(), Middleware2()]
)

And this is the implementation of withMiddlewares:

Line 1: It has the same signature as the lambda middleware.

Lines 2–14: chainMiddlewares returns a function which recursively calls each middleware and finally calls the handler. If there is an error thrown when calling the middleware, it will return a rejected promise.

Lines 16–21: We call chainMiddlewares with all the middlewares and convert the resulting promise into a callback function.

Middleware Template

const Middleware = () => {
  return (event, context, next) => {
    // Before Logic
    return next(event, context)
      .then(result => {
        // After Logic
        return result
      })
      .catch(error => {
        // Error Handling
        return Promise.reject(error)
      })
  }
}

Example 1: Sentry Middleware

Integrating Sentry typically involves:

Before Section — Initialising raven library
After Section — Reporting errors to Sentry

This is the trimmed down version of how this middleware definition would look like:

captureExceptionAndReturnPromisewill wait for Sentry request to complete since it is returned as promise.

If we are not able to send the request to Sentry for various reasons like Sentry is down or network error, we currently bubble up the error. But we can also log the error and return the original error.

Example 2: Dynamodb Insert/Modify Events Filter

Some of our lambdas want to ignore Dynamodb delete events and execute only on the insert/modify events.

Before Section — Remove delete events from event.Records
After Section — No after actions

We are modifying the event.Records here. Another option is to clone the event with filtered records if we want immutability.

Example 3: Lambda Handler

This is how a lambda handler would look like using the above defined middlewares. The order of the middlewares is important. SentryMiddleware should be the first middleware to catch errors in downstream middlewares too.