DEV Community: Jordi

If TDD is Zen, adding Serverless brings Nirvana - part #2

Jordi — Mon, 23 Jul 2018 15:54:22 +0000

If TDD is Zen, adding Serverless brings Nirvana (part #2)

If you are one of the happy readers of my article about TDD + Serverless architecture, just keep reading because today we are going to improve our serverless toolkit with mongoose, connection pooling and monitoring.

If you haven’t yet, I strongly encourage you to read part #1 first and get back once you are happy, up and running.

If TDD is Zen, adding Serverless brings Nirvana

Jordi Moraleda

#serverless#aws#lambda#tdd

So if part #1 is OK for you but still need more, let’s see how we can make our DB perform fast and furious.

Database model

If we want our code to be easily maintainable, lambda functions and TDD are our friends. But when dealing with the database, we clearly can do better to document and enforce a database schema for our backend.

The mongodb NPM package is the lightest way to implement a call to our database. However, we may want to consider using an ODM like Mongoose, as the benefits we get are worth sacrificing just a bit of performance. We will be tweaking performance in a few minutes.

So, starting on the code from episode #1, let’s install Mongoose and get started:

npm uninstall mongodb
npm install mongoose

Ready! Now we can define our first model.

As a team member joining the project I would clearly go to the models folder if I checked out the code for the first time. So let’s create the folder and add a model for a user in models/user.js.

As you may notice, we are defining the three fields we used in the starter project as String and required and we are also definig an extra field created which will be populated with the result of evaluating Date.now upon creation.

Now we can use our DB model object and invoke methods on it to perform operations on the database with type validation and schema compliance. We also get a more compact and cleaner code.

For example, to find all users with mongodb we needed to do:

const dbUsers = dbClient.db(dbName).collection(dbCollection)
const result = await dbUsers.findOneAndUpdate({ _id: ObjectID(userId) }, { $set: newUser })
return result.value._id

Whereas now, the same with mongoose would look like:

const User = require("./models/user.js")
const result = await User.findByIdAndUpdate(userId, newUser)
return result._id

You see, shorter, quicker… and we get schema validation. But you may think: What about the extra overhead? Right, let’s see what happens when we query an object created as { name: "John", lastName: "Smith", email: "john@example.com" }:

Objects returned by Mongoose are not the plain JS objects we would expect. They behave like JS Proxy objects, with internal getters, setters, caches and more. This happens in order to detect whether a field was updated and whether the object needs to be synced or not. All of this magic comes at a cost.

However, mongoose allows us to enable the lean mode, skip the extra housekeeping and deal with plain JS objects. Schema validation will only be done when updating or creating documents. Queries will just pass the data. How that?

await User.findByIdAndUpdate(userId, newUser).lean()

Easy, right? So now, let’s rework our handlers and see how they look:

Now, our beloved tests:

See? We rewrote most our logic, but in just 132ms we know that we broke nothing. That’s what I meant by TDD and zen.

Connection pooling

But we still have room for improvement. As you just saw, our handler connects and disconnects from the DB server at every request. This is due to the constraints of the serverless architecture, the process starts from scratch at every request and exists once the event loop is empty.

However, there is a small serverless trick that can be used to keep our process frozen when a response is sent, instead of exiting. This would allow us to keep our context and reuse database connections that were already established.

“Sounds good, show me the money”.

Sure, in your handler, set the following property to false at the begining of the function:

exports.list = async (event, context) => {
  context.callbackWaitsForEmptyEventLoop = false

  // ...
}

Since now our function may already have DB connections available, we need to encapsulate our connection logic and ensure connectivity across executions:

This is fine, but we need to do an extra step with our DB models. They need to be attached to the currently active mongoose connection, so the previous code will not work anymore.

Let’s change models/user.js and return the schema instead of registering it to the global mongoose context.

Then, we need a last step in our ensureDbConnected function:

Now in our handlers we just need to replace the connection logic by a call to ensureDbConnected and remove the calls to disconnect. So our handlers now look like this:

Every new developer on your team will have little to no problem in mastering such code.

The gotcha

However, now there is a small issue.

If you nice, now the Node process does not just exit when tests are complete. We told serverless to freeze our context instead of letting the event loop be empty, so now the process will not exit because of that.

This is okay if testing and deployment are performed by the project maintainer, but definitely becomes an issue if the testsuite is part of a pipeline that performs CI/CD tasks.

There may well be better approaches, but the solution I’ve found to work in our projects is adding hooks to every test suite and force the process to exit when they are done. Create the file test/test-completion-hooks.js like this:

We need to register them in every test suite (by now, just test/user.spec.js). The result change is very simple:

There are two lines to include in every test suite, but usingaddCompletionHooks is the solution that I’ve found to work best. If you have a better approach to get around this limitation, I’ll be more than glad to read it from you on the comments :)

So we have DB models, we keep connections open and our testing is working. What next?

Webpack

Let’s optimize the code we are shipping. It is no secret that node_modules can easily contain hunderds of megabytes of data that will be never used. However, serverless will package the main folder of your project, except for development dependencies in node_modules.

Even if we only wrote 455 lines of code at the moment, running serverless package results in a 2.3Mb file, which expands into 11Mb of uncompressed data.

We can surely do better by making use of webpack’s tree shaking capabilities. Let’s install the package:

npm install -D serverless-webpack webpack

At the time of writing, webpack version 4 is supported.

Now we need to declare the plugin in serverless.yml > plugins above the declaration of serverless-offline :

# ...

plugins:
  - serverless-webpack
  - serverless-offline
  - serverless-mocha-plugin

Next, we need a config file for webpack, so we will create webpack.config.js :

That’s all! Now we just need to npm run deploy again and now everything will be bundled into a single JS file.

Comparison

After all these changes, how well are we doing? Let’s use ApacheBench to see if we made any difference.

Before we made any change, this was the result of making 1000 requests to the listUser function:

$ ab -k -c 50 -n 1000 [https://hkyod1qhuh.execute-api.eu-central-1.amazonaws.com/staging/users](https://hkyod1qhuh.execute-api.eu-central-1.amazonaws.com/staging/users)

This is ApacheBench, Version 2.3 <$Revision: 1807734 $>

[...]

Concurrency Level:      50
Time taken for tests:   39.649 seconds
Complete requests:      1000
Failed requests:        423
   (Connect: 0, Receive: 0, Length: 423, Exceptions: 0)
Non-2xx responses:      423
Keep-Alive requests:    1000
Total transferred:      493011 bytes
HTML transferred:       28317 bytes
Requests per second:    25.22 [#/sec] (mean)
Time per request:       1982.445 [ms] (mean)
Time per request:       39.649 [ms] (mean, across all concurrent requests)
Transfer rate:          12.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   23  98.7      0     466
Processing:    83 1753 2052.4   1001    7927
Waiting:       83 1753 2052.4   1001    7927
Total:         83 1775 2065.3   1001    7927

Percentage of the requests served within a certain time (ms)
  50%   1001
  [...]
  99%   6996
 100%   7927 (longest request)

The benckmark is made upon a starter plan of MongoDB Atlas, which has a limit of 100 simultaneous connections. Even if we set a concurrency limit of 50, we see that 423 requests are still failing because they try to open a connection that surpasses this limit.

Now let’s run the same with the improvements we made in this article and see how it goes:

$ ab -k -c 50 -n 1000 [https://hkyod1qhuh.execute-api.eu-central-1.amazonaws.com/staging/users](https://hkyod1qhuh.execute-api.eu-central-1.amazonaws.com/staging/users)

This is ApacheBench, Version 2.3 <$Revision: 1807734 $>

[...]

Concurrency Level:      50
Time taken for tests:   10.087 seconds
Complete requests:      1000
Failed requests:        0
Keep-Alive requests:    1000
Total transferred:      465000 bytes
HTML transferred:       2000 bytes
Requests per second:    99.14 [#/sec] (mean)
Time per request:       504.335 [ms] (mean)
Time per request:       10.087 [ms] (mean, across all concurrent requests)
Transfer rate:          45.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   19  85.2      0     468
Processing:    61  484 406.7    370    1079
Waiting:       61  484 406.7    370    1079
Total:         61  503 399.9    487    1079

Percentage of the requests served within a certain time (ms)
  50%    487
  [...]
  99%   1010
 100%   1079 (longest request)

Simple and clear, the actual throughput is 4x. And the number of failed requests (because of no DB connection) drops to zero.

Note: There is no remarkable performance improvement in adding webpack alone. Keep it if you like to store lighter files on S3, but you can also disable the plugin and achieve faster response/rebuild times while developing.

Needless to say: before jumping into production, do your own distributed benckmarks with a scalable DB hosting plan and setting a and much higher number of connections.

Monitoring

Monitoring lambda functions is a subject that could well cover another article by itself. However, there are some tools that we can take advantage of, out of the box.

Let’s have a look at the lambda we have been benchmarking: listUsers. Open AWS Lambda, click the function on the list and select the “Monitoring” tab.

You’ll be presented with a few charts giving you an overview of the function’s status. To check the status of the DB cluster, we can also refer to MongoDB Atlas, open our cluster and open the metrics section.

If we got into a scenario in which we would like to diagnose a problem, we can get back to the Lambda function console and click on the “View logs in CloudWatch” button.

Let’s click on the latest log stream:

Now we can explore execution times, billed times, memory usage, etc.

If you need to investigate the internals of a function’s execution, you can make use of console.log, console.err, etc. Let’s add some logging on our lambda function:

console.log("> I AM AN EVENT PRINTED BY console.log()")

console.error("> AND I AM AN ERROR REPORTED BY console.error()", new Error("Error message"))

If we deploy our code and open the latest log stream entry, we will be able to see them:

There is room to go deeper in this subject. Depending on your needs, you can have a look at serverless-plugin-tracing , along with AWS X-Ray, or check Dashbird.

Wrap up

If you want to experiment with the code of the article, I have updated the repo in a dedicated branch: https://github.com/ledfusion/serverless-tdd-starter/tree/part-2

This concludes part #2 of my article on how to achieve nirvana by using TDD, along with Serverless. I hope that you found it useful, and don’t forget to clap, comment, share and smile if you want more and more :)

BTW: I am available to help you grow your projects.
Feel free to find me on https://jordi-moraleda.github.io/

Thanks for your read.

Cover photo by Marion Michele on Unsplash

If TDD is Zen, adding Serverless brings Nirvana

Jordi — Fri, 20 Jul 2018 10:21:51 +0000

If TDD is Zen, adding Serverless brings Nirvana

Today we are going to build the dream backend for your startup with TDD and Serverless.

Traditional API server have come a long way, but today’s fast moving projects need to kindly consider serverless, the sooner, the better.

Being requirement #1 to ship as soon as possible, the common side effect is that the codebase and infrastructure become harder to maintain once the product and the team grow.

Serverless architectures help mitigating this situation in many ways:

Lambda functions encourage to write granular, clean and specific API operations.
They also force to decouple code from the local architecture (url, ports, etc).
Combined with TDD, you can develop faster and leaner.
No server means no sysadmin. More time to spend on your project.

In short, what we are about to explore is:

A clean serverless API project
Ready for Test Driven Development
Connecting to a cloud database (MongoDB Atlas)
Using secret management
With automated deployment
Using staging environments

So let’s get right to it!

Start me up!

First, make sure that you have NodeJS on your system and install the Serverless framework:

npm i -g serverless       (Windows)
sudo npm i -g serverless  (Linux and MacOS)

Cloud accounts

Create an account on Amazon Web Services and open the IAM Management Console when you are done. You need to add a new user.

Give it a meaningful name and enable “Programmatic access” for it.

By now, we are are just developing, so we will attach the AdministratorAccess policy and keep working on the project.

IMPORTANT: When the project is ready for production, get back to the lambda IAM user and check this article on how to apply the Least Privilege Principle.

Our AWS user is now ready. Copy the keys shown on the screen and run in the console:

serverless config credentials --provider aws --key <the-access-key-id> --secret <the-secret-access-key>

Done! Your serverless environment is ready to connect to Amazon and do its magic. Now let’s a database to connect to.

The choice of a particular database is out of the scope of this article. Because MongoDB is one of the most popular NoSQL databases, we will open an account at MongoDB Atlas.

Select a provider and a region that suit your needs. Since our code will be running on AWS Lambda, it makes sense to select AWS as the provider, and the same region where we will deploy our lambdas.

Check any additional settings and choose a name for your cluster. Wait for a few minutes for it to provision.

When it is ready, open the Security tab of the cluster and add a new user.

Since we are just testing, enter a username/password of your choice and select “Read and write to any database”.

IMPORTANT: When you are ready for production, you should apply again the Least Privilege Principle and restrict privileges to just your app’s database. You should also create different users for production and staging environments.

Keep the user and password and get back to the cluster overview. Click on the “Connect” button. Next we need to whitelist the IP addresses allowed to connect to the database. Unfortunately, on AWS Lambda we have no predictable way to determine the IP address that will connect to MongoDB Atlas. So the only choice is to go for “Allow access from anywhere”.

Finally, click on “Connect your application”, choose version 3.6 and copy the URL string for later.

Let’s code!

Enough accounts, now let’s get our hands dirty. Open the console and create a NodeJS project on folder my-api:

serverless create --template aws-nodejs --path my-api
cd my-api

Let’s invoke the default function:

serverless invoke local -f hello

(output)
{
    "statusCode": 200,
    "body": "{\"message\":\"Go Serverless v1.0! Your function executed successfully!\",\"input\":\"\"}"
}

Ok, running. Let’s create the package.json and add a few dependencies:

npm init -y
npm i mongodb
npm i -D serverless-offline serverless-mocha-plugin

Next, let’s define our environment. The serverless CLI just created the file serverless.yml for us. Clean it up and edit serverless.yml like this:

A few things to note here:

We are using NodeJS 8.10 in order to get the modern Javascript goodies.
We have defined the same region that we previously chose on MongoDB Atlas.
A hello function is added by default in handlers.js
We are adding two plugins at the bottom (from the dependencies we installed before). In a Daft Punk fashion, they will help us develop better, faster, stronger.

Now we can start listening for incoming HTTP requests on our local computer:

serverless offline start

So if we open our browser and visit http://localhost:3000, we will get the output of the hello function, which is attached to the / path by default. The output is a message, in addition to the HTTP headers and parameters that the function gets.

The cool thing is that if we edit handlers.js so that hello returns something different, you’ll notice that refreshing the browser will show the updated content. Live reload API out of the box!

Before we jump into our TDD environment, let’s tidy up a bit our project and arrange our files and routes.

mkdir handlers test
rm handler.js
touch handlers/users.js

The routes we will be supporting are:

GET /users
GET /users/<id>
POST /users
PUT /users/<id>
DELETE /users/<id>

So let’s edit serverless.yml and define them. Edit the functions: block to contain these lines:

As you can see, each key inside the functions block is the name of a Lambda function. Note that handler: handlers/users.list would translate into:
Use the list JS function inside handlers/user.js.

Time to TDD!

The serverless CLI provides a command to add new tests for each function. Let’s check it:

serverless create test -f listUsers
serverless create test -f getUser
serverless create test -f addUser
serverless create test -f updateUser
serverless create test -f removeUser

The test folder now contains a dummy spec file for each of our Lambda functions. As all our user-related functions are pointing to handlers/users.js maybe it’s better that the specs keep the same structure.

So let’s discard the isolated specs rm test/* , combine them into a single file and code the full user spec in test/users.spec.js.

As you can see, instead of importing just one function wrapper, we import them all and define test cases for the whole set.

You may also note that we are not performing HTTP requests. Rather we have to pass the parameters as they would be received by the function. If you are interested to spec like an HTTP client, check this library.

What happens now if we run the test suite?

serverless invoke test

You guessed it! Booom. And that’s because… we haven’t coded yet the handlers! But as you know: in TDD, specs are coded before the application logic.

So, as our runtime is NodeJS 8, we can take advantage of ES6/ES7 features to write cleaner code in our lambda functions. Let’s implement the (still undefined) functions in handlers/users.js:

Note that unlike a traditional NodeJS server, the connection to the database needs to be opened and closed at each execution. This is due to the nature of how serverless works: there is no process running all the time. Instead, instances are created and destroyed when external events occur.

Also note that all routes make sure to always close the DB connection. Otherwise, the internal NodeJS event loop would keep the process alive until the timeout was reached, and you might incur in higher charges.

And finally, note that thanks to using NodeJS v8, we can return our response instead of using callbacks with error-first parameter.

So now, our specs are ready, our implementation is there. The moment of truth:

Yay! Here is our first serverless API. As you may see, local execution times are not particularly impressive, but keep the following in mind:

Our goal is not about single request speed. Rather, we aim for concurrent massive scalability, easy maintainability and future-proof code.
Latencies are mainly due to the time spent by our local computer connecting to the remote database. Tests with 3~4 DB requests will experience much higher latencies than when code is running within the datacenter.
We are using the lightest implementation possible (mongodb instead of mongoose, and plain JS instead of Express/Connect on top of Serverless).
If you switch to a local MongoDB server, running the tests will take around 80ms.

We’ll check performance back when our code is deployed.

Secret management

We are almost ready to deploy, but before we need to deal with an important aspect: keeping credentials out of the codebase.

Luckily for us, Serverless suports Simple Systems Manager (SSM) since version 1.22. This means that we can store key/value data to our IAM user and have it automatically retrieved whenever Serverless needs to resolve a secret.

So, first of all let’s get back to our handlers/users.js file, copy the current URL string and replace:

const uri = "mongodb+srv://lambda:lambda@myapp...."

with:

const uri = process.env.MONGODB_URL

Next, let’s add an environment block to provider in serverless.yml :

provider:
  name: aws
  runtime: nodejs8.10
  stage: prod
  region: eu-central-1
  environment: 
    MONGODB_URL: ${ssm:MY_API_MONGODB_URL~true}

This will bind the MONGODB_URL environment variable to the MY_API_MONGODB_URL SSM key at deploy time and ~true will decrypt the contents with the default IAM user's key.

Finally, let’s grab the string we just copied and store the credential in our SSM:

pip install awscli    # install the AWS CLI if necessary
aws configure         # confirm the key/secret, define your region

aws ssm put-parameter --name MY_API_MONGODB_URL --type SecureString --value "mongodb+srv://lambda:xxxxxx@myapp-.....mongodb.net/my-app?retryWrites=true"

If you are part of a bigger team, read here.

Deploying

Stay with me, we are almost done! Let’s deploy our code to the cloud. We will update serverless.yml before. Then:

serverless deploy

Ready! You can also manage them here (select the appropriate region).

If you call the URL corresponding to the listUsers function you will see that the latency takes under one third of what it was taking from out computer.

This happens because now, the lambda function and the DB server are in the same region (i.e. datacenter), so connection latencies between them are considerably lower. Our round trip to Amazon will always be there, but now DB connections will not.

Production

As already commented during the article, when your API is ready to ship:

Remove the administrative permissions from your IAM user and refer to this page for insights on how to grant fine grained privileges.
Restrict the privileges of the DB user to only the database of your application, and not just any.

Deployments should only be made by project maintainers. The rest of developers shouldn’t need to configure any IAM Lambda credentials.

Cleaning

Deploying a Lambda function will involve (at least) three different services from Amazon. If you intend to wipe an existing Lambda function you need to:

Delete the function from AWS Lambda
Delete the corresponding bucket from S3
Delete the corresponding stack from CloudFormation

Wrap up

I hope you enjoyed reading the article as much as I enjoyed writing it. If you want more, heart, unicorn, comment, share and smile 🙂!

If you want to experiment with the code of the article, feel free to clone the starter repo from GitHub: https://github.com/ledfusion/serverless-tdd-starter/tree/part-1
BTW: I am available to help you grow your projects.
Feel free to find me on https://jordi-moraleda.github.io/

Bonus track #1: Staging

If you are like most of us, you will need at least 3 environments for your project: development, staging and production.

For the database, developers could use a local instance of MongoDB to speed connections up, but for staging and production, we need to provide completely independent database environments.

Head back to MongoDB Atlas and create two different user accounts (my-app-prod and my-app-staging ) with access to two different databases (my-app-prod and my-app-staging respectively).

Let’s remove the key we created before:

aws ssm delete-parameter --name MY_API_MONGODB_URL

And create two, for production and staging:

# PROD
aws ssm put-parameter --name MY_API_MONGODB_URL_prod --type SecureString --value "mongodb+srv://my-app-prod:xxxxxx@myapp-.....mongodb.net/my-app-prod?retryWrites=true"

# STAGING
aws ssm put-parameter --name MY_API_MONGODB_URL_staging --type SecureString --value "mongodb+srv://my-app-staging:xxxxxx@myapp-.....mongodb.net/my-app-staging?retryWrites=true"

Here they are:

Now let’s edit serverless.yml > provider and let’s do some magic:

First of all, the stage field tells Serverless where to deploy the lambdas. If you deploy like below:

serverless deploy --stage prod

Then, provider.stage would be "prod" . If we just ran serverless deploy then provider.stage would default to "dev".

Second, the value of MONGODB_URL is evaluated in 2 steps, depending on the environment. If we are on "prod" or "staging", Serverless will fetch MY_API_MONGODB_URL_prod or MY_API_MONGODB_URL_staging respectively from SSM and use the value. If our IAM user does not have that key, MONGODB_URL will default to "mongodb://localhost:27017".

Ta da! This allows our development team to code and test from their local database, while code running on AWS will get the remote URL connection string.

Here they are, so staging and prod are ready to go!

Bonus track #2: Automated tasks

As the title suggests, we are aiming for nirvana, not just zen. Having to type repetitive commands may be a bit of overhead, so let’s finish our show with a clean set of commands to work with the project.

As a recap, the actions we may perform during the lifecycle of the project are:

Run the app locally
Run the test suite
Run the test suite and deploy to staging if successful
Run the test suite and deploy to production if successful

So let’s define these actions on our package.json :

...
   "scripts": {
    "deploy": "npm test && sls deploy --stage staging",
    "deploy:prod": "npm test && sls deploy --stage prod",
    "start": "serverless offline start",
    "test": "serverless invoke test"
  },

Now the team can run the API with npm start and test with npm test, while the project maintainer can deploy to staging with npm run deploy and to production with npm run deploy:prod. Nobody will interfere with unintended settings, data or environments.

After this whole exercise of integrations, you should have an easy to code, test, upgrade and maintain backend. It should work well with agile teams and fast moving companies and I hope that it does for you!

Thanks for your read.

If you liked it, do not miss the part #2 of this article.

This post was originally posted on medium.com.

Photo by Marion Michele on Unsplash