DEV Community: Dustin Goodman

Migrating a classic Express.js to Serverless Framework

Dustin Goodman — Thu, 21 Apr 2022 03:37:03 +0000

Problem

Classic Express.js applications are great for building backends. However, their deployment can be a bit tricky. There are several solutions on the market for making deployment "easier" like Heroku, AWS Elastic Beanstalk, Qovery, and Vercel. However, "easier" means special configurations or higher service costs.

In our case, we were trying to deploy an Angular frontend served through Cloudfront, and needed a separately deployed backend to manage an OAuth flow. We needed an easy to deploy solution that supported HTTPS, and could be automated via CI.

Serverless Framework

The Serverless Framework is a framework for building and deploying applications onto AWS Lambda, and it allowed us to easily migrate and deploy our Express.js server at a low cost with long-term maintainability. This was so simple that it only took us an hour to migrate our existing API, and get it deployed so we could start using it in our production environment.

Serverless Init Script

To start this process, we used the Serverless CLI to initialize a new Serverless Express.js project. This is an example of the settings we chose for our application:

$ serverless
What do you want to make? AWS - Node.js - Express API
What do you want to call this project? example-serverless-express

Downloading "aws-node-express-api" template...
Installing dependencies with "npm" in "example-serverless-express" folder
Project successfully created in example-serverless-express folder

What org do you want to add this service to? [Skip]
Do you want to deploy your project? No

Your project is ready for deployment and available in ./example-serverless-express
Run **serverless deploy** in the project directory
    Deploy your newly created service
Run **serverless info** in the project directory after deployment
    View your endpoints and services
Run **serverless invoke** and **serverless logs** in the project directory after deployment
    Invoke your functions directly and view the logs
Run **serverless** in the project directory
    Add metrics, alerts, and a log explorer, by enabling the dashboard functionality

Here's a quick explanation of our choices:

What do you want to make? This prompt offers several possible scaffolding options. In our case, the Express API was the perfect solution since that's what we were migrating.

What do you want to call this project? You should put whatever you want here. It'll name the directory and define the naming schema for the resources you deploy to AWS.

What org do you want to add this service to? This question assumes you are using the serverless.com dashboard for managing your deployments. We're choosing to use Github Actions and AWS tooling directly though, so we've opted out of this option.

Do you want to deploy your project? This will attempt to deploy your application immediately after scaffolding. If you don't have your AWS credentials configured correctly, this will use your default profile. We needed a custom profile configuration since we have several projects on different AWS accounts so we opted out of the default deploy.

Serverless Init Output

The init script from above outputs the following:

.gitignore
handler.js
package.json
README.md
serverless.yml

The key here is the serverless.yml and handler.js files that are outputted.

serverless.yml

service: example-serverless-express
frameworkVersion: '2 || 3'

provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: '20201221'

functions:
  api:
    handler: handler.handler
    events:
      - httpApi: '*'

handler.js

const serverless = require("serverless-http");
const express = require("express");
const app = express();

app.get("/", (req, res, next) => {
  return res.status(200).json({
    message: "Hello from root!",
  });
});

app.get("/hello", (req, res, next) => {
  return res.status(200).json({
    message: "Hello from path!",
  });
});

app.use((req, res, next) => {
  return res.status(404).json({
    error: "Not Found",
  });
});

module.exports.handler = serverless(app);

As you can see, this gives a standard Express server ready to just work out of the box. However, we needed to make some quality of life changes to help us migrate with confidence, and allow us to use our API locally for development.

Quality of Life Improvements

There are several things that Serverless Framework doesn't provide out of the box that we needed to help our development process. Fortunately, there are great plugins we were able to install and configure quickly.

Environment Variables

We need per-environment variables as our OAuth providers are specific per host domain. Serverless Framework supports .env files out of the box but it does require you to install the dotenv package and to turn on the useDotenv flag in the serverless.yml.

Babel/TypeScript Support

As you can see in the above handler.js file, we're getting CommonJS instead of modern JavaScript or TypeScript. To get these, you need webpack or some other bundler. serverless-webpack exists if you want full control over your ecosystem, but there is also serverless-bundle that gives you a set of reasonable defaults on webpack 4 out of the box. We opted into this option to get us started quickly.

Offline Mode

With classic Express servers, you can use a simple node script to get the server up and running to test locally. Serverless wants to be run in the AWS ecosystem making it. Lucky for us, David Hérault has built and continues to maintain serverless-offline allowing us to emulate our functions locally before we deploy.

Final Configuration

Given these changes, our serverless.yml file now looks as follows:

service: starter-dev-backend
frameworkVersion: '2 || 3'
useDotenv: true # enable .env file support

plugins: # install serverless plugins
  - serverless-bundle
  - serverless-offline

custom: # configure serverless-offline
  serverless-offline:
    httpPort: 4000

provider:
  name: aws
  profile: exampleprofile # use your own profile
  stage: production # use your specified stage
  runtime: nodejs14.x
  lambdaHashingVersion: '20201221'

functions:
  api:
    handler: handler.handler
    events:
      - httpApi: '*'

Some important things to note:

The order of serverless-bundle and serverless-offline in the plugins is critically important.
The custom port for serverless-offline can be any unused port. Keep in mind what port your frontend server is using when setting this value for local development.
We set the profile and stage in our provider configuration. This allowed us to use specify the environment settings and AWS profile credentials to use for our deployment.

With all this set, we're now ready to deploy the basic API.

Deploying the new API

Serverless deployment is very simple. We can run the following command in the project directory:

$ serverless deploy

This command will deploy the API to AWS, and create the necessary resources including the API Gateway and related Lambdas. The first deploy will take roughly 5 minutes, and each subsequent deply will only take a minute or two! In its output, you'll receive a bunch of information about the deployment, including the deployed URL that will look like:

https://<hash-string>.execute-api.<region>.amazonaws.com

You can now point your app at this API and start using it.

Next Steps

A few issues we still have to resolve but are easily fixed:

New Lambdas are not deploying with their Environment Variables, and have to be set via the AWS console. We're just missing some minor configuration in our serverless.yml.
Our deploys don't deploy on merges to main. For this though, we can just use the official Serverless Github Action. Alternatively, we could purchase a license to the Serverless Dashboard, but this option is a bit more expensive, and we're not using all of its features on this project. However, we've used this on other client projects, and it really helped us manage and monitor our deployments.

Conclusion

Given all the above steps, we were able to get our API up and running in a few minutes. And due to it being a 1-to-1 replacement for an existing Express server, we were able to port our existing implementations into this new Serverless implementation, deploy it to AWS, and start using it in just a couple of hours.

This particular architecture is a great means for bootstraping a new project, but it does come with some scaling issues for larger projects. As such, we do not recommend a pure Serverless, and Express for monolithic projects, and instead suggest utilizing some of the amazing capabilities of Serverless Framework with AWS to horizontally scale your application into smaller Lambda functions.

This Dot Labs is a development consultancy focused on providing staff augmentation, architectural guidance, and consulting to companies.

We help implement and teach modern web best practices with technologies such as React, Angular, Vue, Web Components, GraphQL, Node, and more.

Debugging Node Serverless Functions on AWS Lambda

Dustin Goodman — Thu, 21 Apr 2022 03:31:37 +0000

How many times have you written a function locally, tested it, and had it working only for it to fail when you deployed it to AWS? This is probably more common than you realize and it's usually caused by a misunderstanding of Node or an issue with lambda configuration. In this post, I'll cover some of the most common debugging problems you'll encounter when writing serverless functions and how to fix them.

Improper Use of `async/await`

When I first started writing serverless functions in Node.js, I had a misconception about how asynchronous functions behave. I was under the impression that you could run an asynchronous function as a background process and it would run on its own thread. However, this is not the case. Asynchronous functions are executed in the context of the Node.js event loop and are not run in the background. This means that if you try to run an asynchronous function in the background, it will block the event loop and the function may possibly never run. For example:

const randomBackgroundFunction = async () => {
  console.log('This function may never run');
};

export const handler = async () => {
  // do some stuff ...

  randomBackgroundFunction(); // <-- this most likely won't run without an await
  await randomBackgroundFunction(); // <-- this function will definitely run

  return goodResponse;
};

I say "may" because if no other code is running and the event loop is idle, the function will run, but once your handler returns, it's a race against the CPU clock. The AWS Lambda implementation tries to shutdown the Lambda once the response has executed or the Lambda's timeout has been reached (more to come on this topic!). So it's possible, your invocation may run before the shutdown process beings and you'll get lucky that it ran.

Now you may be asking, "Dustin, how do I run my function in the background and ensure execution?" Luckily, there are 2 great solutions: asynchronous Lambda invocations or AWS's Simple Queuing Service (SQS).

Asynchronous Lambda Invocations

AWS built Lambda to have asynchronous invocations as an out-of-the-box feature. This means you can invoke a Lambda from your primary handler and have it run on its own thread and not block your main instance. So you can rewrite our example from above like this:

// background.js
export const handler = async () => {
  // do our background stuff like we may have before
  console.log('This function will definitely run');
}

// main.js
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";

export const handler = async () => {
  // do some stuff ...
  const client = new LambdaClient(config);
  const command = new InvokeCommand({
    FunctionName: 'background',
    InvocationType: 'Event', // default here is 'RequestResponse'
  });

  await client.send(command); // this acts as a fire and forget

  return resp;
};

See the AWS SDK v3 Docs for more details about the API in use. What we're doing is utilizing the 'Event' invocation type to tell Lambda to just trigger this function and not wait for a response. From Lambda's docs:

Lambda manages the function's asynchronous event queue and attempts to retry on errors. If the function returns an error, Lambda attempts to run it two more times, with a one-minute wait between the first two attempts, and two minutes between the second and third attempts. Function errors include errors returned by the function's code and errors returned by the function's runtime, such as timeouts.

With this, we get the benefit of the event queue without having to set it up and manage it ourselves. The downside is we have to use Lambda's default retry behavior to handle errors giving us less flexibility.

AWS SQS

Similar to invoking via another Lambda, we can utilize SQS to send a message to a queue and have it run our function instead. Like with the above example, we can generate a message in an inconsequential amount of time and send it to the queue. With this, we get the benefit of configurable retry behavior, but it comes at a cost of having to manage the queue ourselves. It also means our Lambda needs to know how to flexibly read event data from the SQS stream instead of being able to parse the Payload.

Lambda Timeouts

Lambda's default timeout settings are the next major hurdle. If your Lambda needs to run for a while or process a lot of data, you might see your function just quit suddenly and not reach a later moment in your code. By default, Lambda has a 6 second timeout. If you're waiting on additional services, long running queries, or a Lambda to start cold, this could prove problematic. A quick way to check your Lambda's timeout is to load up the AWS Console and at the Lambda's general configuration at the bottom of page. In the screenshot below, you'll see the Lambda I'm inspecting has a 5 minute timeout.

Lambda timeouts can be configured in second intervals up to 15 minutes. When I use the Serverless Framwork, I typically set my Lambdas attached to API Gateway triggers to 29 seconds and SQS triggers to 15 minutes via the configuration file. I choose 29 seconds because API Gateway's maximum timeout is 30 seconds and due to latency between the API Gateway and the Lambda, AWS warns when the timeout is equal to 30 seconds as it's not truly 30 seconds. Use your deployment configurations method of choice for setting timeouts but confirm they are what you set them to be.

Other Things to Look out for

These were two of the larger problems I've faced with relatively easy fixes. The following are some smaller problems that are either easy to fix but specific to your utilization of Lambda or are things I haven't experimented with yet but am aware of:

Ensure your Lambda has access to all the resources it interfaces with. You'll need to check the IAM Role attached to your function via the console to see what permissions it has. If you're using the Serverless Framework, you can set the IAM permissions in your serverless configuration file.
Verify your environment variables are set correctly. A Lambda keeps a copy of the environment variables that it accesses and you can verify it via the AWS console. Make sure your values match what you're expecting from your configuration.
If you're doing File I/O or large data operations, make sure you're not running out of memory. If you are, look into utilizing Lambda's new emphermeral storage feature.

Conclusion

I hope you've found these tips and tricks helpful and they save you time down the road!

Announcing a Serverless Microservices Template with GraphQL

Dustin Goodman — Thu, 21 Apr 2022 03:29:14 +0000

For those that know me, they know I love talking about two things more than anything: Serverless Framework and GraphQL. Today, I'm excited to announce a project starter template that I've been developing that allows developers to build serverless microservices with GraphQL. It is built using Nx monorepos and provides a lot of quality of life developer tooling out-of-the-box. I'll discuss what's in the repo and how you can leverage it today for your own projects. If you want to jump into the code, you can find it on GitHub.

Project Goals

Over the past few years, I've developed multiple serverless projects and they all tend to follow a similar pattern especially when using GraphQL as the primary API Gateway for the entire application. As the applications have grown, the service architecture had to expand and my teams discovered a lot of difficulty maintaining the different services and how to stitch the different services together in a streamlined method. Another issue was orchestrating local development through package.json scripts would cause memory issues and some cause system memory leaks. Every team also had to perform deep function analysis to keep bundle sizes down when deploying. This project sought out to solve these key issues and provide a first-class developer experience (DX) for those wishing to build serverless microservices with GraphQL.

Managing Service Generation

In each project, we always introduced a root level directory serverless.common.yml configuration file that centralized configuration of plugins and acted as a central lookup for ports utilized by the serverless-offline plugin. Prior to Serverless v3 and TypeScript configuration files, we had to configure the ports and inject them into the application. This was troublesome for a number of reasons, but mainly, it created problems when creating new services into the stack. Developers would have to create a new service directory, scaffold the service, and configure the global settings and then when they were ready to implement it, they would have to spell their service's name correctly at invocation time. The biggest issue with this was when two services were created at the same time and cause merge conflicts for port configuration.

With this template, we're leveraging the Nx monorepo toolchain which provides us with the ability to create service generators! This project ships with a service generators that quickly solves the issues seen traditionally. You can simply run:

yarn workspace-generator service <service name>

This will:

Create a new service directory with the provided name
Scaffold the serverless configuration, test suite, linting, typescript, and example lambda function
Register the service to the Nx workspace
Update the serverless.common.ts file to include the new service name to the service type and port mappings to next available ports based on existing configuration

For example, if you run yarn workspace-generator service my-service, you'll see the following changes made for you:

- export type Service = 'public-api' | 'background-jobs' | 'example-service';
+ export type Service = 'public-api' | 'background-jobs' | 'example-service' | 'my-service';
export const PORTS: PortConfig = {
  'public-api': {
    httpPort: 3000,
    lambdaPort: 3002,
  },
  'background-jobs': {
    httpPort: 3004,
    lambdaPort: 3006,
  },
  'example-service': {
    httpPort: 3008,
    lambdaPort: 3010,
  },
+ 'my-service': {
+   httpPort: 3012,
+   lambdaPort: 3014,
+ },
};

This will give you type safety when using the provided invoke function when selecting services and identify which local port to use.

The default scaffold attempts to make some reasonable defaults for the serverless configuration. If you don't like the defaults, you can customize them by editing generator default template.

Solving Service Orchestration

Most complex backends require a background jobs worker that relies on a service like SQS or require a database to be running. Unfortunately, these are difficult to emulate services in a local environment and require custom setup. Previously, my teams would attempt to extend start scripts to init the required services. However, the node process manager wouldn't manage the processes correctly and would leave services running in the background and cause memory issues for those trying to run services.

Since a majority of services rely on the same system services, we can leverage Docker to start all the needed project services prior running services with the offline command. The template ships with a base docker-compose.yml that stands up an instance of ElasticMQ to emulate Amazon SQS's API. The Docker image can be extended to include other services like DynamoDB, Redis, or PostgresQL and can be run using yarn infrastructure:build or yarn infrastructure:start depending on if it's your first run or not.

Function Analysis

One of the most important aspects of serverless development is keeping an eye on your bundle sizes and to reduce cold start times on Lambda. Keeping this in mind, the template utilizes serverless-esbuild and serverless-analyze-bundle-plugin to provide function analysis out-of-the-box. I opted for serverless-esbuild over serverless-bundle for a few reasons:

serverless-bundle provides a lot of functionality that we don't need or use out of the box
esbuild is generally faster than webpack for bundling and requires significantly less configuration
I personally had issues with serverless-bundle in the past with function analysis which you can read more about here and have been reluctant to use it as a result. The plugin has matured since and provides the functionality I fought with but the analyzer tool they chose isn't my favorite so the benefits just aren't there.

With this project, you can run yarn analyze <service name> --function=<function name> to get an analysis of your bundle size. For example, if you run yarn analyze public-api --function=graphql, you'll see the following analysis:

Default Services

For the default template, I've shipped with a few services to demonstrate how to utilize the project structure. The default services are:

public-api: Contains the GraphQL API setup and structure
background-jobs: Contains SQS setup and Lambda runners
example-service: Contains examples for resolver functions for cross service communication and a simple SQS message generator

Other projects will have more complexity and this doesn't begin to demonstrate all the different features and functionality you can include in your project. This is solely intended to generate enough of a baseline for you to develop your own applications without having to go through all the verbose setup. If you see a feature that would be useful for others, please drop an issue or pull request on the repo!

Developer Console

I started my career building Ruby on Rails applications. One of my favorite features was the rails console which allows developers to interact with their application code directly which greatly helps with testing and debugging. As such, I wanted to recreate this for this template. You can run yarn console <library name> to get an interactive REPL that you can interface with that library's code. I only included this as part of libraries and not services due to the nature of the code structure and the lack of entry points in which you should interact with the Lambda functions in your services. Below you can see how the console can work:

What's Next?

Obviously, there is a lot here and I could keep plugging away to continually improve the template and never release but that wouldn't help anyone. I'll continue to iterate on the template and add useful features and cleanup the codebase. If you have any suggestions or want to help, the repos issues and pull requests are open!

Thank you!

I want to send a special thank you to the following people for their help in getting this project setup!

sudokar for the original Nx template in which this project was branched from. I frankly could not have solved some of the workspace generator issues I faced without this starting point.
Jesse Tomchak for talking through key issues and helping to make some important architectural decisions.
Chris Trześniewski for assisting with some critical path AST parsing issues. The serverless.common.ts automatic updates wouldn't have been possible without his help.
Tápai Balázs for convincing me to use Nx in the first place. I tried yarn workspaces and frankly, I couldn't make anything work right. Nx was the right call and his suggestion made for a great fit even if I had to modify a lot of their core features!
Nacho Vazquez and Dario Djuric for assisting with some Nx structural decisions that I was fighting and helping me come up with a better long term solution for this template.

Why you should upgrade to Serverless 3

Dustin Goodman — Sat, 19 Feb 2022 21:44:29 +0000

If you haven't read about the release and what's included in this update, I highly recommend you read both the beta release announcement and the full release announcement first as they give a good overview of all the changes. In this post, I'll highlight a few of those core changes and their significance.

Upgrading from v2 to v3

The Serverless Framework team did an amazing job of making the migration to v3 as seamless as possible without major breaking changes and kept up their habit of making things backwards compatibility. The upgrade is as easy as follows:

Upgrade to the latest version of v2. If you're on v1, you'll have a bit more work to do.
Check if you have any deprecation notices.
- If yes, you'll want to fix those but the good news is they should all be restricted to your serverless.yaml files.
- Some deprecations may be related to your plugins. The Serverless Framework team worked with many of the most popular plugins to make them backwards compatible. Go read your plugins' documentation and make the needed upgrades.
Upgrade serverless to v3. Typically you can just do this on your system with the following:

   npm i -g serverless

Now, you can update your serverless.yaml to specify the new version.

   service: <your service name here>
   plugins:
     - <your plugins here>

   frameworkVersion: '3' # This is the line to change from 2 -> 3

If you find issues when you move to v3, you can easily just flip this back to v2 so you should definitely give this a shot.

🎉 New Stage Parameters 🎉

TL;DR:

This is by far my favorite feature included with this update as it removes an old hack that many serverless developers implemented to achieve the end goal of this feature. Specifically, this feature allows you to set service configuration settings based on the current stage, or environment. This is incredibily important for larger teams implementing serverless applications as typically, you'll have a production, staging, and/or development environment that may need custom configuration.

Before, you might have setup your serverless.yaml like this:

# ...

custom:
  profile: my-profile

  stage_env:
    dev:
      PROFILE: dev
      SLS_STAGE: dev

    local:
      PROFILE: dev
      SLS_STAGE: local

provider:
  name: aws
  stage: ${opt:stage, "local"}
  region: ${opt:region, "us-east-1"}
  runtime: nodejs14.x
  profile: ${self:custom.profile}-${self:custom.stage_env.${self:provider.stage}.PROFILE, 'dev'}
# ...

Notice the really hard to grok field, profile, under provider. This was what we had to do to configure our service variables per development stage. For example, if I want the dev environment setup, I would set --stage local when I run my serverless commands and profile would be set to my-profile-dev. This is pretty hard to keep track of and lead to many long configuration debugging sessions.

Now, you can use the new stage parameters by setting their values under the params key and then acessing them via the ${param:xxx} syntax. So our above serverless.yaml can now be rewritten as:

# ...
params:
  dev:
    PROFILE: my-profile-dev
    SLS_STAGE: dev
  local:
    PROFILE: my-profile-dev
    SLS_STAGE: local

provider:
  name: aws
  stage: ${opt:stage, "local"}
  region: ${opt:region, "us-east-1"}
  runtime: nodejs14.x
  profile: ${params:PROFILE}
# ...

Improved CLI

I refer you to the full release post from the Serverless team to really see these highlights as they did a wonderful job showing you the before and after of the enhancements. I'll give my quick opinion here on these improvements:

The CLI now only informs you of what you need to know. Before, there was a little extra white noise which could be helpful under the right circumstances but was mostly unhelpful. Now, it tells you the facts you care about: (1) where was my app deployed and (2) what endpoints can I use to access it.
The --verbose flag now provides all that additional information that's now hidden when you need to debug and provides it in a much more readable format.
Errors now look like errors! 🛑 Before, errors looked like the rest of the output returned by a command so it didn't immediately capture your eye. Now, it's very obvious when a problem occurs with seemingly better messaging. I haven't had too many run ins with this experience yet as the rest of my systems have upgraded so cleanly but the few times I've seen an issue, it took me seconds to realize what the problem was instead of minutes.

If you're new to Serverless Framework, trust me when I say that the devs really thought about you with these updates. These were real problems for folks and now, they're a thing of the past. There's always room to improve so if you see something that could be better, let them know as they're clearly listening to their users. 🥰

Onboarding

One of my least favorite things with the old CLI was starting a new project. Previously, it created your new project in your current work directory so I would accidentally clutter my top-level development folder that I keep and would have to manually clean things up. Additionally, the setup only gave you the minimum files and setup needed to start so the README.md, serverless.yaml, and handler.js.

The new CLI project starter made a ton of improvements. Now, it creates the new folder for your project and initializes all your files in there. The CLI offers you a set of standard templates for common usages and guides you to their full list of amazing starter kits for new projects.

They kept with the old and help you quickly configure your project for the Serverless Dashboard and help you do your first deploy on project init but allow you to opt out as well.

Plugins

As I mentioned earlier, the Serverless team did a great job helping the community migrate popular plugins to be v3 compatible. I've never created a plugin myself, but the team has put out some great resources and new plugin API for those looking to create their own tooling for Serverless Framework. I've gone ahead and experimented with the following plugins and can say they have played really nicely with v3:

This isn't to say they've all been officially upgraded to the latest plugin API and play nice with the new CLI, but they still work so if you're in need of any of these plugins, I'd say they're safe to use but are worth your own investigation. 😉

Conclusion

Serverless Framework continues to be my favorite toolchain for building applications utilizing serverless architectures, and I'm really excited to see the developers continue to improve it in meaningful ways. The other changes I haven't covered here are some specific deprecated syntaxes that I think are uncommon. If you need more info on them though, read the upgrade guide.

I want to send a special thank you to the Serverless Framework team for their hard work and making this great tool even better! You all hit a homerun with this one! 🎉

Migrating from REST to GraphQL

Dustin Goodman — Mon, 15 Feb 2021 20:25:25 +0000

Introduction

GraphQL has been gaining a lot of traction with enterprises and startups for their application data layers. Historically, the web has been built using REST and SOAP APIs which have served their purpose successfully for years, but as applications have gotten more complicated and data has become richer, these solutions have created friction in developing performant software quickly.

In this article, we'll briefly discuss some of the problems with traditional API solutions, the benefits of migrating to GraphQL, and the strategy for migrating to a GraphQL solution.

Traditional API Problems

In traditional API systems, we typically suffer from a few common issues:

Data under-fetching or n+1 fetching
Data over-fetching
All-or-nothing Responses
Lack of batch support

Data Under-fetching

Traditional resources require us to request data on a per-entity basis, e.g. only users or only posts. For example, using REST, if we want to get some user details and their posts, we'd have to make the following requests:

GET /users/1
GET /users/1/posts

Data Over-fetching

Conversely, when we request certain data, it will give us all the available information including data we might not care about. From our previous example, we might only want a user's name and username but the response might provide us their creation time and bio.

All-or-nothing Responses

However, if there's an error somewhere in this process, we might not get any data. Instead, we receive an HTTP status code informing us of a failure with an error message but none of the data that was fetchable.

Lack of Batch Support

Finally, for our more complex page, we might need to run multiple requests that can be parallelized but traditional APIs don't support this behavior out of the box. Dashboards, for example, might need sales and marketing data which will require our clients to make two separate requests to our server and wait on results before displaying that data causing perceived slowness in our application.

The GraphQL Advantage

Out of the box, GraphQL solves all of these described issues due to its declarative querying syntax and data handling. When you fetch data, you can request the exact data you need, and using the connection among entities, you can retrieve those relationships in a single request. If any of the data fails to fetch, GraphQL will still tell you about the data that was successfully retrieved and about the failures in fetching the other data, allowing you to show your users data regardless of failures. GraphQL also allows you to group multiple operations in a single request and fetch all data from a single request, thus reducing the number of round trips to your server and increasing perceived speed of your application.

In addition to these features, GraphQL creates a single gateway for your clients, reducing friction in team communication around how data should be fetched. Your API is now abstracted away behind a single endpoint that also provides documentation on how to use it.

Given all these advantages, it's no wonder teams are moving to GraphQL, but it leaves the question of: how?

Migration Strategy

The GraphQL migration strategy is incremental so you don't have to slow down development to port over existing data or endpoints until you're ready to opt into those changes.

0. Before you begin

Before you start migration, here are some suggestions to think about as you're building new features or modifying the system in any way.

Don't build any new REST endpoints. Any new REST work is going to be additional GraphQL work later. Do yourself a favor and build it in GraphQL already.

Don't maintain your current REST endpoints. Porting REST endpoints to GraphQL is simple and GraphQL will provide you more functionality to build the exact behavior you want.

Leverage your existing REST endpoints to prototype quickly. You can use your existing REST API to power your GraphQL implementation. This won't be sustainable or performant long term, but it's a great way to get started.

1. Pick your GraphQL Implementation

Apollo and Relay are the two most popular fullstack GraphQL solutions, but you can also build your own solutions. Regardless of what you use, you'll use this to implement your server endpoint and connect to it with your client. All GraphQL requests go through a single endpoint, so once this is up and running, you can connect to it and begin porting functionality.

2. Select your first feature to build or port

With our server, we can start adding to it. Following our earlier example, let's migrate user posts.

3. Define your schema types

Now that we've decided on user posts, we have two routes here: (1) migrate users and posts or (2) migrate posts with a filter on user. For this, we're going to migrate posts and filter on user ID for now. To start, we'll define our post type in the schema and define its query type:

type Post {
  id: ID!
  userId: ID!
  content: String!
}

type Query {
  posts(userId: ID): [Post]
}

We now have a Post type that has an id and content and knows which user it belongs to. Additonally, we have a query called Posts that optionally accepts a userId as a filter and returns a list of Posts. It's important to note that it is semantically incorrect in GraphQL to expose the userId as a field. Instead, we should connect a post to its user and expose that entity relation, but those will be choices you make as you design your API.

4. Build our data resolver

Now, we need to connect our schema type and query to our data. For this, we'll use a resolver. The following syntax will vary slightly pending your server implementation, but using JavaScript and the GraphQL specification, we'd end up with the following resolver object:

const fetch = require('node-fetch');

export const resolvers = {
  Query: {
    posts: async (obj, args, context) => {
      const { API_URL } = process.env;
      const { userId } = args;

      if (userId){
        const response = await fetch (`${API_URL}/users/${userId}/posts`);
        return await response.json();
      }

      const response = await fetch (`${API_URL}/posts`);
      return await response.json();
    },
  }
};

If the userId is present in the query arguments, we use our existing REST API to fetch the posts by user, but if no userId is provided, we use the posts route directly. Now, we can make the following request on the frontend to retrieve our data:

query UserPosts($userId: ID!) {
  posts(userId: $userId) {
    id
    content
  }
}

I chose to use node-fetch for my implementation because it was simple, but you can use any HTTP library of your choice. However, if you're in the Apollo ecosystem, they've built a RESTDataSource library that will create an extension to your GraphQL implementation for handling resolvers to microservice APIs that can setup the boilerplate for that service so you only worry about fetching the data.

5. Next Steps

Extending Our Graph

Now that we have our data integrated, we need to complete the graph by connecting related types. Instead of Post having a userId, it can have a User and fetch the author details directly from the same query, e.g.

query UserPosts($userId: ID!) {
  posts(userId: $userId) {
    id
    content
    user {
      id
      avatarUrl
      displayName
    }
  }
}

Monoliths

Because we now have queries and types with full control of our schema, we can update our resolver functionality to rely on the codebase and not our REST API abstraction which will give us some added perfromance benefits. We can keep stitching together new types and extend our API further.

Microservices

GraphQL and microservices go hand-in-hand pretty well. GraphQL supports schema stitching, which allows us to build individual GraphQL APIs in our microservices and then combine them to make up our larger interface. Now, instead of configuring our clients to define all the different connections to different services, our GraphQL server understands where to collect all the data from, simplifying the amount of information the frontend needs to know about in order to complete requests.

Performance

A major downside to GraphQL can be the server-side overfetching, or n+1 problem. Because GraphQL doesn't know exactly how data is structured in the database, it cannot optimize for redundant requests in the graph tree. However, the GraphQL DataLoader library is here to solve exactly that. It determines any data that's already been fetched and caches for use in any sub-query to follow.

Conclusion

With all this power, it's no wonder GraphQL is picking up so much steam in the community. That being said, GraphQL isn't for everyone or might not be a good solution for your team today. However, I would suspect lots of future APIs we rely upon will start utilizing GraphQL more heavily and we'll see a trend away from traditional REST. Hopefully, you've seen the opportunity of GraphQL in your codebase and how it will help your team deliver quality products faster, and you can have a conversation with your team about a possible migration.

This Dot Labs is a modern web consultancy focused on helping companies realize their digital transformation efforts. For expert architectural guidance, training, or consulting in React, Angular, Vue, Web Components, GraphQL, Node, Bazel, or Polymer, visit thisdotlabs.com.

This Dot Media is focused on creating an inclusive and educational web for all. We keep you up to date with advancements in the modern web through events, podcasts, and free content. To learn, visit thisdot.co.

DEV Community: Dustin Goodman

Migrating a classic Express.js to Serverless Framework

Problem

Serverless Framework

Serverless Init Script

Serverless Init Output

Quality of Life Improvements

Environment Variables

Babel/TypeScript Support

Offline Mode

Final Configuration

Deploying the new API

Next Steps

Conclusion

Debugging Node Serverless Functions on AWS Lambda

Improper Use of async/await

Asynchronous Lambda Invocations

AWS SQS

Lambda Timeouts

Other Things to Look out for

Conclusion

Announcing a Serverless Microservices Template with GraphQL

Project Goals

Managing Service Generation

Solving Service Orchestration

Function Analysis

Default Services

Developer Console

What's Next?

Thank you!

Why you should upgrade to Serverless 3

Upgrading from v2 to v3

🎉 New Stage Parameters 🎉

Improved CLI

Onboarding

Plugins

Conclusion

Migrating from REST to GraphQL

Introduction

Traditional API Problems

Data Under-fetching

Data Over-fetching

All-or-nothing Responses

Lack of Batch Support

The GraphQL Advantage

Migration Strategy

0. Before you begin

1. Pick your GraphQL Implementation

2. Select your first feature to build or port

3. Define your schema types

4. Build our data resolver

5. Next Steps

Extending Our Graph

Monoliths

Microservices

Performance

Conclusion

Improper Use of `async/await`