DEV Community: Diogo Souza

Server-side Rendering in JavaScript: A Modern Approach

Diogo Souza — Wed, 23 Dec 2020 10:47:03 +0000

Let's talk about SPAs. It all starts from a blank page which is subsequently filled with HTML and JavaScript.

If we take PHP pages as an example, they already come bundled with the server, which is an advantage in terms of performance, right?

For situations like these, server-side rendering frameworks (such as Next.js) come to the rescue. They process the code on the server-side to pre-fill the HTML result page with something (if not the whole page) before it reaches the browser.

But is that all? Are there any other options, different paradigms or approaches to deal with this?!

In this article, we're going to explore a few alternatives brewing in the community regarding server-side rendering.

Do You Know What JAMStack Is?

Jamstack is a public effort to design an architecture that makes the web faster and scalable in terms of tools and workflows that us developers use today.

It's built on top of some core principles that include:

Pre-rendering: in order to become a Jamstack-compliant developer, you'll need to dominate pre-rendering tools such as Gatsby and Next.js, and deliver your websites through prebuilt static pages.
Decoupling: a famous concept that requires services and components to be clearly separated within your apps, reducing complexity and enhancing component independence.

You can read more on the movement here. Some of the things we'll discuss below are Jamstack-related, so give it a read if possible.

What if the Clients Stop Data Fetching by Default?

What do you mean? By default, most front-end frameworks today preach a complete separation between the front-end code and the back-end API that provides the endpoints that feed the client pages.

What if we take a step back and let the server deal with data fetching by allowing it to generate client interfaces (GraphQL-based, for example) that handle everything 一 from routing to ORM management.

Let's see an example with RedwoodJS as the framework of choice. Redwood is an opinionated, full-stack, serverless web framework that easily allows the development of JAMstack apps.

How Does It Work?

Rather than splitting the front and back-end sides of the application, Redwood aims to connect them through predefined GraphQL standards. Its goal is to be the full-stack framework you'd pick to create your SPAs. Take a look at the following graph:

How Redwood works. Source: https://redwoodjs.com/

As you can see, both front and back-end worlds coexist within the same code repo. As we used to (and still) do with frameworks like Rails, .NET, etc. Yet, React is the front-end library of choice for the client-side.

Redwood divides itself into two main containers:

/web: which contains the front-end stuff such as components, cells, forms, CSS, etc.
/api: which contains the back-end API (built with GraphQL by default), as well as other optional services and lambdas.

The Main Parts

To achieve that, Redwood makes use of a bunch of features at its core. Like most of the frameworks, it comes with a custom routing system very similar to React Router, to take one example.

However, one of the most important parts refers to the concept of Cells. Redwood cells work as a scaffolded component that embraces the most common phases of an ordinary React component, such as retrieving data from the server, showing/hiding a loading placeholder, dealing with errors and success messages, and displaying the results in a proper listing component.

Take a look at the following cell example extracted from the official docs:

export const QUERY = gql`
  query USERS {
    users {
      id
      name
    }
  }
`
export const Loading = () => <div>Loading users...</div>
export const Empty = () => <div>No users yet!</div>
export const Failure = ({ message }) => <div>Error: {message}</div>
export const Success = ({ users }) => {
  return (
    <ul>
      { users.map(user => (
        <li>{user.id} | {user.name}</li>
      ))}
    </ul>
  )
}

Since the components are attached to the GraphQL architecture, they must also embrace the gql schema structure within.

Each of the phases (loading, empty, failure, success) is automatically managed by Redwood. You only need to overwrite them with your code or remove them in case they're not needed.

Great! I got it. But, how does it work on the back-end side?

Redwood is GraphQL-based by default, which means that you'll need to define a GraphQL SDL. Usually, you need to write resolvers to let GraphQL understand where to route the incoming requests and deliver the outgoing responses.

Redwood simplifies this by doing it automatically. Based on your SDL specifications, services are auto-generated and each query or mutation is redirected to the specific service method. Take the following SDL as an example:

export const schema = gql`
  type Post {
    id: Int!
    title: String!
    body: String!
    createdAt: DateTime!
  }

  type Query {
    posts: [Post!]!
    post(id: Int!): Post!
  }

  input CreatePostInput {
    title: String!
    body: String!
  }

  input UpdatePostInput {
    title: String
    body: String
  }

  type Mutation {
    createPost(input: CreatePostInput!): Post!
    updatePost(id: Int!, input: UpdatePostInput!): Post!
    deletePost(id: Int!): Post!
  }
`

It simply exposes two queries and three mutations to create a CRUD API over the posts' domain.

The generated services usually work directly with the database to retrieve and update the information, but you can customize the service with whatever actions you want:

import { db } from 'src/lib/db'

export const posts = () => {
  return db.post.findMany()
}

export const post = ({ id }) => {
  return db.post.findOne({
    where: { id },
  })
}

export const createPost = ({ input }) => {
  return db.post.create({
    data: input,
  })
}

...

You can customize these functions to retrieve data from a database, other API services, serverless lambdas, etc. Whatever you prefer.

Each operation will also automatically provide successful results within the Success cell component that we've previously seen. As simple as that!

Redwood also offers other features like generators to avoid boilerplate code and forms to simplify the development of web forms along with React. For more on what you can do, please refer to its official docs.

Turbine Your SPAs Without JavaScript Frameworks

Have you ever found yourself uncomfortable with the "blinks" when transitioning from one SPA page to another? Have you ever heard about Turbolinks?

It is a small and lightweight library that coexists with your current server-rendered apps and makes navigating between pages faster by replacing the usual full page loads with partial page loads.

It works by intercepting the clicks within your page that target the same domain, i.e., the same server-based application. When the click is intercepted, the browser is prevented from requesting it and, instead, Turbolinks changes the browser's URL via history API.

Then it processes the request through an AJAX call and renders the response in the form of HTML.

It sounds simple, doesn't it? It is, in fact, simple.

Import the script into your head tag or add the npm package to your Node.js project, and you're ready to go:

npm install turbolinks

While you don't need to reload the whole page and, consequently, improve performance; you also need to pay attention to your code design. You can't rely on page loads to restart a state anymore and must be aware that your JavaScript global objects (like window) will retain the in-memory state. So, be careful.

Other than that, Turbolinks also provides awesome features like:

Caching. It keeps a cache of the recently visited pages. If you go back to some of the history pages, it'll optimize the experience to make sure no call to the server is performed.
On-demand Scripts. If the subsequent pages you navigate to need to load new script elements, Turbolinks will handle that by appending them to the head tag. That's great to have — loaded on-demand scripts — they enhance the overall performance.

Make sure to refer to the official docs for the API Reference and some nice examples.

What if We Don't Use JavaScript at All?

I know, that sounds disruptive, not to mention too contrarian, but there are some guys revisiting the past to create new stuff, like Phoenix LiveView, for example.

Some parts of the web community has critics debating the number of languages (or tools) needed to create something for the web. For example, is it really necessary to replicate the same JavaScript logic developed in the front-end to the Node.js back-end?

What if the state becomes fully controlled by the back-end rather than having agnostic APIs to provide endpoints for every change performed by the client?

Take the LiveView use case. LiveView is a server-state framework, which means that the state is kept under the server, and managed within it.

In other words, LiveView controls the state of the app — watching for changes made by the client and re-rendering the partial chunks related to that interaction back to the browser. The browser, in turn, will have a mechanism that understands these dynamics and updates the pages accordingly.

This means that we don't need to track down every single change happening to the client. We create the client HTML, program the server capabilities, and leave the change listening to the framework.

That's just one framework example (made in Elixir) out of many fermenting out there, such as Stimulus, and Laravel Livewire.

There are some work-in-progress Node.js ones, like Purview, but it's still in the early stages. Take this example from the official repo:

import Purview from "purview"
import * as Sequelize from "sequelize"

const db = new Sequelize("sqlite:purview.db")

class Counter extends Purview.Component<{}, { count: number }> {
  async getInitialState(): Promise<{ count: number }> {
    // Query the current count from the database.
    const [rows] = await db.query("SELECT count FROM counters LIMIT 1")
    return { count: rows[0].count }
  }

  increment = async () => {
    await db.query("UPDATE counters SET count = count + 1")
    this.setState(await this.getInitialState())
  }

  render(): JSX.Element {
    return (
      <div>
        <p>The count is {this.state.count}</p>
        <button onClick={this.increment}>Click to increment</button>
      </div>
    )
  }
}

Remember that this code exists within the back-end side of the application, which is really cool.

It resembles a bit of what we have with Redwood. The server code communicates directly with the database, has some well-defined phases (like the init state from React), and sets up a render method with the HTML output.

Chances are that Next.js is going to provide similar features in the near future, which would be groundbreaking for the Node.js universe.

Wrapping Up

Where to go from here? There are so many options that it's hard sometimes to choose a path... we know!

The first tip I'll give you is to measure and discuss what's the purpose of the app you're building. Not every framework and library may suit your app's needs every time.

Take the htmx library as an example. It is a super small ~8k dependency-free lib that helps you to easily perform AJAX calls and deal with WebSockets and SSE in your HTML. There's no need for a full SPA framework here.

You first import it and then, program your HTML elements to perform a POST request via AJAX updating the DOM once it's finished. For example:

<!-- Load from unpkg -->
<script src="https://unpkg.com/htmx.org@0.3.0"></script>
<!-- have a button POST a click via AJAX -->
<button hx-post="/clicked" hx-swap="outerHTML">
  Click Me
</button>

Chances are that you've never heard of some of the tools we've talked about here. Whatever the case, they represent strong alternatives that you can try and figure out if they fit your reality or not. Give them a try!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node.js or you're already familiar with AppSignal, go and check out AppSignal for Node.js.

Diogo Souza has been passionate about clean code, software design and development for more than ten years. If he is not programming or writing about these things, you'll usually find him watching cartoons.

JavaScript Internals: Garbage Collection

Diogo Souza — Tue, 15 Dec 2020 15:28:03 +0000

Garbage collection (GC) is a very important process for all programming languages, whether it's done manually (in low-level languages like C), or automatically.

The curious thing is that most of us barely stop to think about how JavaScript — which is a programming language, and hence, needs to GC — does the trick.

Like the majority of high-level languages, JavaScript allocates its objects and values to memory and releases them when they’re no longer needed.

But, how? How does it work internally?

Well, this article aims to tackle this particular side of the language. Let’s go, then!

👋 Did you know that AppSignal gives away free accounts for amazing OSS projects?

JavaScript Memory Lifecycle

First of all, let’s clarify that this article is targeting how JavaScript tackles GC on web browsers. We’ve already covered GC on Node.js’s V8 in another article. Yep, go for it too!

The memory lifecycle for pretty much every programming language works as follows:

Languages’ memory lifecycle.

The differences reside in the way they do it (i.e. what algorithms they use) and how each phase must be addressed (manually or automatically).

In JavaScript, the allocation and deallocation phases are automatic. However, it doesn’t mean that developers should only care about the use of the available memory.

Stuff like infinite loops, badly-implemented recursion, and callback hells can drown your memory in no time and lead to memory leaks.

So, yes, the way you code — and, therefore, allocate/release memory slots — is also very important in avoiding such scenarios from happening.

Back to the cycle.

JavaScript works pretty much this way. It allocates space when new variables are created:

var bar = "bar"

And when the memory is no longer being used, respecting the language limitations in terms of variable scopes, the memory is released.

But, how does JavaScript know the memory that's no longer in use? Through its Garbage Collector.

Garbage Collection Strategies

JavaScript uses two famous strategies to perform GC: the Reference-counting technique and the Mark-and-sweep algorithm.

The reference-counting approach is known for its versatility. You can count the number of references pointing to each allocated resource, whether it's a bunch of files, sockets, or memory slots.

It considers that each allocated object in memory will contain a count field (that works as a reference) attached to it. Whenever the object has no references pointing to it anymore, then it is automatically collected.

Consider the following example:

var bar = {
    name: "bar"
};
bar = "";

Two objects are created here: bar and name. Since bar is receiving a new value on the last line, then name can be garbage collected.

Simple, isn’t it? Now, imagine that your code evolves to the following:

var bar = {
    name: "bar"
};
var bar = "foo";

function check() {
    var bar = {};
    var foo = {};
    bar.name = foo;
    foo.name = bar;

    return true;
}
check();

JavaScript is a reference-based language when it comes to its objects, which means that the object names point to in-memory instantiated values. More than that, children’s objects/variables are automatically referenced by their parents.

In the above example, we have a cycle being created. The bar inside the check function is referencing foo and vice versa.

Usually, when a function finishes its execution, its inner elements are garbage collected. However, in this case, the GC is unable to do it since the objects are still referenced to one another.

And that’s where the second JavaScript GC actor comes into the scene: the mark-and-sweep algorithm.

This algorithm works by searching for objects that are unreachable from JavaScript’s top object — the root’s global object.

Take the following representation of the previous bar object:

How JavaScript tracks its objects.

As you can see, JavaScript can easily track down the name object since its hierarchy is well defined.

What happens, then, when the following code snippet runs?

var bar = "foo";

Here you go:

No longer reachable object.

See? We can’t track the object from the root anymore.

The rest of the process is pretty intuitive: the algorithm will go a couple of times, from the root to the bottom objects (and their respective hierarchies) marking — to be ignored — all the objects that are reachable and sweeping from memory at the end of the process, the ones that are not. Like the name object.

It actually makes a lot of sense, doesn’t it?

This process is repeated over and over through some internal conditions that only JavaScript’s GC knows, which is common to most of the GCs out there.

Node.js Garbage Collection

Before we can jump right into the details of how Node.js performs garbage collection, we need to understand two special actors on the set: the heap and stack.

The heap refers to the portion of memory dedicated to the storage of reference types. Reference types are everything that includes objects, strings, closures, etc.

So, whenever you see an object created in JavaScript, this object will be placed on the heap:

const myCat = new Cat("Joshua");

Meanwhile, the stack is the place where references to those objects created on the heap are contained. Function arguments, for example, are good examples of references existing on the stack:

function Cat(name) {
   this.name = name;
}

With all that said, how does V8, which is the JavaScript engine behind Node.js, perform GC?

The heap is divided into two main parts called New Space and Old Space.

New Space vs Old Space.

The New Space is the region of memory that allocates new objects and variables and, therefore, is way faster to GC since everything is fresh. As the name suggests, objects living here belong to the Young Generation.

The Old Space is the place that the objects that weren't collected in the New Space head to after some time. They're called the Old Generation. It also stores other types of objects here like too large objects and V8 compiled code, but we won't focus on them.

Node.js will do the best it can to avoid GC into the Old Space since it costs more to do so. This is why only up to 20% of the objects migrate from the Young to the Old Generation. That is also the reason why we have two different algorithms to deal with each generation:

Scavenge: this garbage collector takes care of the Young Generation by cleaning up small portions of memory every time it runs. It is super fast, which fits very well with the Young Generation nature.
Mark-and-Sweep: we know this guy already. Since it is slower, it's the perfect choice for the Old Generation.

Identifying Memory Leaks in Node.js

A great way to see how JavaScript deals with memory in Node.js is through a classic memory leak example. Remember that a memory leak happens when all GC strategies have failed to find the object because it lost its connection to the root object. Other than that, we can also have a leak when an object is always referenced by other objects and, at the same time, continues to grow in size.

For example, imagine that you have a simple Node.js server that you manually created and you want to store some important data from all the requests, as seen below:

const http = require("http");

const ml_Var = [];
const server = http.createServer((req, res) => {
  let chunk = JSON.stringify({ url: req.url, now: new Date() });
  ml_Var.push(chunk);

  res.writeHead(200);
  res.end(JSON.stringify(ml_Var));
});

const PORT = process.env.PORT || 3000;
server.listen(PORT);

So, we're creating a manual audit log from our requests. The variable ml_Var is the dangerous spot in our code since it is a global variable and, therefore, is going to live in memory until the server shuts down (which can take a long time).

Objects like that can become a huge problem in your apps, especially because other developers can add items to the array in other places that you won't be able to monitor.

To simulate the scenario, we're going to make use of the Google Chrome DevTools. Wait, but this is a Node.js application... right? Yes, because both Chrome and Node.js uses the same JavaScript Engine (V8), the DevTools can understand how to debug and memory inspect both universes. Isn't it great?

All that you need to do is to start your Node.js server with an --inspect flag:

node --inspect index.js

After that, you may see the following output:

Debugger listening on ws://127.0.0.1:9229/16ee16bb-f142-4836-b9cf-859799ce8ced
For help, see: https://nodejs.org/en/docs/inspector

Now, head to your Chrome (or Chromium) browser and enter the chrome://inspect address. The following screen may appear:

Google Chrome DevTools Remote Target.

Within the "Remote Target" section, there's an "inspect" link. When you click on it, the DevTools extension may open up with a direct session for your Node.js application. You'll be able to see the logs, sources, perform CPU profiling and memory analysis as well.

If you head to the Memory tab, you'll see a "Take snapshot" button located at the bottom of the page. Click on it and the DevTools will generate a heap snapshot profile (a memory dump) of our current running application. Since the goal is to compare the memory before and after the leak happens, that's our first step in the process.

However, before we can take the other memory dumps, we need an auxiliary tool to help with benchmarking. In other words, we need to stress the application with many requests to validate the memory leak. And siege.js is the perfect tool for that.

Siege is a Node.js benchmarking tool that simplifies the task of running hundreds or thousands of requests against an endpoint.

First, we'll need to run the npm install siege --save command to get it installed and then, create another JavaScript file called benchmark.js and add the following content:

const siege = require("siege");

siege()
  .on(3000)
  .for(2000).times
  .get('/')
  .attack()

Here, we're asking siege.js to run a total of 2000 requests on the root endpoint located under port 3000. As simple as that!

Great! Now, we can move on to the other heap snapshots. Run the benchmark file:

node benchmark.js

Wait until it finishes. It'll produce the following output:

GET:/
    done:2000
    200 OK: 2000
    rps: 1709
    response: 5ms(min)  23ms(max)   9ms(avg)

Get back to DevTools and hit the "Take snapshot" button again. Just for safety, let's repeat the process once again until we have 3 snapshots. This will help to fine-tune the overall memory analysis.

DevTools results.

There are a couple of points to clarify here:

The list of head snapshots. Select the third one to compare against the second.
We need to select "Comparison" to enable the DevTools comparison features.
Select the snapshot you'd like to compare with.
The list of constructors created within the memory. The "# New" column will show the number of new objects created from the previous snapshot to the current one. Pay attention to the content of each string, they correspond to the JSON request logs we've created.
The "Object" section brings details over the stack that has created each object. For the JSON strings, ml_Var is the context in which they were created.

It's interesting to see that 2014 string objects were created from one snapshot to another. The 2k refers to the request logs we introduced, the other 14 are strings created and managed by Node.js itself.

In our example, only 3 executions led to 4k new objects in memory. Imagine such a scenario in a real application running in production. In no time, the memory would leak until there was nothing left.

Now that you've identified the leak, the solution is quite simple. Just make sure to store those logs into a file, to an external service (like Splunk) or even to a database.

Wrapping Up

Do you now understand the importance of proper attention when coding your JavaScript applications in terms of object allocation and deallocation?

As further reading, I’d recommend the famous IBM study of memory leak patterns in JavaScript, which explores the consequences of circular references in the language.

If you'd like to read more about memory leaks in Node.js, I strongly recommend Deepu Sasidharan's article in which he talks about the best practices for performance in Node.js.

Mozilla's official docs also bring a handful of great articles about performance, including profiling, performance measurements, and automation. See you around!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node.js or you're already familiar with AppSignal, go and check out AppSignal for Node.js.

Getting Started With Web Vitals in Next.js

Diogo Souza — Thu, 01 Oct 2020 13:04:15 +0000

In this article, I’ll try to guide you through some examples and definitions that aim to clarify the Web Vitals landscape from a Next.js perspective. Let's dive in!

How Google Grades Your Websites

User experience is something that Google appreciates when its robots scan your website. They perform checks to make sure your website deserves a good spot on Google's famous search engine results page.

They look for quality indicators such as performance, interactivity, the structure of the pages, responsiveness, safety (e.g. HTTPS), etc.

If you've ever ventured into SEO waters, chances are that at first, you felt overwhelmed by the amount of stuff to worry about.

For this reason, Google came to the rescue with Web Vitals. They are a new way of analyzing your web pages and checking beforehand for things you might need to address and improve.

Web vitals are a guide made with everybody in mind so that you can easily find out how your website performs. In case there are issues, you should be able to figure out how to address them with ease.

What Are Web Vitals?

To describe this a bit better, let’s check out a Chrome tool called Lighthouse. If you’ve never heard about it, it’s an open-source automated tool that analyzes and collects quality metrics of web pages; and, yes, it makes use of Web Vitals principles.

The tool is pretty straightforward. On the page that you want to analyze, right-click -> inspect -> look for Lighthouse in the top bar. From there, there are a few options you can choose from:

Picking up your Lighthouse preferences.

When we run the tool against the AppSignal homepage, we’ll get similar results to these:

AppSignal’s performance metrics.

Here, we’re just showing the metrics related to the Performance section because they embrace more of what Web Vitals do. However, Lighthouse does more.

In short, Web Vitals are divided into six main categories, among which, three are classified as Core Web Vitals to know:

Metric	Description	Graph reference
Largest Contentful Paint (LCP)	Here, Google tries to analyze the page loading time as it is perceived by the user, i.e., how long does it take for the main content of your page to load? If the answer is >2.5s then it needs to be improved. Of course, the lower the better.
First Input Delay (FID) / Total Blocking Time (TBT)	Now, let’s measure how long it takes for the first user interaction to happen within your page. Whether it is through a button click, a page scroll, your browser needs to be ready to respond to those commands in no time, even if the page is not fully loaded. If the time is greater than 100ms, Google asks you to work on that. Oh, yes, the second indicator, TBT, is an auxiliary measurement that analyzes the difference between the FCP and the TTI 一 more on them soon. In the end, they’re birds of a feather.
Cumulative Layout Shift (CLS)	This one counts the number of times that stuff or components within a page move (shift) around during the loading. Have you ever entered a website in which the elements start to “dance” alone before we can even figure out what’s the website about? So, the more you have this, the poorer the experience.

Graph’s source: https://web.dev/vitals/

As you may have perceived, The Core Web Vitals are worried about three main points: the loading time, the responsiveness (interactivity), and the stability of the page.

The other three non-Core Web Vitals are:

Time to First Byte (TTFB): it’s the time the browser takes to receive the first byte of page content.
First Contentful Paint (FCP): calculates the time it takes for any content on the page to first appear on the screen.
Time to Interactive (TTI): it’s a metric of the time from when the page loading gets started until it reliably responds to user input.

Those Vitals are considered more as of auxiliary since they help the Core ones (as well as Lighthouse) to calculate their scores.

Now that you understand a bit more about them, you can refer back to AppSignal’s Lighthouse metrics in the previous figure and recognize what all the indicators, along with their values, are for.

For more details on how Google calculates these metrics behind-the-scenes, please refer to the official docs.

Web Vitals With Next.js

Yes, since version 9.4.0, Next.js comes with a built-in layer that understands Web Vitals and allows you to collect such metrics within your Next apps.

Let’s see how it works. First, we need a Next.js app. We’ll be making use of Yarn as the package manager.

Run the following command:

yarn create next-app

When it prompts you to fill in the app name, give it the value “web-vitals-next”. Once the creation is finished, open your project with VS Code and run the build command:

yarn build

Finally, run the project with the yarn dev command. It’ll be available, by default, at http://localhost:3000/.

Introducing Web Vitals to your Next.js app is very simple. You just need to make sure you have a custom App component (which our yarn creation command already did) and add the following function to your pages/_app.js (or .ts, for TypeScript) file:

export function reportWebVitals(metric) {
  metric.label === "web-vital" && console.log(metric);
}

This function alone will execute whenever one of the Web Vitals metrics takes place. In the image below, you get to see how it works by logging to the browser’s console, the metrics as they happen:

Web Vitals metrics logged in the browser’s console.

The structure of a metric object follows this pattern:

Web Vitals metric object.

It’s important to sort your logs by the label value, especially because Next.js also logs some custom metrics that you may not need.

However, metrics in the browser’s console are not useful. We need to send them to an analytics platform so they can be processed and digested to generate real-time and accountable information.

If you use Google Analytics as such tool, sending the data would be as simple as this:

export function reportWebVitals({ id, name, label, value }) {
  ga('send', 'event', {
    eventCategory:
      label === 'web-vital' ? 'Web Vitals' : 'Next.js custom metric',
    eventAction: name,
    eventValue: Math.round(name === 'CLS' ? value * 1000 : value), // values must be integers
    eventLabel: id, // id unique to current page load
    nonInteraction: true, // avoids affecting bounce rate.
  })
}

Sending Metrics to AppSignal

To have a better view of the metrics, let’s take our example to send Web Vitals metrics to AppSignal dashboards as shown below:

Pushing metrics to AppSignal is super easy and fast!

First, you need an AppSignal account 一 you have a 30-days free trial. Once you’ve logged in, you’ll be redirected to the startup page, on which you get to choose the language of the app you desire AppSignal to be installed to.

Select Node.js. The page that follows will show the instructions to install AppSignal to your Node.js project. Keep the APPSIGNAL_PUSH_API_KEY key, since it’s going to be important later on.

Let’s move on to the app changes. You first need to add the AppSignal npm packages by running:

yarn add @appsignal/nodejs @appsignal/nextjs
yarn install

Pay attention to the log results and check if it finished successfully.

Secondly, AppSignal integration doesn’t work with Next CLI, which is the tool that our example app was built with. That means that we have to create our own custom server script file and start the application through it.

Go ahead and create a new file called server.js into the app’s root folder. This is the content it must have:

const { Appsignal } = require("@appsignal/nodejs");
var http = require('http');

const appsignal = new Appsignal({
  active: true,
  name: "web-vitals-next",
  apiKey: "PUT_YOUR_KEY_HERE", // <<-- Change this!!
});

const {
  getRequestHandler,
  EXPERIMENTAL: { getWebVitalsHandler },
} = require("@appsignal/nextjs");

const url = require("url");
const next = require("next");

const PORT = parseInt(process.env.PORT, 10) || 3000;
const dev = process.env.NODE_ENV !== "production";
const app = next({ dev });

const handle = getRequestHandler(appsignal, app);
const vitals = getWebVitalsHandler(appsignal);

app.prepare().then(() => {
  http.createServer((req, res) => {
    // Be sure to pass `true` as the second argument to `url.parse`.
    // This tells it to parse the query portion of the URL.
    const parsedUrl = url.parse(req.url, true);
    const { pathname, query } = parsedUrl;

    if (pathname === "/__appsignal-web-vitals") {
      vitals(req, res);
    } else {
      handle(req, res, parsedUrl);
    }
  }).listen(3000, (err) => {
    if (err) throw err;
    console.log("> Ready on http://localhost:3000");
  });
});

Attention: Don’t forget to change the apiKey in the code listing to yours.

Note that this is creating a new server with custom settings, but it’s still Next-only based. You can also configure the whole thing with Express if you want to.

Special attention to the Appsignal object, which is where AppSignal’s Node.js library tries to connect to the remote API for the first time. If anything goes wrong, this is the place where you should inspect errors.

We’re setting up the app with minimal configs, but you can find a list with all the available config options here.

Note also that, within the server setup, we’re checking if the pathname equals to /appsignal-web-vitals. This is necessary because the feature works by providing a handler function, which is designed to be used as an endpoint in your application.

This handler function, in turn, is the reportWebVitals we’ve seen before. Here’s its new content:

export function reportWebVitals(metric) {
  metric.label === "web-vital" && console.log(metric);

  const body = JSON.stringify(metric);
  const url = "/__appsignal-web-vitals";

  // Use `navigator.sendBeacon()` if available, falling back to `fetch()`.
  (navigator.sendBeacon && navigator.sendBeacon(url, body)) ||
    fetch(url, { body, method: "POST", keepalive: true });
}

Alternatively, you may remove the console logging if you don’t want to see the logs in the browser.

The implementation makes use of the sendBeacon function with a fallback to a POST request to the AppSignal API. As simple as that!

Finally, let’s go to the package.json file and change our generated scripts:

"scripts": {
   "dev": "node server.js",
   "build": "next build",
   "start": "NODE_ENV=production node server.js"
},

Now, start the project locally via yarn dev command, get back to the AppSignal setup wizard and click Next step. After that, another page will appear with a 60s timer waiting for the requests to arrive from your Next app.

Always good to remember that the usage of this feature is EXPERIMENTAL and may change or be deprecated in future releases. So, be aware!

Summary

In terms of web quality measuring, Web Vitals represent the best of what’s available in the community. It’s Google’s child and has been largely embraced by the community as a trustworthy metric system to check whether your apps are good enough or still need more work.

Obviously, it’s continuously evolving, that’s why closely watching the official docs and repo is always recommended.

Apart from that, since it's not bulletproof, be sure to make other tests and/or check out the results over different platforms to ensure that the quality is what you expected. Good luck!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node or you're already familiar with AppSignal, go and check out AppSignal for Node.js.

Security Best Practices for Node.js

Diogo Souza — Thu, 03 Sep 2020 15:30:31 +0000

Because a lot of systems are connected to the web these days (or, at least, communicate/integrate with it at some level), companies are giving more and more attention to web security.

Web security usually comes to public attention when certain events reach the news, for example, security leakages, hacker activities, and/or data-stealing over big companies, some of them really large (like Google, LinkedIn, etc.).

Apart from that showbiz world of giant players that most of us are probably not working for, implementing security on your systems is not only important but impressively underestimated or even forgotten by many devs.

Setup, best practices, performance, testing, and metrics are probably things that you consider in your daily programming life. However, unfortunately, that’s not the same for security best practices.

And it’s not due to warnings. If you work in the open-source universe, within GitHub’s protective arms, chances are that you’ve faced some of its alerts for vulnerable dependencies. The code community platform is becoming increasingly good – as well as worried – at detecting vulnerabilities in thousands of different libs across many different languages.

Today, it’s way more accessible for small and medium companies to afford security tools (or perhaps whole platforms) to assist their developers with the gaps in their code and apps.

👋 AppSignal now supports Node.js as well. We've added a whole array of integrations i.e. support more things to be instrumented right out-of-the-box. We already had support for Next.js, PostgreSQL, Redis, and Express and just added Apollo. Learn more about AppSignal for Node.js.

Nevertheless, whether you use or don't use such security platforms, understanding and being aware of the security threats that your apps may suffer from and fighting against them through simple (but powerful) best practices is the main goal of this article.

Actually, we’ll pick Node.js as the analysis guinea pig, but many of the items here perfectly align with other platforms as well.

As a matter of reference, the OWASP (Open Web Application Security Project) will guide us through its Top Ten most critical security risks for web applications, in general. It is a consensus board created out of the analysis of its broad list of members. Let’s face it under the light of Node then.

Injection Attacks

One of the most famous threats to web applications relates to the possibility of an attacker sending pieces of SQL to your back-end code.

It usually happens when developers concatenate important SQL statements directly into their database layers, like so:

// "id" comes directly from the request's params
db.query('select * from MyTable where id = ' + id);
   .then((users) => {
     // return the users into the response
   });

If the developer didn’t sanitize the input parameters arriving within the request, an attacker could pass more than a single integer id, like an SQL instruction that could retrieve sensitive information or even delete it (not to mention the importance of proper backup policies here).

Most of the programming languages, and their respective ORM frameworks, provide ways to avoid SQL injection usually by parameterizing inputs into query statements that, before executing directly into the database, will be validated by the inner logic of your language libs machinery.

In this case, it’s very important to know your language/framework closely in order to learn how they do this.

If you make use of Sequelize, for example, a simple way to do it would be:

const { QueryTypes } = require('sequelize');

await sequelize.query(
  'select * from MyTable where id = :p1',
  {
    replacements: { p1: id }, // id comes from the request's param
    type: QueryTypes.SELECT
  }
);

Authentication Pitfalls

Authentication is usually a part of the system that requires a lot of attention, especially if you make use of frameworks or tools that easily allow developers to expose sensitive user’s information.

OWASP considers this item critical. Standards like OAuth (on its 2nd version now, working on the 3rd) are constantly evolving in an attempt to embrace as much as possible the many different realities of the web world.

Its implementation can be tricky, depending on your project’s scenarios or on how your company decides to customize the standard usage.

If your team (and company) can afford to add big – and therefore, mature – players like Auth0, Amazon Cognito, and many others in the market to your projects, that would be halfway there.

When it comes to implementing OAuth2 in Node.js, there’s plenty of compliant and open source options that can help you with not starting from scratch. Like the famous node-oauth2-server module.

Make sure to always refer to the official docs of whatever module or framework you’re adding to your projects (whether it's open source or paid). Plus, when adding security to your auth flows, never go with small and recent open-source projects (it’s too much of a critical part of the app to take that kind of risk).

Sensitive Data Exposure

It’s important to define what sensitive data is. Depending on the type of project, it can vary. However, regardless of the app nature, things like credit card and document ids will always be sensitive, for sure.

How is that information going to transit over to your system? Is it encrypted? No? Really?

After you’ve separated what’s “really important” from the rest, it’s time to decide what needs to be stored, and for how long.

You’d be amazed at the number of apps out there that store sensitive information for no further use, or worse, without the user’s consent. That can easily break the laws of data privacy which, by the way, are different depending on the country your app is running on (another thing to worry about).

Let’s go to the to-do (aka must-to) list:

Encrypt your sensitive data. Forget about MD5, your data deserves to be strongly protected under the right algorithms. So go for Scrypt.
Warn your users about how your application deals with sensitive info. You can periodically mail them with explaining infographics, pop up some informative modals when logging in, and yes, your terms of use must state this too.
Go for HTTPS. Period. Google won’t like you nowadays if you’re not.
If you can, go a bit further and do HSTS. It is a policy mechanism that enhances your web security against the famous man-in-the-middle attacks.

Setting HSTS in a Node app is as easy as:

const hsts = require('hsts');

app.use(hsts({
  maxAge: 15552000  // 180 days in seconds
}));

You can fine-tune your settings by defining, for example, if subdomains should be included or not:

app.use(hsts({
  maxAge: 15552000,
  includeSubDomains: false
}));

You’ll need, obviously, the hsts npm package. Make sure to refer to its official docs for more info.

Old XML External Entities (XXE)

XXE attacks occur by exploring the vulnerabilities that older XML processors have, in which they allow attackers to specify external entities and send them to applications that parse XML inputs.

If the parser is weakly configured, the attacker could have access to sensitive information, confidential data like passwords in a server, among others.

Consider, as an example, an XML-based Web Service that receives the following XML content as input:

<?xml version="1.0" encoding="ISO-8859-1"?>
   <id>1</id>
   <name>attacker@email.com</name>
   ...
</xml>

At first sight, it looks just like all the other inputs you’ve seen so far. However, if your app that's hosted on a server is not prepared to deal with attacks, something like this could be sent over:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE foo [
  <!ELEMENT foo ANY >
  <!ENTITY xxe SYSTEM "file:///c:/boot.ini" >]>
    <id>1</id>
    <name>attacker@email.com</name>
    ...
  <foo>&xxe;</foo>
</xml>

And this would return in the response the boot.ini file content.

Another good example is if your app deals with uploading files. If, for example, you restrict it to only accepting some group of files, then XML-based formats like DOCX or the famous SVG for images could be accepted and carry on malicious code as well.

The easiest way to prevent such attacks is by disabling your library’s parsing features. The node-libxml npm package, for instance, provides a bunch of functions to validate your DTD and helps you secure your apps against these attacks.

Broken Access Control

This item is mostly related to how well-tested an application has been when it comes to user permissions to different areas (or URLs) of it.

In other words, if you’re supposed to have restricted areas on the application, like an admin dashboard, for example, and ordinary users without a proper role can access it anyway, then you have an access vulnerability.

It is easily correctable and doesn’t require any specific solution, you can go with whatever you’re using already. The only point is the attention to implement it correctly and cover it with proper tests that guarantee the coverage over new endpoints as well.

Node provides plenty of libraries to help with that, as well as middleware to check for the current user’s permissions and you can also implement one on your own.

Security Misconfiguration

It’s common, in the early stages of an app’s life, to define three major environments (development – or stage, QA, and production) and leave the settings equal among them.

This type of misconfiguration sometimes goes on for ages without being noticed and can lead to critical attacks, since the app is vulnerable considering that staging and QA configurations are weakly protected most of the time.

When we talk about configurations, make sure to associate them to all types of dependencies (databases, external integrations, APIs, gateways, etc.).

It’s fundamental to have well-defined setups, distinct and separated from each other. Also, consider storing your credentials (and sensitive setting’s data) in remote places apart from the project files.

The cultural aspects of your company may take place here as well. If you use Splunk, for example, or any other logging tool, make sure to have policies (and ways to check that) that force the devs to not log sensitive data since Splunk can be way more easily accessed than the database that stores the same data.

That reminds me of a time in a company in which the main database’s password went up to a public GitHub repo due to a developer that “innocently” copied one of the company’s repo to study at home. And don’t get me wrong... I’m not saying that the biggest error was his; it was not.

The Notorious XSS

XSS is a notorious rebel. Even though it's insanely famous, everyday rush can easily make you forget about it.

The problem here resembles SQL injection. You have an endpoint in your web app that receives a request and returns a response. Not a big deal. However, it becomes one when you concatenate the request data with the response without sanitizing it.

A classic example would be:

app.get('/users', (req, res) => {
  const user = db.getUserById(req.query.id);
  if (!user) {
    return res.send('<span>Sorry, the user "' + req.query.product + '" was not found!</span>');
  }
  ...
});

Guess what's going to happen when the client sends a request with the following id param:

<script>alert(Uh la la, it's me! XSS!!)</script>

For now, it’s just an innocent alert message, but we all know that an attacker would put a bit more of JavaScript code in there.

Node’s full of options to address this issue by simply adding a new middleware. Pick one, implement it properly, and move on.

Insecure Deserialization

This breach takes place mostly when applications accept serialized objects from untrusted sources that could be tampered with by attackers.

Imagine, for example, that your Node web app communicates with the client and returns after the user has logged in, a serialized object to be persisted in a cookie that will work as the user's session, storing data like the user id and permissions.

An attacker could then change the cookie object and give an admin role to himself, for example.

Here’s where terms like CSRF (Cross-site Request Forgery) pop up. Basically, the server app generates a token (known as CSRF token) and sends it to the client in every request to be saved in a form’s hidden input.

Every time the form is submitted it sends the token along and the server can check if it has changed or is absent. If that happens, the server will reject the request. In order to get this token, the attacker would have to make use of JavaScript code. If your app, however, doesn’t support CORS, the attacker has his hands tied and the threat is eliminated.

Again, Node has some great middleware packages to help you out, like the csurf, one of the most famous. Within less than 2 minutes, you’re safe and sound.

Insufficient Logging & Monitoring

This item speaks for itself. We’ve talked about Splunk before, but this is just the tip of the iceberg in terms of available options.

Tons of different tools, plenty of them even integrating and talking to each other, provide the perfect layers to enhance your system’s protection, based on information.

Information is crucial to analyze and detect possible invasions and vulnerabilities of your app. You can create lots of routines that execute based on some predefined behaviors of your system.

The logs speak for what’s happening inside your app. So the monitoring represents the voice of it that’ll come at you whenever something wrong is detected.

Here, we won’t talk about specific tools. It’s an open field and you can play with the sea of great solutions out there.

Wrapping Up

We’ve taken a look at the OWASP Top Ten Web Application Security Risks at the time of writing. But obviously, they’re not the only ones you should pay attention to.

The list works as a compass for developers, especially beginners, to better understand how threats exist on the web and how they can affect your apps, even though you disbelieve someone would try to hack you.

Remember, the bigger and important your applications are, the more susceptible to security breaches and bad-intended people they are.

As further reading, I’d very much recommend a tour over the OWASP website, as well as at its Source Code Analysis Tools page. Good luck!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node or you're already familiar with AppSignal, go and check out AppSignal for Node.js.

A Deep Dive Into V8

Diogo Souza — Thu, 16 Jul 2020 10:05:22 +0000

A majority of front-end developers deal with this buzzword all the time: V8. A big part of its popularity is due to the fact that it led JavaScript to a new level of performance.

Yes, V8 is very fast. But, how does it perform its magic and why is it so responsive?

The official docs state that “V8 is Google’s open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and Node.js, among others”.

In other words, V8 is a software developed in C++ that translates JavaScript into executable code i.e. machine code.

In this epiphanic moment, we start seeing things more clearly. Both Google Chrome and Node.js are just bridges to transport the JavaScript code to its final destination: machine code running in that specific machine.

Another important role in V8's performance play belongs to its generational and super accurate Garbage Collector. It was optimized to collect the objects that JavaScript no longer needs, using low memory.

Besides that, V8 counts on a set of other tools and features to improve some inherent JavaScript functionalities that historically, make the language slow (like its dynamic nature, for example).

In this article, we'll explore those tools (Ignition and TurboFan) and features in more detail. More than that, we'll cover the basics of V8's internal functioning, compilation and garbage collection processes, single-threaded nature, and more.

Let's go!

Starting With the Basics

How does machine code work? Machine code, in short, is a bunch of very low-level instructions that execute in specific parts of the machine’s memory.

The process of generating it, using C++ language as a reference, is similar to this:

Before going any further, it’s important to point out that this is a compilation process, which is different from the JavaScript interpretation process. While the compiler, in fact, generates a whole program at the end of the process, the interpreter works as a program itself that does the job by reading the instructions (usually as scripts, like the JavaScript scripts) and translating them into executable commands.

The interpreting process can happen both on-the-fly (on which the interpreter parses and runs only the current command) or fully-parsed (that’s when the interpreter first translates the script entirely before proceeding with the respective machine instructions).

Back to the figure, the compilation process usually starts with the source code, as you know. You implement the code, save it and run. The running process, in turn, starts at the compiler. The compiler is a program, like any other, running on your machine. It then goes through all the code and generates object files. Those files are the machine code. They’re optimized code that runs in that specific machine, that’s why you’ll have to use a specific compiler when you move from one OS to another.

But you can’t execute separate object files, you need to combine them into a single file, the well-known .exe file (the executable file). That’s the linker’s job.

Finally, the loader is the agent responsible for transferring the code inside of that exe file to the virtual memory of your OS. It is basically a transporter. And here, you have your program finally up and running.

Sounds like a lengthy process, isn't it?

Most of the time (unless you’re a developer working with Assembly in a bank’s mainframe) you’ll spend your time programming in high-level languages: Java, C#, Ruby, JavaScript, etc.

The higher the language is, the slower it is. That’s why C and C++ are so much faster, they’re very close to the machine code language: the assembly language.

One of the main benefits of V8, apart from the performance, is the possibility of going beyond the ECMAScript standards and understand, for example, C++ as well:

JavaScript is restricted to ECMAScript. And V8, in order to exist, must be compliant but not restricted to it.

Having the ability to incorporate C++ features into V8 is great. Since C++ has evolved to be very good at specificities of the OS — like file manipulation and memory/threads handling — having all this power in JavaScript's hands is very useful.

If you think about it, Node.js itself was born in a similar way. It followed a similar path to V8’s plus the server and networking capabilities.

Single-Threaded

If you’re a Node developer, you’ll be familiar with V8’s single-threaded nature. Each JavaScript execution context is directly proportional to one thread.

Of course, V8 manages the OS threading mechanism behind the scenes. It works with more than one thread because it’s a complex software and executes lots of stuff at the same time.

We have the main thread that executes the code, another one to compile the code (yes, we can’t stop the execution every time new code has to be compiled), some others to deal with garbage collection, and so on.

However, V8 creates an environment of a single thread for each of JavaScript’s execution context. The rest is kept under its control.

Imagine the stack of function calls your JavaScript code is supposed to make. JavaScript works by stacking one function on top of another, following the order by which each one was inserted/called. Before reaching each function’s content, we can’t know if it calls other functions. If and when that happens, then the called functions will be placed right after the caller in the stack.

When it comes to callbacks, for example, they are placed at the end of the pile.

The management of this stack organization and the memory the process will need is one of the main tasks of V8.

Ignition and TurboFan

Since version 5.9, released in May 2017, V8 comes with a new JavaScript execution pipeline that was built on top of Ignition, V8’s interpreter. It also includes a newer and better optimizing compiler ⁠— TurboFan.

These changes were totally focused on the overall performance and the difficulties Google developers were facing when adapting the engine to all the quick and considerable changes that the JavaScript universe brought up.

From the very beginning of the project, V8 maintainers were always worried about finding a good way to improve V8’s performance at the same pace JavaScript was evolving.

Now we can see huge improvements when running the new engine against the biggest benchmarks:

Source: https://v8.dev/blog/launching-ignition-and-turbofan

You can read more about Ignition and TurboFan here and here.

Hidden Classes

This is another one of V8's magic tricks. JavaScript is a dynamic language. That means that new properties can be added, replaced and removed during execution time. This is not possible with languages like Java, for example, in which everything (classes, method, objects and variables) must be defined before program execution and can’t be dynamically changed after the app starts.

Because of its particular nature, the JavaScript interpreters usually perform a dictionary lookup based on a hash function to know exactly where this variable or that object is allocated in memory.

This costs a lot to the final process. In other languages, when objects are created, they receive an address (a pointer) as one of their implicit attributes. This way, we know exactly where they’re placed in memory and how much space to allocate.

With JavaScript, that’s impossible since we can’t map what doesn't exist yet. That’s where the hidden classes reign.

Hidden classes are almost the same as they are in Java: static and fixed classes with a unique address to locate them. However, rather than doing it before program execution, V8 will do it during runtime, every time we have a “dynamic change” in the object’s structure.

Let’s look at an example to clarify things. Consider the following code snippet:

function User(name, fone, address) {
   this.name = name
   this.phone = phone
   this.address = address
}

Within JavaScript’s prototype-based nature, every time we instantiate a new User object, let’s say:

var user = new User("John May", "+1 (555) 555-1234", "123 3rd Ave")

Then V8 creates a new hidden class. Let’s call it _User0.

Each object has a reference to its class representation in memory. It’s the class pointer. At this point, since we just instantiated a new object, only a hidden class was created in memory. It is empty for now.

When you execute the first line of code in this function, a new hidden class is going to be created based on the previous one, this time _User1.

It is basically the memory address of a User that has a name property. In our example, we’re not using users with just a name as an attribute, but every time you do it, this is the hidden class V8 will load as reference.

The name property is added to the offset 0 of your memory buffer, which means this will be considered our first attribute in the final order.

V8 will also add a transition value to the _User0 hidden class. This helps the interpreter to understand that every time a name property is added to a User object, the transition from _User0 to _User1 must be addressed.

When the second line in the function is called, the same process happens again and a new hidden class is created:

You can see that the hidden classes keep track of the stack. One hidden class leads to another one in a chain maintained by the transition values.

The order in which the properties are added determines how many hidden classes V8 is going to create. If you change the order of the lines in the code snippet we’ve created, different hidden classes will be created as well. That’s why some developers try to maintain the order to reuse hidden classes and therefore, reduce the overhead.

Inline Caching

This is a term very common in the JIT (Just In Time) compilers world. And it connects directly with the concept of hidden classes.

Every time you call a function passing an object as a parameter, for example, V8 will take a look at this action and think: “Hmm, this object was successfully passed twice or more as a param to this function… why not store it in my cache for future calls rather than perform the whole time-consuming-hidden-class-validation process again?”

Let’s recap our last example:

function User(name, fone, address) { // Hidden class _User0
   this.name = name // Hidden class _User1
   this.phone = phone // Hidden class _User2
   this.address = address // Hidden class _User3
}

After sending this User object twice instantiated with any values as a parameter to a function, V8 will jump the hidden class lookup and go directly to the offset’s properties. This is much faster.

However, remember that if you change the order of any attribute assignment in the function, it will result in different hidden classes, so V8 won’t be able to make use of the inline caching feature.

This is a great example to show that developers shouldn’t avoid getting to know the engine more intimately. On the contrary, having such knowledge will help your code perform better.

Garbage Collecting

Do you remember that we mentioned V8 collects memory garbage in a different thread? So, this helps a lot, since our program execution won’t get affected.

V8 uses the well-known strategy of “mark-and-sweep” to collect dead and old objects in memory. In this strategy, the phase where the GC scans memory objects to “mark” them for collection is a bit slow, because it pauses execution in order to achieve it.

However, V8 does it incrementally, i.e., for each GC stop, V8 tries to mark as many objects as possible. It makes everything faster because there’s no need to stop the entire execution until the collection finishes. In large applications, the performance improvement makes a lot of difference.

A Continuous Ride

I hope you enjoyed this piece. The goal was to clarify a bit on V8's structural details, which are at many times misunderstood or ignored.

The whole thing is naturally complex. And it’s constantly evolving. But these are pretty much the core concepts.

At the time of writing this article, the GitHub repo counts 15.3k stars and 2.9k forks. You too can fork the open source code and change the V8 engine as you please. Add your own custom C++ instructions, change the way the hidden classes are dealt with, use your imagination.

But first, don’t forget to give a good read over the official docs. Good reading!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node or you're already familiar with AppSignal, go and check out AppSignal for Node.js.

Building APIs With GraphQL in Your Node Application

Diogo Souza — Wed, 24 Jun 2020 15:04:31 +0000

REST has reigned for a long time in the world of web services. It's easy to implement, allows standardization through RESTful patterns and has lots of libraries that support and facilitate its development. Then came GraphQL, the famous query language for APIs.

What’s GraphQL

To better understand GraphQL, we need to look at what defines it. GraphQL was created to be:

declarative — meaning, you should have the power to choose the data that you want. In other words, you query (request for) some data, defining exactly what you want to get (that is where the schema comes in).
compositional — just like it is in many programming language objects, you can have one field inheriting from another or inside another. Or from both, if you prefer.
strongly-typed — once a field has its type defined, that’s it—a different type isn't allowed.
self-documented — the schema, by itself, offers great documentation (with data types, structure, queries and mutations, etc.).
less verbose — we only get what we asked, which greatly differs from REST, which gives you everything (which isn't very efficient, especially if this everything means a lot of unnecessary data).
among others.

GraphQL is a whole new paradigm. It brings to light the discussion of whether your APIs should have organized and well-structured request and response data in the same way we have when programming data structures in our back-end applications.

The more the number of points discussed above that your API lacks, the more of an indicator that it could benefit from GraphQL. But you don’t have to abruptly migrate to it. Some developers start slowly by creating and exposing some endpoints and asking the clients to consume them. In that way, they gather more insight from both sides that determine if that’s the right path to take.

When it comes to the Node.js universe, we have a bunch of useful tools to help out. express-graphql, for example, is one of the popular server middlewares for integrating GraphQL with Node.js. Apollo is a piece of cake in terms of GraphQL APIs development. It embraces some of the downsides of express-graphql, like the easy enabling of graphql-tools and its patterns. We’ll see more on this later.

Let’s go to some practical stuff. Nothing better than seeing in action how GraphQL fits into a common API example. For this, we’ll be creating a complete API to access some beer data.

First, our API example will enable the registration, login and authentication of users. This way, we can ensure it's secure and unauthorized users can't see our favorite beer list.

Then, we’ll dive into the construction of our API operations, set up a Postgres database to store the credentials and tokens, as well as test everything out.

After we finish, we can celebrate with a beer from our list. So let’s get started.

Did you know that AppSignal launched the support for Node applications? 🎉 We'll be rolling out a whole array of integrations one by one in the coming weeks. Check out the docs and learn how to add your Node app to AppSignal.

Setting Up Our Project

The example we’re about to develop expects that you have Node.js installed. Make sure that it's at least version 8.0.

Next, select a folder of your preference and run the following commands:

npm init -y
npm i apollo-server-express bcrypt express express-jwt graphql jsonwebtoken pg pg-hstore sequelize
npm install -g sequelize-cli

They initialize our Node project with default settings, install the npm dependencies required for the GraphQL + Apollo example, and install the Sequelize CLI Tool, respectively.

Regarding the dependencies, we have:

apollo-server-express: provides direct connection between Express and Apollo GraphQL server.
graphql: the implementation per se of GraphQL in JavaScript.
bcrypt: it’ll be used to hash our passwords.
express and express-jwt: the Express framework itself along with the middleware for validating JWT (JSON Web Tokens) via the jsonwebtoken module. There are a bunch of ways of dealing with the authentication process, but in this article, we’ll make use of JWT bearer tokens.
pg and pg-hstore: the client for Postgres and the serializer/deserializer of JSON to hstore format (and vice versa).
sequelize: the Node.js ORM for Postgres (among other databases) that we’ll use to facilitate the job of communicating with the database.

Note that the Sequelize CLI tool had to be installed globally, otherwise, it wouldn’t be available at any command line interface. As its first command, let’s run the one that will initialize our Node project as an ORM one:

sequelize init

It will create some folders related to the ORM framework, like models, config and migrations (since the framework also handles the migration of our databases).

Now, let’s move on to the database related configs. First of all, we need a real Postgres database. If you still don’t have Postgres installed, then go ahead. As a GUI tool for managing the database, we’ll use pgAdmin. We'll use the web GUI that comes with it.

Next, we’ll create our example’s database. For this, access the web pgAdmin window and create it:

Then, go back to the project and update the content of config/config.json as shown:

"development": {
    "username": "postgres",
    "password": "postgres",
    "database": "appsignal_graphql_db",
    "host": "127.0.0.1",
    "dialect": "postgres",
    "operatorsAliases": false
},

We’re only showing the development section since it’s the only one we’ll be dealing with in the article. However, make sure to update the other related ones as well before deploying your app to production.

Next, let’s run the following command:

sequelize model:generate --name User --attributes login:string,password:string

This is another command from Sequelize framework that creates a new model in the project—the user model, to be exact. This model will be important to our authentication structure. Go ahead and take a look at what's been generated in the project.

For now, we’ll only create two fields: login and password. But feel free to add any other fields you judge important to your design.

You may also notice a new file created under the migrations folder. There, we have the code for the user’s table creation. In order to migrate the changes to the physical database, let’s run:

sequelize db:migrate

Now you can check the results in pgAdmin:

You may wonder where's the table that will store our beer data. We won’t store it in the database. The reason is that I’d like to demonstrate both paths: fetching from the db and from a static list in the JavaScript code.

The project’s set. Now we can move on to implementing the authentication.

Let’s Authenticate!

The authentication must be implemented first because no other API method should be exposed without proper safety.

Let’s start with the schema. The GraphQL schema is the recipe that the API clients must follow to properly use the API. It provides the exact hierarchy of field types, queries and mutations that your GraphQL API is able to execute. It is the contract of this client-server deal. With very strong and clear clauses, by the way.

Our schema should be placed in the schema.js file. So, create it and add the following content:

const { gql } = require("apollo-server-express");

const typeDefs = gql`
    type User {
        id: Int!
        login: String!
    }

    type Beer {
        id: Int!
        name: String!
        brand: String
        price: Float
    }

    type Query {
        current: User
        beer(id: Int!): Beer
        beers(brand: String!): [Beer]
    }

    type Mutation {
        register(login: String!, password: String!): String
        login(login: String!, password: String!): String
    }
`;

module.exports = typeDefs;

For more details on how the schema is structured, please refer to this. In short, the Query type is where we place the API methods that only return data, and the Mutation type is where the methods that create or change data go.

The other types are our own types, like Beer and User—the ones we create to reflect the JavaScript model that will be defined in the resolvers.

The gql tag is used to infer syntax highlighting to your editor plugin (like Prettier). It helps to keep the code organized.

The resolvers, in turn, are the executors of the methods defined in the schema. While the schema worries about the fields, types and results of our API, the resolver takes all this as reference and implements the execution behind.

Create a new file called resolvers.js and add the following:

const { User } = require("./models");
const bcrypt = require("bcrypt");
const jsonwebtoken = require("jsonwebtoken");

const JWT_SECRET = require("./constants");

const resolvers = {
    Query: {
        async current(_, args, { user }) {
            if (user) {
                return await User.findOne({ where: { id: user.id } });
            }
            throw new Error("Sorry, you're not an authenticated user!");
        }
    },

    Mutation: {
        async register(_, { login, password }) {
            const user = await User.create({
                login,
                password: await bcrypt.hash(password, 10),
            });

            return jsonwebtoken.sign({ id: user.id, login: user.login }, JWT_SECRET, {
                expiresIn: "3m",
            });
        },

        async login(_, { login, password }) {
            const user = await User.findOne({ where: { login } });

            if (!user) {
                throw new Error(
                    "This user doesn't exist. Please, make sure to type the right login."
                );
            }

            const valid = await bcrypt.compare(password, user.password);

            if (!valid) {
                throw new Error("You password is incorrect!");
            }

            return jsonwebtoken.sign({ id: user.id, login: user.login }, JWT_SECRET, {
                expiresIn: "1d",
            });
        },
    },
};

module.exports = resolvers;

The resolvers follow a pattern that’s inherently async because it’s Promise-based. Each operation must have the exact same signature as the one defined in the schema.

Note that, for all query operations, we’re receiving a third argument: user. That one is going to be injected via context (still to be configured in index.js).

The jsonwebtoken dependency now takes over signing in the user according to the provided credentials and then generating a proper JWT token. This action will happen in both registration and login processes.

Also, notice that an expiry time must be set for the token.

Finally, there’s a JWT_SECRET constant that we’re using as the value for secretOrPrivateKey. That is the same secret we’ll use in the Express JWT middleware to check if the token is valid.

This constant will be placed in a new file, called constants.js. Here’s its content:

const JWT_SECRET = "sdlkfoish23@#$dfdsknj23SD";

module.exports = JWT_SECRET;

Make sure to change the value to a safe secret of yours. The only requirement is that it be long.

Now, it’s time to configure our index.js file. Substitute its content with the following:

const express = require("express");
const { ApolloServer } = require("apollo-server-express");
const jwt = require("express-jwt");
const typeDefs = require("./schema");
const resolvers = require("./resolvers");
const JWT_SECRET = require("./constants");

const app = express();
const auth = jwt({
    secret: JWT_SECRET,
    credentialsRequired: false,
});
app.use(auth);

const server = new ApolloServer({
    typeDefs,
    resolvers,
    playground: {
        endpoint: "/graphql",
    },
    context: ({ req }) => {
        const user = req.headers.user
            ? JSON.parse(req.headers.user)
            : req.user
            ? req.user
            : null;
        return { user };
    },
});

server.applyMiddleware({ app });

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
    console.log("The server started on port " + PORT);
});

If you use Express as your web server, this code may look familiar, except for the fact that we have two servers being set here.

Express app is going to be used as usual. We’re creating it, adding a middleware (jwt) and starting it up. However, the ApolloServer may come along to add the necessary GraphQL settings.

ApolloServer receives the schema (typeDefs), resolvers, playground and a context as arguments. The playground property states which endpoint is going to redirect to Prisma’s GraphQL Playground view. It is a built-in IDE to help us with testing of our GraphQL APIs.

The context, in turn, is an optional attribute that allows us to make quick conversions or validations prior to the GraphQL query/mutation executions. In our case, we’ll use it to extract the user object from the request and make it available to our resolvers functions.

The server object is the one that applies the middleware, passing the app object as a param.

This is it. Let’s test it now. Run the application with the following command:

node index.js

Then, access the address http://localhost:3000/graphql and the Playground view will show up.

Our first test will be to register a new valid user. So, paste the following snippet into the query area and hit the “Execute Query” button:

mutation {
  register(login: "john", password: "john")
}

A valid token will return as shown in the figure below:

This token can already be used to access sensitive methods, like the current.

If you don’t provide a valid token as an HTTP header, the following error message will be prompted:

To send it properly, click the “HTTP HEADERS” tab at the bottom of the page and add the following:

{
  "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6NSwibG9naW4iOiJhcHBzaWduYWwiLCJpYXQiOjE1ODk5MTYyNTAsImV4cCI6MTU4OTkxNjQzMH0.bGDmyi3fmEaGf3FNuVBGY7ReqbK-LjD2GmhYCc8Ydts"
}

Make sure to change the content after Bearer to your version of the returned token. You will have a result similar to the figure below:

Obviously, if you already have a registered user, you can get the token by logging in via login mutation:

mutation {
  login(login: "appsignal", password: "appsignal")
}

Once again, if one of your credentials is wrong, you’ll get the corresponding error message.

Our Beer API

For the sake of simplicity, we won’t create our Beer domain in the database. A single JS file will do the job. But I’d recommend that you migrate to our ORM model as well, making use of the knowledge you’ve got so far.

Let’s start with this, then. This is the code for our beers.js file (make sure to create it too):

var beersData = [
    {
        id: 1,
        name: "Milwaukee's Best Light",
        brand: "MillerCoors",
        price: 7.54,
    },
    {
        id: 2,
        name: "Miller Genuine Draft",
        brand: "MillerCoors",
        price: 6.04,
    },
    {
        id: 3,
        name: "Tecate",
        brand: "Heineken International",
        price: 3.19,
    },
];

module.exports = beersData;

Feel free to add more data to it. I reserve the right of not knowing their correct prices.

Once the main GraphQL setup structure has been set, adding new operations is quite easy. We just need to update the schema with the new operations (which we’ve already done) and add the corresponding functions into the resolvers.js.

These are the new queries:

async beer(_, { id }, { user }) {
    if (user) {
        return beersData.filter((beer) => beer.id == id)[0];
    }
    throw new Error("Sorry, you're not an authenticated user!");
},

async beers(_, { brand }, { user }) {
    if (user) {
        return beersData.filter((beer) => beer.brand == brand);
    }
    throw new Error("Sorry, you're not an authenticated user!");
},

They’re simply filtering the data based on the given arguments. Don’t forget to import the beersData array object:

const beersData = require("./beers");

Restart the server and refresh your Playground page. Note that we made those new queries safe too, so it means you’ll need to provide a valid token as header.

This is the result of a query by brand:

In this call, we’re making use of Query Variables. It allows you to call GraphQL queries by providing arguments dynamically. It’s very useful when you have other applications calling the GraphQL API, rather than just a single web IDE.

This is the magic of GraphQL. It allows even more complicated query compositions. Imagine, for example, that we need to query two specific beers in one single call, filtering by a list of ids.

Currently, we only have operations that filter by one single id or one single brand name. Not with a list of params.

Instead of going directly to the implementation of a new query function that would do it, GraphQL provides a feature called Fragments. Look how our query would be:

query getBeers($id1: Int!, $id2: Int!) {
  beer1: beer(id: $id1) {
    ...beerFields
  }
  beer2: beer(id: $id2) {
    ...beerFields
  }
}

fragment beerFields on Beer {
  id
  name
  brand
  price
}

For this case, you’d need to provide the exact beer name for each of the results. The fragment defines from where it’s going to inherit the fields, in our case, from the Beer schema.

Basically, fragments allow you to build a collection of fields and then include them in your queries. Don’t forget to feed the Query Variables tab with the ids:

{
  "id1": 1,
  "id2": 3
}

The result will look like the following:

Notice that the Authorization header is also there, hidden in the tab.

Conclusion

It took a while, but we got to the end. Now you have a fully functional GraphQL API designed to provide queries and mutations and, more importantly, in a secure manner.

There is a lot you can add here. Migrate the Beer’s model to store and fetch data directly from Postgres, insert some logs to understand better what’s going on, and place some mutations over the main model.

Apollo + Express + GraphQL have proven to be a great fit for robust and fast web APIs. To learn more, please be sure to visit http://graphql.org/learn/. Great resource!

P.S. If you liked this post, subscribe to our new JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you'd love an all-in-one APM for Node or you're already familiar with AppSignal, go and check out AppSignal for Node.js.